Reliable SFTP file upload without duplicates

SFTP is not the most fancy technology, but there are quite many areas where it's used. Some systems use it for exchanging messages. In this scenario one system generates a file and uploads it to SFTP. Another system scans SFTP for new files and processes them. Processed files are removed from incoming folder. Simple, unless you start thinking about error handling.

File upload can be easily interrupted. In this case remote system will pick up partially uploaded file. If you're lucky, remote system is smart enough to detect corrupted files and handle them accordingly, but what if not? Trick is to use temporary file.

Temporary file is not picked up by remote system. If upload crashes, we can resume it or even restart whole upload. When we're sure that temporary file was uploaded, it's time to rename or move it to proper location. Now remote system can pick it up.

Let's imagine that remote system is even less reliable and does not recognize duplicate files. Now we need to make sure that we will never rename temporary file to proper name twice. Unfortunately, if network will drop off before we will receive answer from rename operation. We can verify state by checking whether temporary file still exists and retry renaming if it does.

It is important to persist fact of successful upload of temp file. In case of transactional persistence, make sure that it won't be rolled back.

Here is a flow that worked well for us:

After several months in production we did not have any issues with that solution.

Choosing partner to develop MVP for startup

Probably best option for startup is when founders can develop MVP on their own, but what if not? In such case, outsourcing companies are ready to offer some help. Here are some points which seem to be important for choosing such development partner.

• Maturity of team - in case of MVP for startup, there is not much time to invest into forming team. Forming and storming phases can easily take a month. During that period team productivity will be reduced. It’s ok for long term cooperation, but would be nice to avoid when short time result is important.
• Team member profiles - nothing will happen without qualified team members. It’s ok to have different levels in team, but team lead has to guarantee result. Would be nice to have a team with 1 senior and remaining medium level developers. Too many seniors spend too much time on theory. Juniors waste time of seniors. Team lead that had previous experience with startups is a big benefit as he might know which technical compromises are ok in short term and which will bite immediately.
• Knowledge sharing inside of team - team should have a plan to cover unexpected absence of any team member. People get sick, go to vacation, change projects.
• Experience with required technologies - would be nice to make sure that team members have expertise in core technologies that will be used in project. It’s ok to have some minor innovation, but bigger research can take too much time and lead to moderate results.
• Dedication - team that has too many obligations from previous projects/clients can be distracted too often. Would be nice to know if team has to maintain previous projects and in what amount.
• Schedule - as MVP has quite strict deadlines, most outsourcing companies won’t be able to provide well established team that fast.
• Flexibility - contract should allow introducing changes to requirements. Would prefer time-and-material to fixed price, as value of project done by fixed price can be much lower. Also it would be nice to know about team plans after expected delivery date. It is possible that team has next project scheduled and it will be problematic to finish started MVP and resume works in case of MVP success.
• UX capabilities - if MVP includes UX, good cooperation with designer is a big benefit. If team has worked with this designer previously, there will know how to play nice together, otherwise sides can pull project into different directions.
• Mobile experience - if project requires mobile development, would be ideal to get it from same partner.
• Billing and pricing models - need to know what side activities of team members will be included into project time.
• Price - should take all previously mentioned points into account, as it might turn out that highest price per hour will lead to lowest total price.