Requesting advice for accepting large file uploads

Question

I asked this question over at StackExchange but was advised to try here...

I'm looking at building a web app that includes a file upload element. I'd like users to be able to upload files of any type and of fairly large size (say, up to 100MB). This will be a publicly accessible site, so security is obviously very important.

I've done a decent amount of googling in search of answers, but it's difficult when I don't really know exactly what I'm searching for.

My experience is mainly with PHP, but I realise that PHP is not considered to be the best when it comes to file uploading, so I'm happy to look at other languages if necessary. Although, if a decent solution using PHP can be acheived, that would be preferable.

As I have no experience with this kind of project, I'm also fairly in the dark on what kind of server setup is required for such an app.

I have braistormed a few ideas, but am willing to budge on them if unreasonable:

• I'd like to use Amazon S3 to store the files if possible (to reduce the load on the server)
• I'd like to be able to rename the files after upload
• I'm considering Uploadify (uploadify.com) for the client side

Basically, imagine I was looking to build a file-sending app like wetransfer.com or yousendit.com and you'll get the general idea.

I'm familiar with all the usual PHP file upload issues (checking mime-types, upload_max_filesize, memory_limit, etc, etc) covered by 99% of posts on the internet on this topic, but obviously this project goes a fair bit beyond your average, run-of-the-mill avatar upload script.

I know this is a massive topic and I'm obviously not expecting anyone to present me with a magic solution, but basically I'm looking for some pointers on where to start. Can anyone recommend any good books, articles or websites where I can gain a better understanding of the requirements of the task? Covering everything from the programming to the server requirements? Even if it's just a list of keywords or phrases that I should be googling.

Answer 1

There is an open source php5 project similar to yousendit that you might be able to get some ideas from: http://zend.to/

This solution works nicely if you are not going over 2GB file uploads, and works okay for larger uploads as long as you are aware of the limitations of http uploads using browsers:

• Not all browsers support file uploads greater than 2GB. In fact.. Safari, Chrome and Opera are the only browsers that I know of that do.
• The uploading computer needs as much free space on their disks as the file size they are uploading. i.e., if you are uploading a 5GB file, you will need 5Gig of free space during the upload process. The space is freed after completion or if they cancel.
• The upload process on large files can take a long time on old/slow computers.

Plus sides to http uploads:

• No client to install/configure
• No java apps to deal with
• Simple https to secure uploads/downloads

If you decide to not use http based uploads, then you will need to implement some kind of client/server based tool: java, browser based plugins or a downloadable app. At this point your hurdles will be the normal client/server issues such as authentication, resuming uploads, etc.

From a hosting point of view (off the top of my head):

• Virus/Malware scanning
• Retention policies
• File integrity (i.e., md5 sums on client and server)
• Firewall/NAT/Proxy issues
• Bandwidth throttling
• Maybe look into data-dedup stores to reduce storage costs and bandwidth fees. S3 is cheap on storage, but the costs can run up on transfer fees.
• Preventing hot linking to stored assets (i.e., keeping people from passing out the url's to files)
• Caching files
• Location aware hosting like a CDN (Amazon has a CDN like s3 plan)

node.js might be a place to look into for a client/server socket app.

If you do not have the budget for enterprise storage, maybe a ZFS based file server with some SSD's for write caching and a boat load of ram: www.nexentastor.org, www.openindiana.org, etc.

Really just brainstorming here. I have to deal with very large file transfers daily and it is always a pain (cross-platform 10GB+ file transfers). I use the zend.to for quick and easy email based transfers, SFTP for transfers that need to linger a while, and we pay $$$ for an enterprise file exchange tool (aspera faspex) for our large file deliveries to external clients.

Hope this helps some!

Answer 2

If you are willing to look at other languages my advice will be to go with Django (Python webframework), here is exactly what you need: https://docs.djangoproject.com/en/dev/topics/http/file-uploads/ Documentation is extensive, all round cases are handled by framework, so you just need to learn it. It's not hard but takes some time. This is not a simplest or fastest route but if you are going to do some webdev in the future consider this option as it will pay out definitely.

If python + django is too much to learn try to go with some php webframework. A good framework always has some helper functions for you handle this situation in professional manner. I'm no expert in php webframeworks so I can advice no one.