Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Let's say I have a website with frontend and backend. The website allows users to upload some data ranging from hundreds of MB to GB. How do you effectively upload this data?

  • The easiest way would be to upload it from client/browser to the frontend server. Then send it via API or something to the backend server which has connected data storage, where we'll save the data. This would, however, run really slow, because the upload will be twice as big.
  • Another way that occurred to me would be to send the data directly from javascript running inside the browser to the backend's API. This can be inappropriate when I don't want the backend/backend's API to be accessible to public. It's also not good for the architecture (if you have fe and be, you probably don't want to communicate from client to be).

So, do you have any ideas? Or is there some kind of general way to do this effectively? Does it include CDNs? Or should I store the data in a database as base64 strings if it'd only be few hundreds of MB pre file?

share|improve this question

1 Answer 1

I definitely do not recommend encoding the uploads and storing them in a relational database, that's just introducing a lot of overhead and complexity for no gain. File systems were designed for files :)

You definitely want to design this to be flexible, because today you might use your own server, tomorrow AWS, and months from now a CDN. Or any combination of the above. To that effect, try:

  • Have uploads post to upload.yourdomain.com where the minimum needed to check auth / permissions and the upload itself will run.

  • Create static.yourdomain.com to use for linking to these uploads in order to serve them to browsers.

These could, at least originally, all reside on the same server. The point is, you have the flexibility of having the upload sub domain send files to a storage network, large RAID array, content delivery network or whatever else you come up with. This sub domain could also handle jobs like virus scanning, re-sizing, encoding, or whatever other processing has to happen when an upload completes.

The static domain is then easy to change as you change your infrastructure, it's job is to point to the correct location of the processed uploads and ensure that the proper headers for a static server are set.

For access, just use a *.yourdomain.com cookie and you're done - no need to worry about third party cookies, certificate complexity (one wildcard is fine) and other quirks associated with spreading this out.

This of course makes more or less sense depending on what kinds of uploads you're dealing with, but the general idea should remain pretty much the same. Just think a bit about the TTLs you set for the DNS-A records for both sub domains. Don't set them over 4 hours until you've settled on a rather permanent setup, and quite possibly much lower until you have.

share|improve this answer
    
Yes.. great answer, thank you. Few follow up questions though. 1. "Have uploads post top upload.mydomain.com" is meant to upload directly from browser or from frontend? 2. If I'd forward these uploads from upload.mydomain.com to FTP, AWS or whatever, it would cost additional time. This is something I'm trying to avoid. See what I mean? If I choose AWS in the future and the upload will first go from user to upload.md.com and then again via network to AWS, it'd be double time for user to wait until the upload is finished, right? Is this how it's done (eg. by Google Drive and others)? –  Jakub Žitný Nov 14 '13 at 6:23
    
@JakubŽitný Hard to really say without more detail but you could easily point it right at AWS (or wherever you're storing the files), then make download serve them with the appropriate headers. You could probably avoid the double time by having the storage servers accept uploads directly. –  Tim Post Nov 14 '13 at 6:27
    
But wouldn't it be a security risk to upload directly from client to aws? The client could reveal the "password" to the public right? Isn't there any possibility to forward the upload directly to another server? Something like chunked forwarding through the upload.mydomain.com to the AWS would add only little overhead. –  Jakub Žitný Nov 14 '13 at 6:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.