Effective upload to backend?

Question

Let's say I have a website with frontend and backend. The website allows users to upload some data ranging from hundreds of MB to GB. How do you effectively upload this data?

• The easiest way would be to upload it from client/browser to the frontend server. Then send it via API or something to the backend server which has connected data storage, where we'll save the data. This would, however, run really slow, because the upload will be twice as big.
• Another way that occurred to me would be to send the data directly from javascript running inside the browser to the backend's API. This can be inappropriate when I don't want the backend/backend's API to be accessible to public. It's also not good for the architecture (if you have fe and be, you probably don't want to communicate from client to be).

So, do you have any ideas? Or is there some kind of general way to do this effectively? Does it include CDNs? Or should I store the data in a database as base64 strings if it'd only be few hundreds of MB pre file?

Answer 1

I definitely do not recommend encoding the uploads and storing them in a relational database, that's just introducing a lot of overhead and complexity for no gain. File systems were designed for files :)

You definitely want to design this to be flexible, because today you might use your own server, tomorrow AWS, and months from now a CDN. Or any combination of the above. To that effect, try:

• Have uploads post to upload.yourdomain.com where the minimum needed to check auth / permissions and the upload itself will run.

• Create static.yourdomain.com to use for linking to these uploads in order to serve them to browsers.

These could, at least originally, all reside on the same server. The point is, you have the flexibility of having the upload sub domain send files to a storage network, large RAID array, content delivery network or whatever else you come up with. This sub domain could also handle jobs like virus scanning, re-sizing, encoding, or whatever other processing has to happen when an upload completes.

The static domain is then easy to change as you change your infrastructure, it's job is to point to the correct location of the processed uploads and ensure that the proper headers for a static server are set.

For access, just use a *.yourdomain.com cookie and you're done - no need to worry about third party cookies, certificate complexity (one wildcard is fine) and other quirks associated with spreading this out.

This of course makes more or less sense depending on what kinds of uploads you're dealing with, but the general idea should remain pretty much the same. Just think a bit about the TTLs you set for the DNS-A records for both sub domains. Don't set them over 4 hours until you've settled on a rather permanent setup, and quite possibly much lower until you have.