I'm assessing the options for users to submit a significant amount of data to a web solution I'm writing.

I can write the validations and proper input of the received data, so I'm going to pass over those concepts in this question. But I am assessing the best way of users to submit a myriad of records to my system. Entering items one by one is out of the question. These records would be kept up to day either manually or by some database export on their end. That's their responsibility and I can only assume it will be done somehow.

The only two options that I can see working are CSV/Tab-Delimited Values, or an Excel spreadsheet. I am preferring the latter because it's the easiest for the user, and easy for me to parse on my end, then archive. This is done through file upload. I've considered Dropbox, but that adds a layer for those that might not know how to use Dropbox. That is a concern.

Are there other accepted or tried & true ways of users submitting records, where the data set is massive? I'm looking for more options. The two challenges I face are:

  1. Ease of use for the user
  2. Parsing large files on my end after the file is uploaded

Looking forward to your insight.


Edited && Answered:

I completed the task by providing an XLXS template - Portable and easy to follow/enter data for noobs.

Three columns. No extra columns. I don't need that, and it will only confuse the parser as it is now.

I'm testing every row individually pre-import. Some of these files will be many many rows.

CSV is too complicated for my target market as per the expected pitfalls of more commas. A big portion of my market isn't even online so being simple is absolutely critical. A lot of people can't follow instructions, and since this is organized, an Excel template is the best.

  • 1
    HTTP does not always lend well to uploading. Very large files will at some physical size consistently fail. This is also true for FTP though FTP does allow for very large files. Whether or not you can do this depends upon the file sizes. Gzip can help except that decompression can also have limits. You will have to experiment to find your limits. FTP should give better results. If you find you are hitting limits, the only solution is to create an API and a client to upload the data. CSV is always best. Any spreadsheet can export CSV. Keep it simple, secure, and consistent. Cheers!! – closetnoc Jul 8 '18 at 15:52
  • OK, so I have to ask what you mean by "very large files". So 20k records, a quick .xlsx output is about 11 MB. Let's double that, maybe double it again. FTP is beyond the ability of my users. They won't compress either. They are not computer people so an API is out of the question. Hey I'd love to run a separate PostgreSQL table for them, but I can't rely on anything remotely close to that. I've also heard that CSV is a problem, and the exporting might be an issue for some. – Rich_F Jul 8 '18 at 16:02
  • Using HTTP to upload a file depends upon not only your end, but theirs as well. Still, 11meg may not be too bad. I am just saying there are limits. – closetnoc Jul 8 '18 at 17:44
  • Yes. I’m more into looking at options. – Rich_F Jul 8 '18 at 17:47
  • I have gone through similar excersizes before. It is a balancing act for sure! With broadband being faster these days, you will likely have more options than I did. Good luck! Cheers!! – closetnoc Jul 8 '18 at 18:10

Browse other questions tagged or ask your own question.