2

I have users that need to upload files larger than 4GB. What's the best way to do this? Most upload scripts fail with large objects. Also, when sending huge data sets, TCP/IP's 1-in-4billion promise of consistency will frequently result in file corruption.

Is there a good flash or java uploader that will chunk and compute MD5 hashes on every chunk?

| improve this question | |
  • 1
    I would avoid the HTTP protocol for transmitting files that large. You may want to look into creating your own protocol that implements error checking, etc. – George Edison Oct 1 '11 at 7:42
  • 2
    It's not just 'upload scripts' that will fail, the server configuration might reject it. – MrWhite Oct 1 '11 at 11:26
  • 1
    Creating my own protocol is probably a mistake. – vy32 Oct 4 '11 at 2:24
2

I have users that need to upload files larger than 4GB. What's the best way to do this?

FTP (SFTP/FTPS) - not http at all

| improve this answer | |
  • 1
    FTP has no error checking. – vy32 Oct 4 '11 at 2:24
  • 2
    @vy32 FTP doesn't, but neither does HTTP. But you shouldn't be using FTP anyway. If you use SFTP, not only are file resumes better supported, but it also does a checksum every 32kb. Also, TCP/IP has error detection/correction using checksums. Not only that, but most physical layer protocols also have error correction built in (e.g. Ethernet uses CRC-32 checksums). – Lèse majesté Jan 1 '12 at 0:07
  • 1
    For the record, Ethernet does not use CRC-32 checksums. FTP is based on TCP which uses a 32-bit IP checksum and a 32-bit TCP checksum. However, the problem with a 32-bit checksum is that you'll get an undetected error with 1 in 4 billion probability. – vy32 Jan 2 '12 at 0:43
  • 2
    @vy32: Yes, it does. The final field in an Ethernet frame is a CRC-32 checksum. And according to the IETF TCP and UDP use 16-bit CRCs, not 32-bit. But, as I said, you should be using SFTP anyways, so you'd have error correction across 3 layers. And SFTP uses SHA or MD5 for its checksums. So the chances of a randomly induced error slipping through is vanishingly small (even more so than CRC-32 on top of CRC-16). – Lèse majesté Jan 2 '12 at 3:00
  • 1
    Well, we see transmission errors all the time. They are not vanishingly slow. We are doing transfers of 10TB and 100TB sized objects and we routinely see a transmission error every 1-10 TB. – vy32 Jan 2 '12 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.