2

I have users that need to upload files larger than 4GB. What's the best way to do this? Most upload scripts fail with large objects. Also, when sending huge data sets, TCP/IP's 1-in-4billion promise of consistency will frequently result in file corruption.

Is there a good flash or java uploader that will chunk and compute MD5 hashes on every chunk?

Improve this question
3
  • 1
    I would avoid the HTTP protocol for transmitting files that large. You may want to look into creating your own protocol that implements error checking, etc.
    – George Edison
    Oct 1 '11 at 7:42
  • 2
    It's not just 'upload scripts' that will fail, the server configuration might reject it.
    – MrWhite
    Oct 1 '11 at 11:26
  • 1
    Creating my own protocol is probably a mistake.
    – vy32
    Oct 4 '11 at 2:24
2

I have users that need to upload files larger than 4GB. What's the best way to do this?

FTP (SFTP/FTPS) - not http at all

Improve this answer
8
  • 1
    FTP has no error checking.
    – vy32
    Oct 4 '11 at 2:24
  • 2
    @vy32 FTP doesn't, but neither does HTTP. But you shouldn't be using FTP anyway. If you use SFTP, not only are file resumes better supported, but it also does a checksum every 32kb. Also, TCP/IP has error detection/correction using checksums. Not only that, but most physical layer protocols also have error correction built in (e.g. Ethernet uses CRC-32 checksums).
    – Lèse majesté
    Jan 1 '12 at 0:07
  • 1
    For the record, Ethernet does not use CRC-32 checksums. FTP is based on TCP which uses a 32-bit IP checksum and a 32-bit TCP checksum. However, the problem with a 32-bit checksum is that you'll get an undetected error with 1 in 4 billion probability.
    – vy32
    Jan 2 '12 at 0:43
  • 2
    @vy32: Yes, it does. The final field in an Ethernet frame is a CRC-32 checksum. And according to the IETF TCP and UDP use 16-bit CRCs, not 32-bit. But, as I said, you should be using SFTP anyways, so you'd have error correction across 3 layers. And SFTP uses SHA or MD5 for its checksums. So the chances of a randomly induced error slipping through is vanishingly small (even more so than CRC-32 on top of CRC-16).
    – Lèse majesté
    Jan 2 '12 at 3:00
  • 1
    Well, we see transmission errors all the time. They are not vanishingly slow. We are doing transfers of 10TB and 100TB sized objects and we routinely see a transmission error every 1-10 TB.
    – vy32
    Jan 2 '12 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.