12

We are debating whether to use the Content-MD5 header.

Pros:

  • The CMS allows us to easily include it with minimal overhead (cached responses in 80%+ of the cases).
  • It would add another layer of protection against problems.

Cons:

  • The Content-Length header is always present (even on dynamically created pages) so the client should not need another form of validation.
  • So far we are unaware of any problems caused by corruption.
  • MD5 checks add latency to web page load times.

Points:

  • Do certain media types include their own form of digest that makes this unnecessary?
  • If TCP offers this already why was it included in the HTTP standard?
  • What are the existing real-life uses?
  • Is the MD5 check negligible?

It is no real problem for this to be added to the unit tests and implemented, about an hours work; however if it is detrimental then we want it added to the higher level sniff tests used in website "health check"s.

Improve this question

2 Answers 2

Reset to default
10

TCP already has error correction, but this only helps you on the TCP layer. An intermediary HTTP proxy or load balancer can corrupt the data on the HTTP layer, and then retransmit it. A HTTP MD5 makes it possible to detect this corruption. The reason why nobody really talks about this need is that the problem is very rare indeed; most HTTP proxies etc "just work".

The RFCs allude to security. IMHO this is so weak it should be ignored -- if you need any kind of real security and confidentiality, then you need HTTPS.

Do certain media types include their own form of digest that makes this unnecessary?

Not anything really good. But a few bit errors in photos, streaming video etc will often be imperceptible to humans.

I would say it depends on the use case:

  • For REST based web services a digest adds a useful layer of additional error correction. See this AWS failure as an example.
  • For applications dealing with mission-critical data over plain HTTP it is worth implementing. Content-MD5 gives clients the option to verify the end-to-end transmission integrity.
  • For 'normal' web sites serving up text and media of 'normal' value the Content-MD5 header serves no purpose. And I honestly don't even know how many mainstream browsers (PC, especially mobile) actually support it.
Improve this answer
6
  • 1
    That AWS failure case is really insidious. It's a few years old, but really a fascinating example of a failure mode I would never have thought of. Very interesting one to look out for when using storage of data remotely. I wonder about some of the NoSQL solutions and how they handle such issues.
    – artlung
    Commented Sep 10, 2010 at 1:49
  • This makes it quite easy to pass the decision onto the client. An option like this can now be offered as a "nice to have" but not an essential criteria. If Amazon can deploy a load balancer and cause these errors, its likely to crop up somewhere eventually and nothing looks worse than an inconsistently troublesome website.
    – Metalshark
    Commented Sep 10, 2010 at 9:59
  • That really depends on where the flipped bit is. If it's the least-significant bit, then it'll be imperceptible. But there's a huge difference between the colors rgb(255, 0, 0) and rgb(127, 0, 0). With raw video a single pixel corruption will be less perceptible because it's on screen for so brief a moment, but because most online video use highly efficient compression algorithms, a single bit being flipped can cause half of the picture to be corrupted or shifted across the screen.
    – Lèse majesté
    Commented Sep 10, 2010 at 15:00
  • Also, like you said, banks ought to just use HTTPS, so isn't there no point for them to use Content-MD5 either, since SSL/TLS already provides a message digest at the application layer?
    – Lèse majesté
    Commented Sep 10, 2010 at 15:17
  • 1
    @Lèse majesté: Regarding bit errors, I agree in the abstract case. But remember that most streaming video fx uses an application-specific transport over UDP or TCP to give the 'right' tradeoff between error correction and speed -- and streaming video thus wouldn't be a use case for Content-MD5. Regarding banks should use HTTPS, I agree, and I'm rephrasing to make it more clear.
    – user1857
    Commented Sep 10, 2010 at 16:15
1

MD5 checks add latency to web page load times.

If true (and the latency is not entirely trivial) then I'd say it wasn't worth it.

In general, I believe, the last modified header is most commonly used to determine if a page has changed. Assuming you provide meaningful value there, I see no need for the content-md5 header.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.