5

Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD requests. I've seen these come from Squid proxy servers and bots from AddThis, Google and Bing.

My understanding is that most search engines will make GET requests when crawling sites so is there a genuine reason for allowing HEAD requests?

5

Some search engines and bots send HEAD request to pages before sending the GET request for reasons like:

  • Checking if the page size has changed
  • Checking the last modified date
  • etc. (Any other info the head would give them!)

This would help large crawlers save a lot of bandwidth if they know a page has not been changed meanwhile and they don't have to crawl it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.