Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD requests. I've seen these come from Squid proxy servers and bots from AddThis, Google and Bing.

My understanding is that most search engines will make GET requests when crawling sites so is there a genuine reason for allowing HEAD requests?

share|improve this question

1 Answer 1

Some search engines and bots send HEAD request to pages before sending the GET request for reasons like:

  • Checking if the page size has changed
  • Checking the last modified date
  • etc. (Any other info the head would give them!)

This would help large crawlers save a lot of bandwidth if they know a page has not been changed meanwhile and they don't have to crawl it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.