3

Does Google's bot send HEAD requests as well as GET requests?

I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in Google Search Console.

To verify that the Tornado application is not accepting HEAD requests I do:

curl -I example.com/this-url-exists

HTTP/1.1 404 Not Found
Date: Thu, 10 Mar 2016 08:49:07 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 7520
Connection: close
X-Ua-Compatible: IE=edge,chrome=1
Server: TornadoServer/4.2.1

I suspect that the Googlebots are doing the same for some of the crawling.

Can this be right?

Improve this question
3
  • 2
    See the spec. The server SHOULD send the same header fields in response to a HEAD request as it would have sent if the request had been a GET. Whether or not GoogleBot makes HEAD requests is somewhat beside the point, your server is violating the rules of HTTP and is broken.
    – Quentin
    Mar 10, 2016 at 11:57
  • yes agree, we will fix that
    – a_b
    Mar 10, 2016 at 12:09
  • check this answer too webmasters.stackexchange.com/a/55246/33802
    – Ali Hashemi
    Sep 12, 2017 at 7:30

1 Answer 1

Reset to default
2

I'm pretty sure Googlebot does not send HEAD requests - at least not with respect to crawling and indexing a website.

My access logs from January and February 2016 show no HEAD requests from the Googlebot.

Even if Googlebot did use HEAD requests I would be very surprised if these resulted in a 404 error in Google Search Console. Google should only report a 404 error on a failed GET request.

Improve this answer
5
  • Ok, thank you. Maybe a 404 for a HEAD request does however show up as a "Soft 404" in Search Console (I don't know...). Our logs do show some HEAD request, but we didn't log the user-agents so couldn't tell if it was Goolge - I will try to log those as well...
    – a_b
    Mar 10, 2016 at 10:00
  • 1
    A "soft 404" is simply a response that "looks like" a 404, but isn't returning a 404 HTTP status code. (For example: accidentally returning your 404 page with a 200 OK status.) Yes, I see some HEAD requests in the logs, but none from the Googlebot. (In order to validate the Googlebot you need to check both the User-Agent and a reverse lookup on the IP address - since there are a lot of bots pretending to be the Googlebot.)
    – MrWhite
    Mar 10, 2016 at 10:08
  • aah yes, good point. i will try to do that. thank you
    – a_b
    Mar 10, 2016 at 10:11
  • In fact, checking the IP addresses might be enough in this case. Only if it did resolve to Google would you need the User-Agent to confirm. If none resolve to Google then job done.
    – MrWhite
    Mar 10, 2016 at 10:26
  • Google did used to use a HEAD request to check modification dates and conserve bandwidth. However, as you said, I have not seen a HEAD request in years. So that blows that theory!! It was the case at one point. I would advise that the IP address be checked. Also, keep in mind that it may not all be googlebot. Check the user agent carefully. A 404 can result from a HEAD request, however, how Google deals with this is up to debate since I agree that Google no longer uses HEAD requests. Kind of a catch-22 huh?? What came first? The Pterodactylus or the Pterodactylus egg? Does it even matter??
    – closetnoc
    Mar 10, 2016 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.