Is there a reason why a URL on our website would sometimes return 200 and sometimes return a 301? This appears to happen on a number of our URLs even something basic like example.com/robots.txt

When I load the URL myself - it always returns a 200

The site is configured to work as www.example.com/robot.txt - could it be that the bots are looking at the non-www version and getting redirected?

  • 1
    Where are you seeing this information? Are you seeing this in your server logs or in Google Search console, or some other data source? Dec 1, 2021 at 16:13
  • Through server access logs. However, it doesn't list the full URL - just the path after the domain.
    – xylar
    Dec 1, 2021 at 16:23
  • 1
    Can you reconfigure you logging to add the host? I usually recommend doing that. Dec 1, 2021 at 16:27
  • Yes, that's a good idea. Why would the bots be looking at non-www non-http URLs, especially as we've specified the correct domain in Search Console. Is there another way to force it?
    – xylar
    Dec 1, 2021 at 16:29
  • 2
    You can't control what bots crawl. If they find a link to it, they crawl it. Dec 1, 2021 at 16:49

1 Answer 1

  1. When access logs show differnt behavior for the "same" URL, make sure it's indeed the same url: check the protocol, the subdomain, the TLD, zones, path and the query string. Default access log logging pattern is pretty bad and misses a lot of dimensions.

  2. Google does not normally use sitemap for crawling and when it does, it's just a supplemental reference. Main reference would be links found on the web. So you can easily get google crawling urls that are different from what's in sitemap. You have rel canonical and redirections logic to mend that.

  3. Here how access log works with redirections:

    1. Client (Googlebot in our case) sends a request url (say, with "wrong" subdomain or protocol) to the server to port :80.

    2. On the server, on port 80, we have a web-server software that listens for requests like that. It gets the request and runs the request through a bunch of logic to understand what to do to it. It sees that it has rules (typically in .htaccess, but not always) to redirect all http:// urls to their https:// version, so it sends a response to the client with code 301 and location field set to the https:// version of the url.

    3. Right after it sends the client away, without waiting for the next hit, the web-server makes a record in its access logs indicating that there had been a hit and it dealt with it so and so. After it, the web-server forgets about that client forever.

    4. The client gets the response from the server and then it organically makes another request to the location of the received response, so to the https:// version of the page.

    5. The web-server gets another request and treats it as if there were never the original request: it goes through a bunch of logic and sees that this pattern of request has to be given away to php to get a response, so it gives the request (and the cookies) to the php endpoint and php returns a body to respond to the client. The web-server responds to the client and makes another record in the access log.

    Thus, you will almost always get another hit in logs after a 301 hit.

I'm missing a lot of nuances here, like CDNs, SSL/TLS handshakes, http2 handshakes, but this is just to give the very basic understanding of how web-server logs work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.