4

This might be a stupid question, but I never came across this issue before and were not able to find any definite answers to this on the web:

Our client migrated to HTTPS a few months ago, using their HTTP sites along their HTTPS sites. We told them to 301 redirect their HTTP sites to their corresponding HTTPS sites. So far everything was fine...

UNTIL we got an error essage in Google Webmaster Tools for https://www.example.com/robots.txt:

Googlebot encountered 5429 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 12.9%.

We asked their IT service provider to look into the issue, assuming that they made some mistake by setting the 301 redirect in the .htaccess file. However, they referred back to us stating that redirection for robots.txt might be discouraged by Google generally, see here, and that this might well be the issue. They recommend keeping the HTTP robots.txt with a 200.

I actually have never come across this issue so far. Have you any idea what might be causing the issue?

I figure if we do no longer redirect the HTTP robots.txt file, Googlebot might try to crawl the HTTP versions of the website. Which shouldn't be an issue really if all HTTP versions are 301 redirected to the HTTPS versions properly. It just doesn't feel right ;) I'm interested in fixing the issue (by finding out the cause) more than finding a quick fix.

  • 2
    People do blanket redirects to entire sites without a problem. This is so common, that the tech support statement does not make sense. I would question how they are doing the redirect and get my hands dirty into the code. It is clear that something went wrong and it has nothing to do with the robots.txt file which I assumed was perfectly fine before they made the redirect changes. – closetnoc Mar 5 '15 at 15:56
  • As closetnoc states. The robots.txt file needs to be accessible on HTTPS - it sounds as if it is not? Which part of that linked document are you referencing when "stating that redirection for robots.txt might be discouraged"? – MrWhite Mar 5 '15 at 16:03
  • 1
    The robots.txt is in fact accessible on HTTPS. Which is why I just don't get the problem. – tentakellady Mar 5 '15 at 16:10
  • Were you involved in the code?? If not, get a copy and paste it here. I suspect something is wrong and it is fairly fundamental and simple to fix. Do not trust the hosting company- not always. – closetnoc Mar 5 '15 at 16:17
  • No, I was not involved in the code. I assume you're talking about the .htaccess, right? I'm going to ask them to send it to me and then post it here. – tentakellady Mar 5 '15 at 16:20
3

The best way to determine why Google can't access a page (including robots.txt) is to use the fetch as Google feature in Google Webmaster Tools.

  1. Log into Google Webmaster Tools
  2. Select your site (Make sure you have it registered with the https://)
  3. Navigate to "Crawl" -> "Fetch as Google"
  4. Enter /robots.txt in the text box
  5. Click the "Fetch" button

Google will then tell more detailed information about why it is not able to get your robots.txt file.

3

First, make sure you set your preferred site(s) to HTTPS mode in GWT. This may require you to make a new property and re-verify it.

Now once it's looking for SSL mode, hit the sidebar and nav to "Crawl > robots.txt Tester". You should see a field at the bottom that starts with https://yoursite.com followed by a text box and red "TEST" button.

You should see your robots directives loaded up. Run the test on both an allowed and disallowed page and see what it says. If it still can't access it, yet you can, then you should header test it. Open Chrome inspector then the networking tab. Refresh the page then click the first or second entries to expose headers. You are looking for any fishy looking responses or non-200 codes...there may be a hint there as to why G is not able to enter.

If you find a redirect issue, its all on the shoulders of the "IT service provider". They should be able to correctly route to HTTPS mode....if they can't, I would suggest to the client that they find a new "IT service provider" who is capable of understanding how things work in regards to forwarding.

As a bonus, they should be setting an HSTS header as well. HSTS uses a client side 307 redirect and is more strict/stateful than 301 style redirects. It also has a better ability to mitigate [block] insecure elements.

1

My hypothesis on the 5429 errors is that Google is trying to parse an HTML doc, (i.e., the redirect is working, but ending up in the wrong place). Could be a 404 page, an error page or even the home page.

I had this very problem yesterday where the http://example.com/robots.txt was redirecting to /index.php and then again to my home page due to a dodgy .htaccess.

If that's the case, that means Google likely does follow redirects on the robots.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.