3

Like I said in the title, the Google Webmaster Console claims not to find the robots.txt file of my homepage and therefore it won't crawl it.

But there is a robots.txt file (https://soundhaus-band.de/robots.txt). And Google can read the correct content of the robots.txt in the robots-tester.

I also tried to fetch the robots.txt as Google and then add it to the index. But this does not help either.

Can anyone help me?

Improve this question
6
  • Did you just add the robots.txt file, or has it been there for at least 24 hours already?
    – Stephen Ostermiller ♦
    Nov 7, 2016 at 19:02
  • 1
    Keep in mind that not being able to find the robots.txt file can simply be a network error. This happens from time to time. If you can get to it, then just leave it alone and things will work out in time.
    – closetnoc
    Nov 7, 2016 at 19:45
  • 1
    Do you get this error when visiting the "robots.txt Tester" tool in GSC? Are you by chance looking at the http://... property instead of the https://... one? "fetch the robots.txt as Google" - yeah, don't do that, it's not a file you want indexed.
    – MrWhite
    Nov 7, 2016 at 20:35
  • 1
    Yes, the robot.txt has been there for a week now.
    – sschwaab
    Nov 7, 2016 at 21:12
  • 1
    Is this reported as a permanent crawl error? I've just noticed that for one of my sites, GSC is reporting a crawl error on robots.txt a few days ago (on 01/11/2016). However, it was just a spike on that day (which coincided with a server update).
    – MrWhite
    Nov 7, 2016 at 21:56

2 Answers 2

Reset to default
4

If you're seeing this in the "crawl errors" section as "Inaccessible robots.txt" then that would be due to network or server issues when Google tried to fetch the robots.txt file (eg, server returned a 500 or timed out for the robots.txt). You can check to see if Google has the current version by using the "robots.txt test tool" in Search Console -- that should show the current contents of the file, and the dropdown on top shows you the difference over time.

Improve this answer
0

As much as we all think google is smart at processing everything in a perfect manner, I believe there are still some things it can't do, and I think one of them is establishing a secure connection just to download a plain text file.

What I would suggest trying in order to make every single search engine happy is to make your robots.txt accessible without requiring a secure connection to be established first.

When I tried accessing:

http://soundhaus-band.de/robots.txt

I get a redirect to:

https://soundhaus-band.de/robots.txt

This can throw search engines off because they're looking for robots.txt content only to find a redirect.

You should make http://soundhaus-band.de/robots.txt produce the exact same output as https://soundhaus-band.de/robots.txt to make everyone happy.

If that doesn't work, then send google negative feedback about their robots.txt detection service.

Improve this answer
3
  • Google follows redirects for robots.txt. Not all crawlers are that smart, but Google is very good about it. I wouldn't redirect robots.txt to a custom name, but in this case, when the entire site redirects to HTTPS it makes sense for the robots file to do so as well.
    – Stephen Ostermiller ♦
    Nov 8, 2016 at 9:59
  • The OP has already stated (in comments) that this "error" is reported under the https property in GSC - when https://.../robots.txt is requested directly.
    – MrWhite
    Nov 8, 2016 at 14:23
  • Today google decided, that it can open the robots.txt. After one week... I now will check if it stays so. Thanks and I'll keep you up to date.
    – sschwaab
    Nov 8, 2016 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.