4

Today I got a email notice from Google that there were search problems with my site. I logged into Google Search Console. It shows that 1 file was being blocked by robots.txt, and has been for several days.

I run the Robots check on that file, it says, "There is no robot.txt file" This is what I expected. So the error message claims that ONE file was un-indexable due to a file I don't have.

If I run 'Fetch as Google' it shows what I expect.

I've submitted feedback to Google.

I've checked other answers here, and so far all have talked about formatting/syntax in the robots.txt file. I've not found the combination of getting a URL blocked when there isn't a robots.txt file.

Why just one file? If it was a temporary service issues, why is it still there 5 days later>

Is this one of those things that just happens, and goes away?

  • 3
    'when I run the check, it says, "There is no Robot.txt file"' - did you think you had a robots.txt file? – MrWhite Dec 2 '18 at 23:54
  • 2
    And in addition to MrWhite's question, if you do have one, is it accessible to search engines and placed at the root as expected? – Henry Visotski Dec 3 '18 at 0:06
  • Clarified the quesiton.: I have no robots.txt file, never have had one. – Sherwood Botsford Dec 4 '18 at 14:45
  • Sometimes, such as with WordPress, you may have a virtual robots.txt that you should override with your own. I'd recommend explicitly creating one and allowing everything to all bots; it's better than not having one. Also, try checking with Screaming Frog, see if it finds anything: screamingfrog.co.uk/seo-spider/user-guide/general/#robots-txt – Henry Visotski Dec 4 '18 at 22:02
1

Did you received the message via email or via Search Console?

if it is via email ignore. If message is on search console, try to overwrite it by adding a robots.txt to allow all bots to crawl your website, like for example:

User-agent: *
Disallow:

Also check that no meta robots tags are present (check every page) if the following appears and remove it:

<META NAME="ROBOTS" CONTENT="NOINDEX">

Also check if you are bloking access via X-Robots-Tag vis HTTP Header

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)
  • I received an email, then followed the link to search console. Search console shows the single file error. – Sherwood Botsford Dec 5 '18 at 23:50

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.