4

Today I got a email notice from Google that there were search problems with my site. I logged into Google Search Console. It shows that 1 file was being blocked by robots.txt, and has been for several days.

I run the Robots check on that file, it says, "There is no robot.txt file" This is what I expected. So the error message claims that ONE file was un-indexable due to a file I don't have.

If I run 'Fetch as Google' it shows what I expect.

I've submitted feedback to Google.

I've checked other answers here, and so far all have talked about formatting/syntax in the robots.txt file. I've not found the combination of getting a URL blocked when there isn't a robots.txt file.

Why just one file? If it was a temporary service issues, why is it still there 5 days later>

Is this one of those things that just happens, and goes away?

4
  • 3
    'when I run the check, it says, "There is no Robot.txt file"' - did you think you had a robots.txt file?
    – MrWhite
    Dec 2, 2018 at 23:54
  • 2
    And in addition to MrWhite's question, if you do have one, is it accessible to search engines and placed at the root as expected? Dec 3, 2018 at 0:06
  • Clarified the quesiton.: I have no robots.txt file, never have had one. Dec 4, 2018 at 14:45
  • Sometimes, such as with WordPress, you may have a virtual robots.txt that you should override with your own. I'd recommend explicitly creating one and allowing everything to all bots; it's better than not having one. Also, try checking with Screaming Frog, see if it finds anything: screamingfrog.co.uk/seo-spider/user-guide/general/#robots-txt Dec 4, 2018 at 22:02

2 Answers 2

2

Did you received the message via email or via Search Console?

if it is via email ignore. If message is on search console, try to overwrite it by adding a robots.txt to allow all bots to crawl your website, like for example:

User-agent: *
Disallow:

Also check that no meta robots tags are present (check every page) if the following appears and remove it:

<META NAME="ROBOTS" CONTENT="NOINDEX">

Also check if you are bloking access via X-Robots-Tag vis HTTP Header

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)
1
  • I received an email, then followed the link to search console. Search console shows the single file error. Dec 5, 2018 at 23:50
2

My bad:

My httpd configuration redirects 403 errors to a custom error page, but apparently doesn't report it as an error.

Since robots.txt doesn't exist, httpd dutifully did what it was supposed to and returned my custom error page instead.

Hopefully this helps someone else.

1
  • Thank you for the follow up! I learned something new
    – Raul Reyes
    Aug 29, 2019 at 7:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.