Google Search Console says "submitted URL blocked by robots.txt" even though there is no robots.txt file

Question

Today I got a email notice from Google that there were search problems with my site. I logged into Google Search Console. It shows that 1 file was being blocked by robots.txt, and has been for several days.

I run the Robots check on that file, it says, "There is no robot.txt file" This is what I expected. So the error message claims that ONE file was un-indexable due to a file I don't have.

If I run 'Fetch as Google' it shows what I expect.

I've submitted feedback to Google.

I've checked other answers here, and so far all have talked about formatting/syntax in the robots.txt file. I've not found the combination of getting a URL blocked when there isn't a robots.txt file.

Why just one file? If it was a temporary service issues, why is it still there 5 days later>

Is this one of those things that just happens, and goes away?

Answer 1

Did you received the message via email or via Search Console?

if it is via email ignore. If message is on search console, try to overwrite it by adding a robots.txt to allow all bots to crawl your website, like for example:

User-agent: *
Disallow:

Also check that no meta robots tags are present (check every page) if the following appears and remove it:

<META NAME="ROBOTS" CONTENT="NOINDEX">

Also check if you are bloking access via X-Robots-Tag vis HTTP Header

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)

Answer 2

My bad:

My httpd configuration redirects 403 errors to a custom error page, but apparently doesn't report it as an error.

Since robots.txt doesn't exist, httpd dutifully did what it was supposed to and returned my custom error page instead.

Hopefully this helps someone else.