I've got a rule in my
# Crawlers Setup User-agent: * # Directories Disallow: /my_directory/
Yet I am getting entries in my
Crawl Errors section of Google Webmaster Tools for this URL:
Googlebot couldn't access the contents of this URL because the server had an internal error when trying to process the request. These errors tend to be with the server itself, not with the request. More info.
I've even tested manually entering the URL in the
robots.txt Tester in Google Webmaster Tools and it returned "Blocked" as expected for the exact URL that is being reported as an error in Crawl Errors.
How can I resolve this?