2

I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt file, the most recently added website fails to index, displaying the error message "Page cannot be indexed: Blocked by robots.txt" when inspected using the Google Search Console URL inspection tool.

The robots.txt content:

User-agent: *
Allow: /

The error I get: enter image description here

I am seeking information from others who may have experienced this issue and whether it could potentially be attributed to a cache problem on Google's end as clicking the retest URL results in the same error. It is worth noting that the domain had a previous website prior to the implementation of the new site, and all other websites on the server return valid results without issue. Additionally, various robots.txt checkers have confirmed that the site is not blocked, indicating that the error message is unique to Google's tool.

Improve this question
7
  • How long has it been since you added this website?
    – Stephen Ostermiller ♦
    Mar 31 at 7:08
  • 1
    Why are you using Allow: / rather than the standard Disallow: (disallow nothing) as specified by the offical robots.txt spec at robotstxt.org/robotstxt.html under the heading "To allow all robots complete access"
    – Stephen Ostermiller ♦
    Mar 31 at 7:10
  • Is that the entirety of your robots.txt file? Aside from it not being a standard way of allowing all crawlers as Stephen mentioned, I don't see any reason those directives should cause googlebot to be blocked.
    – Maximillian Laumeister ♦
    Mar 31 at 8:20
  • @StephenOstermiller It's been more than 48 hours since I've added it. I've always used that, and even after I've changed it to "Disallow:" the error still persists
    – Clipso
    Mar 31 at 13:29
  • @MaximillianLaumeister Yes that the entirety of the robots.txt file
    – Clipso
    Mar 31 at 13:31

1 Answer 1

Reset to default
2

Google Search Console does indeed cache robots.txt files. I have personal experience of it doing this.

However, Google allows you to see exactly how they view your robots.txt file and whether they are caching an out of date version, using their robots.txt tester tool (part of Google Search Console).

https://www.google.com/webmasters/tools/robots-testing-tool

Be aware, this does not work for domain properties, so if your website is already added as a domain property, you'll have to add it as a standard property as well.

You can trigger Google to fetch the live version of the file, and test variations of your rules, and directly establish if Google considers itself blocked from any URLs within your property.

Hitting the submit button beneath the tester's textarea (where you can edit the robots.txt rules) brings up this dialogue, allowing you to require Google update their cached robots.txt. GSC Robots.txt tester modal with instructions to request recaching of robots.txt file

Improve this answer
8
  • Thank you for sharing your information. I appreciate your insight on the caching solution as it appears to be the most viable option. I have attempted to add a property to my account, but the tool displays a screen with the message "Don't see your property? You can add it to your account." When I attempt to add a property, I am redirected to a page where I am only able to view my existing properties or verify them. Despite watching instructional videos, I have not been able to resolve this issue and have decided to discontinue using the tool.
    – Clipso
    Mar 31 at 13:36
  • Once in Google Search Console, click the hamburger (three horizontal lines) icon in the top left, then the select property drop down, then Add Property. Enter your website address as a "URL prefix" property (use the URL as you want it to appear and it should then be available through the robots.txt tester.
    – GeoffAtkins
    Mar 31 at 13:45
  • Thank you for providing a solution to an issue that I have been grappling with for quite some time. With your assistance, I am now able to use the tester effectively. However, I must note that the approach taken by Google is somewhat cryptic, and it would be beneficial for them to provide clearer guidance on this matter. It appears that the tester is utilizing cached data, which may explain the error I encountered. I am curious if there are any methods to force the use of updated data, or if I should simply wait for Google to refresh its cache before conducting further tests?
    – Clipso
    Mar 31 at 13:54
  • You can use the URL inspection tool and perform a "live" test there that will force it test the file again by downloading it. I don't think it will cause a re-download of robots.txt though.
    – Stephen Ostermiller ♦
    Mar 31 at 14:08
  • You can certainly see the last time Google accessed your robots.txt file. If you hit the submit button below the text area (even without changing anything), Google gives you the option to ask them to update what they have cached. I do heartily agree that Google's UX is awful.
    – GeoffAtkins
    Mar 31 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.