2

I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt file, the most recently added website fails to index, displaying the error message "Page cannot be indexed: Blocked by robots.txt" when inspected using the Google Search Console URL inspection tool.

The robots.txt content:

User-agent: *
Allow: /

The error I get: enter image description here

I am seeking information from others who may have experienced this issue and whether it could potentially be attributed to a cache problem on Google's end as clicking the retest URL results in the same error. It is worth noting that the domain had a previous website prior to the implementation of the new site, and all other websites on the server return valid results without issue. Additionally, various robots.txt checkers have confirmed that the site is not blocked, indicating that the error message is unique to Google's tool.

7
  • How long has it been since you added this website? Mar 31, 2023 at 7:08
  • 1
    Why are you using Allow: / rather than the standard Disallow: (disallow nothing) as specified by the offical robots.txt spec at robotstxt.org/robotstxt.html under the heading "To allow all robots complete access" Mar 31, 2023 at 7:10
  • Is that the entirety of your robots.txt file? Aside from it not being a standard way of allowing all crawlers as Stephen mentioned, I don't see any reason those directives should cause googlebot to be blocked. Mar 31, 2023 at 8:20
  • @StephenOstermiller It's been more than 48 hours since I've added it. I've always used that, and even after I've changed it to "Disallow:" the error still persists
    – Clipso
    Mar 31, 2023 at 13:29
  • @MaximillianLaumeister Yes that the entirety of the robots.txt file
    – Clipso
    Mar 31, 2023 at 13:31

1 Answer 1

2

Google Search Console does indeed cache robots.txt files. I have personal experience of it doing this.

However, Google allows you to see exactly how they view your robots.txt file and whether they are caching an out of date version, using their robots.txt tester tool (part of Google Search Console).

https://www.google.com/webmasters/tools/robots-testing-tool

Be aware, this does not work for domain properties, so if your website is already added as a domain property, you'll have to add it as a standard property as well.

You can trigger Google to fetch the live version of the file, and test variations of your rules, and directly establish if Google considers itself blocked from any URLs within your property.

Hitting the submit button beneath the tester's textarea (where you can edit the robots.txt rules) brings up this dialogue, allowing you to require Google update their cached robots.txt. GSC Robots.txt tester modal with instructions to request recaching of robots.txt file

8
  • Thank you for sharing your information. I appreciate your insight on the caching solution as it appears to be the most viable option. I have attempted to add a property to my account, but the tool displays a screen with the message "Don't see your property? You can add it to your account." When I attempt to add a property, I am redirected to a page where I am only able to view my existing properties or verify them. Despite watching instructional videos, I have not been able to resolve this issue and have decided to discontinue using the tool.
    – Clipso
    Mar 31, 2023 at 13:36
  • Once in Google Search Console, click the hamburger (three horizontal lines) icon in the top left, then the select property drop down, then Add Property. Enter your website address as a "URL prefix" property (use the URL as you want it to appear and it should then be available through the robots.txt tester. Mar 31, 2023 at 13:45
  • Thank you for providing a solution to an issue that I have been grappling with for quite some time. With your assistance, I am now able to use the tester effectively. However, I must note that the approach taken by Google is somewhat cryptic, and it would be beneficial for them to provide clearer guidance on this matter. It appears that the tester is utilizing cached data, which may explain the error I encountered. I am curious if there are any methods to force the use of updated data, or if I should simply wait for Google to refresh its cache before conducting further tests?
    – Clipso
    Mar 31, 2023 at 13:54
  • You can use the URL inspection tool and perform a "live" test there that will force it test the file again by downloading it. I don't think it will cause a re-download of robots.txt though. Mar 31, 2023 at 14:08
  • You can certainly see the last time Google accessed your robots.txt file. If you hit the submit button below the text area (even without changing anything), Google gives you the option to ask them to update what they have cached. I do heartily agree that Google's UX is awful. Mar 31, 2023 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.