2

Recently I realized Disallow /was set on my staging sites.

I didn't realize until I got a warning in Search Console...

Mistakes happen

I've removed the directive and cleared all caches at least 10 times.

However, when I use the URL inspection tool I'm still getting the following:

Google Search Console URL Inspection Tool showing that a page is blocked by robots.txt

Never experienced this before - and I've been doing SEO for a long time. Usually I'd see this reflected in a live test right away.

Anyone experienced this before? Is there some sort of processing time or cache on Google's end? Shouldn't be if the "live test" is actually live.

I typically don disallow a staging/development site from Google. Additional thoughts:

Rather than disallow, I've always had no problem setting meta robots to noindex follow and listing the main site as the canonical URL for all pages. This way Google is able to crawl the staging site and understand what we're doing so the effect is:

"Ah ok gotcha, this is staging, I'll ignore it."

Would love to hear thoughts on this, or alternative approaches.

Improve this question
4
  • 2
    How long has it been? Googlebot only fetches robots.txt once every 24 hours.
    – Stephen Ostermiller ♦
    Commented Aug 24, 2021 at 19:01
  • I disagree about staging sites. Robots.txt disallow is a fine way to handle staging sites. If Google can't crawl it, it usually wont index it. Occasionally Google will index a URL it can't crawl if it has inbound links, but that shouldn't be the case for a staging site. Even then the staging site would usually only appear in site: searches.
    – Stephen Ostermiller ♦
    Commented Aug 24, 2021 at 19:03
  • I usually recommend password protecting staging sites. If Googlebot sees that authentication is required, it won't index it, nor will it waste time crawling lots of pages from it.
    – Stephen Ostermiller ♦
    Commented Aug 24, 2021 at 19:04
  • @StephenOstermiller "Googlebot only fetches robots.txt once every 24 hours." - perfect thanks, I never thought to look into that. You're not wrong, I think all of the methods are valid. I've never had issues with the method I described, but I have had content indexed that is blocked. Ex: Two of my top performing posts got indexed on staging despite robots.txt blocking them. To set the record straight I want to show google that noindex and canonical. Password protection is definitely the most foolproof tho...I don't have an excuse there lol
    – Mike Ciffone
    Commented Aug 24, 2021 at 20:37

1 Answer 1

Reset to default
2

Googlebot only fetches robots.txt once every 24 hours. After making a robots.txt change, you need to wait a day before testing. Changes won't take effect right away.

Google says:

Caching

Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn't possible (for example, due to timeouts or 5xx errors). The cached response may be shared by different crawlers. Google may increase or decrease the cache lifetime based on max-age Cache-Control HTTP headers.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.