2

Is that the way to go if I want to visually see what the googlebot sees? I'm trying to check a folder which I have just blocked in my robots.txt.

  • If I fetch the folder as googlebot, it fetches ok, so that doesn't tell me nothing about whether the block is working
  • I know there's a tool to check for blocking, but it is dependent on the input of the robots.txt

Therefore I've tried the Instant Preview, and I don't get a preview for what the bot sees ("pre-render), so I think that means that it's because the robots.txt blocks it; however - I don't see the bot tried beforehand to access my updated robots.txt, so I'm not sure how does it know that this folder is blocked? (it does preview another new folder, that is not blocked)

1

If you followed https://support.google.com/webmasters/answer/156449?hl=en then you should be fine. The robot.txt should block those areas properly. Wait a couple of days to a week or so then check Webmaster Tools (sign up if you haven't already) and it will show you how many pages were blocked by robots.txt and you will see those pages vanish from Google Search with time.

Also if you have a sitemap that you submit to Google or even a HTML sitemap, don't have the blocked pages listed on there as this can lead to those pages being indexed even though they are blocked by robots.txt

Even if you block things by robots.txt it does not guarantee that search engines won't index it.

1

I believe instant preview in Webmaster tools does take in account of robots.txt as the faqs for Instant Preview state that an error would be generated if blocked.

Roboted: The resource is being disallowed from being fetched through a robots.txt directive for Googlebot.

https://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews#14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.