In the Index Status section of Google Webmaster Tools a site I manage has seen a sudden increase in pages "Blocked from robots".

Index Status - Blocked from robots

There's been no spike in Crawl Errors since this happened a week and a half ago, so how can I best diagnose this? Is there any way to find a list of the blocked URLs? Is there a reason this spiked suddenly?


I cannot see any options in Google Webmaster Tools. Unfortunately, Google does not seem to give a clue. At first I thought that the Blocked URLs and testing your robots.txt file would give details, but I am not seeing it. Then I thought perhaps Index Status data could help but no go.

I looked to other tools like Screaming Frog and it is not clear if it will list issues with URLs and the robots.txt file. There is a whole section for the robots.txt file, but I am not sure how deep it will go.

Sorry. This should be a feature in Google Webmaster Tools. Perhaps the test button should list all know URLs that would be blocked.

  • I know what's in the robots.txt file. I'm just trying to get a list of URLs the Google is suddenly reporting as blocked when it wasn't before. – Andrew Lott Jun 5 '14 at 14:32
  • It may not give you a list of URLs unless you specify files, but it should give you an indication in the form of a directory or some other structure of what Google is talking about. As far as Google is concerned that is as close as you get- that I know of. The blocked from robots should be referring to the robots.txt file. – closetnoc Jun 5 '14 at 14:40
  • I know this means the robots.txt file, but that's had the same directories blocked for months. How can I tell what 800 URLs are being blocked that weren't before? – Andrew Lott Jun 5 '14 at 14:43
  • Does the download chart data button in the index status help? No. Sorry. I am still looking. – closetnoc Jun 5 '14 at 14:49
  • 1
    @dan I just could not believe (somehow) that Google would miss such an opportunity. But then again, I have to remind myself that it is Google we are talking about. They can do something fantastic and forget an obvious detail. But the fantastic is still worth the price of admission. – closetnoc Jun 7 '14 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.