This question is a continuation for this answer https://stackoverflow.com/questions/2788528/how-to-check-if-googlebot-will-index-a-given-url/2788735#2788735
As was told I did go to the Webmaster Tools and tested contents of my robots.txt file. However this is just giving me the info if that content is good enough or not. However for my scenario I need to test whether disallowing some patterns is being indexed or not. For example I have something like this below in my robots.txt


My understanding is the URLs with word pattern should not crawled, but how do I test this pattern is enforced while indexing the website?

  • From the very post you linked to: "tell you what they crawled, how it rates for certain queries, and what they will crawl and what not.". If you exclude something, it should NOT show up in the "what they will crawl", right?
    – Ken White
    Nov 18 '11 at 18:27
  • I wanted to know list of all the pages that would be indexed on the website URL I gave, so that I can find out if the provided pattern is working as I expected or if I need to change anything or if it doesn't even work. And also the tool doesn't tell me anything except the robots.txt is a success or failure not even the what site it has traversed.
    – Amar Jarubula
    Nov 18 '11 at 18:44

There are a couple of things that might help.

One is to look in Google's Webmaster tools under Diagnostics...Crawl Errors and click on the link Restricted by robots.txt. This helps to determine that what you enter in robots.txt is actually blocking what you expect.

The other thing to check is Google's index. I'll do a search query of the form:

site:yourdomain.com inurl:url text to check

When I've made robots.txt changes with wildcards I tend to worry about matching more than what I want, so I use the above checks to make sure only what I want excluded is actually excluded from the index.

I'm not sure if this answers your question, but hopefully it is close enough :-)

  • While this answer is close enough to what I am thinking, I wanted to test the changes I made in robots.txt without actually placing it in my domain, so that I don't cause any problems. The webmaster tools only tells me if my robots.txt works or not, but not the exact results what would happen. I want to make sure no issues happen. Dec 5 '11 at 15:39
  • I understand your concern. I've felt the same way when making this type of change, but I'm not sure there is a way to test it as thoroughly as we'd like.
    – Nathan Fox
    Dec 5 '11 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.