Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

This question is a continuation for this answer How to check if googlebot will index a given url?
As was told I did go to the Webmaster Tools and tested contents of my robots.txt file. However this is just giving me the info if that content is good enough or not. However for my scenario I need to test whether disallowing some patterns is being indexed or not. For example I have something like this below in my robots.txt

disallow:/pattern*     

My understanding is the URLs with word pattern should not crawled, but how do I test this pattern is enforced while indexing the website?

share|improve this question

migrated from stackoverflow.com Nov 20 '11 at 1:06

This question came from our site for professional and enthusiast programmers.

    
From the very post you linked to: "tell you what they crawled, how it rates for certain queries, and what they will crawl and what not.". If you exclude something, it should NOT show up in the "what they will crawl", right? –  Ken White Nov 18 '11 at 18:27
    
I wanted to know list of all the pages that would be indexed on the website URL I gave, so that I can find out if the provided pattern is working as I expected or if I need to change anything or if it doesn't even work. And also the tool doesn't tell me anything except the robots.txt is a success or failure not even the what site it has traversed. –  Amar Jarubula Nov 18 '11 at 18:44

1 Answer 1

There are a couple of things that might help.

One is to look in Google's Webmaster tools under Diagnostics...Crawl Errors and click on the link Restricted by robots.txt. This helps to determine that what you enter in robots.txt is actually blocking what you expect.

The other thing to check is Google's index. I'll do a search query of the form:

site:yourdomain.com inurl:url text to check

When I've made robots.txt changes with wildcards I tend to worry about matching more than what I want, so I use the above checks to make sure only what I want excluded is actually excluded from the index.

I'm not sure if this answers your question, but hopefully it is close enough :-)

share|improve this answer
    
While this answer is close enough to what I am thinking, I wanted to test the changes I made in robots.txt without actually placing it in my domain, so that I don't cause any problems. The webmaster tools only tells me if my robots.txt works or not, but not the exact results what would happen. I want to make sure no issues happen. –  Amar Jarubula Dec 5 '11 at 15:39
    
I understand your concern. I've felt the same way when making this type of change, but I'm not sure there is a way to test it as thoroughly as we'd like. –  Nathan Fox Dec 5 '11 at 16:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.