I am working on this site dealsin.us and the Google webmasters tool shows that there are about 9800+ urls blocked by robots.txt. You can view the robots.txt here I have blocked some directories which are not meant for the users and our only to the staff behind the website. I am really confused and would appreciate any help on this.


Webmaster Tools used to show you all the URLs that you have blocked with robots.txt (under Crawl Errors), however that functionality appears to no longer exist. There is only the Crawler Access section that lists how many URLs are blocked.

If your pages are appearing in search results without problems (a quick site: search shows that is the case) then there is probably no need to worry. It's likely stemming from some extra URL parameters somewhere, for example if your 'submit' page has a parameter for every category then all those will show up as blocked.

However, looking at the robots.txt I do notice a few things. First, if I am not mistaken the 'allow' line actually overrides the lines above it! As noted by Ilmari in the comments, it does not override other rules but is simply redundant. You should remove that line since everything is crawled by default.

Second, the 'sitemap' line should be separate from the rest, i.e. have a blank line after it. And the * wildcard after /engine/ does nothing, as robots.txt only matches from the start of the URL anyway.

| improve this answer | |
  • The Allow line is just redundant: while the parsing of Allow rules is somewhat non-standardized, pretty much all robots.txt parsers that support them at all will obey either a) the first or b) the most specific matching rule. As the Allow: / rule is both the last and the least specific, and also equal to the default (which is to allow all URLs), it has no effect. Still, given that it's redundant, I'd recommend removing it, if only to shorten the file by a few bytes. – Ilmari Karonen May 26 '12 at 17:24
  • @Ilmari yes, it looks like that's the case. This goes to show how sparingly robots.txt should be used. If you want to disallow a directory but allow a subdirectory then the site structure could probably be improved. – DisgruntledGoat May 27 '12 at 12:33
  • I am still facing the issue google is still showing 7500+ urls blocked which is hurting the traffic on my website. – Vijay Sharma Jun 2 '12 at 3:54
  • @Vijay how are they hurting your traffic? If there are folders being blocked then remove those folders from robots.txt. – DisgruntledGoat Jun 2 '12 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.