Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my robots.txt file.

I know some people may frown on the use of wild cards etc, but my web framework generates dynamic URLs - so there is no otherway of 'pattern matching' a set of Urls.

I have included both my robots.tx and a list of the ignored URLs here. Hopefully, someone can spot the rule(s) that are restricting Googlebots access to the URLs.

First of all, here is a (truncated) list of restricted URLs:

URL Detail Detected
http://www.example.com/community/groups/cat-lovers-united.html?filter=events URL restricted by robots.txt 10/4/10
http://www.example.com/community/knowledge-center/questions/newbie/1/how-to-love-your-cat.html URL restricted by robots.txt 10/4/10
http://www.example.com/community/news/comments/1 URL restricted by robots.txt 10/4/10
http://www.example.com/feeds/blogposts?request_type=comments&id=2 URL restricted by robots.txt 10/4/10
http://www.example.com/news/106/related-headlines?page=1 URL restricted by robots.txt 10/2/10
http://www.example.com/news/search?category=agriculturals&keywords=argentina URL restricted by robots.txt 10/6/10
http://www.example.com/news/submissions/list?site=ibnlive.in.com URL restricted by robots.txt 9/30/10

Here is my robots.txt file:

User-Agent: *
Disallow: /api
Disallow: /api/
Disallow: /logout
Disallow: /logout/
Disallow: /login
Disallow: /login/
Disallow: /registration
Disallow: /registration/
Disallow: /admin/*
Disallow: /news/submit
Disallow: /*/new
Disallow: /*/createnew
Disallow: /*/edit
Disallow: /*/edit/*
Disallow: /*/editprofile
Disallow: /*/delete
Disallow: /*/delete/*
Disallow: /*/update
Disallow: /*/update/
Disallow: /*/submit
Disallow: /*/submit/
Disallow: /*/report
Disallow: /*/report/*
Disallow: /search
Disallow: /search/
Disallow: /controllers/*
Disallow: /community/members/*/joinNetwork
Disallow: /community/admin/*
Disallow: /lostpassword
Disallow: /lostpassword/
Disallow: /registrationComplete
Disallow: /registrationComplete/
Disallow: /accountActivate
Disallow: /accountActivate/
Disallow: /generate-token
Disallow: /create-widget
Disallow: /buttons
Disallow: /accountDeleted
Disallow: /accountSuspended
Disallow: /news/submit
Disallow: /news/process-submission
Disallow: /news/submit/success
Disallow: /news/errorpage
Disallow: /community/members/*/mailinglist/add
Disallow: /community/members/*/editprofile
Disallow: /community/members/*/joinNetwork
Disallow: /community/users/*/follow
Disallow: /community/users/*/unfollow
Disallow: /trustnetwork/createtoken
Disallow: /trustnetwork/applytoken
Disallow: /community/members/*/mail/*
Disallow: /community/mail/*
Disallow: /community/blogs/blogposts/unpublished/*
Disallow: /community/blogs/blogposts/reported/*
Disallow: /tools/pipeline/vote-submitted.html
Disallow: /community/classifieds/replies/*
Disallow: /report
Disallow: /thumbvote
Disallow: /like
Disallow: /dislike
Disallow: /voteRemove
Disallow: /flagAsInteresting
Disallow: /addFavorite
Disallow: /recommend/*
Disallow: /completed-recommendation.html
Disallow: /rate
Disallow: /notify
Disallow: /addTestimonial
Disallow: /comments/update/*
Disallow: /comments/great-comment
Disallow: /community/members/admin/*
Disallow: /community/blogs/blogposts/comments/moderated/*
Disallow: /community/blogs/blogposts/comments/reported/*
Disallow: /community/articles/reported/fetch/*
Disallow: /community/articles/reported/comments/moderated/*
Disallow: /community/articles/comments/reported/*
Disallow: /community/groups/*/requests
Disallow: /community/groups/*/moderation
Disallow: /community/groups/*/headlineMgmt
Disallow: /community/groups/*/pollMgmt
Disallow: /community/groups/*/join
Disallow: /community/groups/*/leave
Disallow: /community/groups/*/shout
Disallow: /community/groups/*/topics/reply/*
Disallow: /community/groups/*/banUser
Disallow: /countries/cities
Disallow: /country/schools
Disallow: /community/knowledge-center/answers/accept
Disallow: /community/knowledge-center/answers/unAccept
Disallow: /requestPassword
Disallow: /restorePassword
Disallow: /restoredPassword
Disallow: /changePassword
Disallow: /changedPassword
Disallow: /adverts/replies/mailbox
Disallow: /security/error403
Disallow: /home/error404
Disallow: /home/disabled-notification
Disallow: /home/error500
Disallow: /*?
Allow: /

Sitemap: http://www.example.com/sitemap-index.xml

It looks ok to me, but obviously, there must be something in the robots.txt which is restricting access to quite a few pages. I will be very grateful if someone can spot the offending lines in my robots.txt file

share|improve this question
1  
Actually, I have just spotted the last Dissallow pattern /*? I think that is what is preventing a lot of the URLs with params?. Not sure, but would like to get the opinion of someone who actually knows more about allowed syntax in robots.txt files –  morpheous Oct 11 '10 at 13:04
1  
For understanding, article-stack.com/education/web-design-n-development/… –  articlestack Oct 11 '10 at 15:36
    
I also have a same problem. I've restricted some urls of my website using robots.txt. But why it's showing crawl errors (restricted by robots.txt). Why this is an error? Why this shows up in crawl error section? I want to remove the errors, but want to keep the restricted url as it was. Is there any way to do that? –  user10135 Sep 8 '11 at 8:37
add comment

2 Answers

See you answered your own question, but as you point out, "Disallow: /*?" is the source of your problems. The "*" is a regex, or regular express pattern, which basically means any STRING of text of ANY length, "*?" means the same thing, but limits the pattern to the SHORTEST possible answer, and in the case of robots.txt, I can't think of a way that the "*?" expression would have any meaning.

Reading ROBOTS.TXT

    The Disallow line lists the pages you want to block.
    The User-Agent line lists the crawlers you want to block.

ERRORS_IN_YOUR_ROBOTS.TXT

    (1) All the use of * in "/*/" may or may not correctly used, all the "Disallow: /INSERT_XYZ/*" are wrong, all you need is "Disallow: /INSERT_XYZ/"
    (2) "Disallow: /*?" should be "Disallow: /" since the reference is to directories, not agents; with "User-Agent: *" that's correct, and "User-Agent: /" would be wrong. But since you want your site crawled in part, remove it.
    (3) All the "Disallow: /INSERT_XYZ" should likely be "Disallow: /INSERT_XYZ/" if they're a reference to a directory.

Google's webpage for webmasters for Robots.txt is here.

NOTE: You should also Google these meta-tags: noindex, nofollow, noarchive, nocache

share|improve this answer
1  
Or use the x-robots-tag in place of the meta tags. –  John Conde Oct 11 '10 at 13:56
    
thanks for the feedback. I have made some of the changes you recommended - I will wait till Googlebot crawls the site again to see which errors have gone away. BTW, I am not sure your reference to directory structures is entirely relevant here, as the urls are being generated by a URL router - i.e. they do not reflect the actual file structure on the server. –  morpheous Oct 11 '10 at 21:40
    
@morpheous: "Good" bots are looking at both directory structures, links and final URLs rendered -- and comparing it to the robots dot text. Robots.txt is not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there no guarantee that all current and future robots will use it. Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots. –  blunders Oct 11 '10 at 22:02
add comment

This is my experience with crawl errol by robots.txt. A few months ago i've found the message like "URL restricted by robots.txt", my site is Blogger (*.blogspot.com). I ask my problem to Google Webmaster and they send me the answer about my robots txt as you can read :

  • rayhanzhampiet.blogspot.com :

"Google was unable to crawl the URL due to a robots.txt restriction: your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located; or it might prohibit access to the URL specifically. Often, this is not an error" can anybody tell me how can i cancel the robots.txt restriction?

  • Top Contributor Webmaster Help :

Hey rayhanzhampiet, i guess we are dealing with your blog on blogger*com (blogspot*com)? If so you don't need to do anything about robots.txt restrictions.,

This is your robots.txt: User-agent: Mediapartners-Google Disallow:

User-agent: * Disallow: /search -- si only restricts duplicated stuff that not even you want to have indexed keep cool, see all URLs restricted have /search/ in their path as specified in your robots.txt?

They are restricted because they duplicate what is already seen in posts and on the homepage, for example here:

http://rayhanzhampiet.blogspot.com/search/label/Tips-Blogging You recognize having seen all this before --> on your original posts which are not restricted and thus indexed ;-)

/search/ has to be restricted to avoid duplication in the index, and, as I said, you even want to restrict them for robots for your own sake. This restriction is done by default on blogspot*com (and domains that use blogger like yours, of course) and does not need to be resolved in any way, it's not an error (messages in webmastertools are diagnostic only here).

I hope the people (blogger) who have a problem with robots.txt like me will known that they dont need to be resolved.

Thanks to Pro Webmasters,

share|improve this answer
add comment

protected by Community Sep 8 '11 at 11:32

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?