Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

Question

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my robots.txt file.

I know some people may frown on the use of wild cards etc, but my web framework generates dynamic URLs - so there is no otherway of 'pattern matching' a set of Urls.

I have included both my robots.tx and a list of the ignored URLs here. Hopefully, someone can spot the rule(s) that are restricting Googlebots access to the URLs.

First of all, here is a (truncated) list of restricted URLs:

URL Detail Detected
http://www.example.com/community/groups/cat-lovers-united.html?filter=events URL restricted by robots.txt 10/4/10
http://www.example.com/community/knowledge-center/questions/newbie/1/how-to-love-your-cat.html URL restricted by robots.txt 10/4/10
http://www.example.com/community/news/comments/1 URL restricted by robots.txt 10/4/10
http://www.example.com/feeds/blogposts?request_type=comments&id=2 URL restricted by robots.txt 10/4/10
http://www.example.com/news/106/related-headlines?page=1 URL restricted by robots.txt 10/2/10
http://www.example.com/news/search?category=agriculturals&keywords=argentina URL restricted by robots.txt 10/6/10
http://www.example.com/news/submissions/list?site=ibnlive.in.com URL restricted by robots.txt 9/30/10

Here is my robots.txt file:

User-Agent: *
Disallow: /api
Disallow: /api/
Disallow: /logout
Disallow: /logout/
Disallow: /login
Disallow: /login/
Disallow: /registration
Disallow: /registration/
Disallow: /admin/*
Disallow: /news/submit
Disallow: /*/new
Disallow: /*/createnew
Disallow: /*/edit
Disallow: /*/edit/*
Disallow: /*/editprofile
Disallow: /*/delete
Disallow: /*/delete/*
Disallow: /*/update
Disallow: /*/update/
Disallow: /*/submit
Disallow: /*/submit/
Disallow: /*/report
Disallow: /*/report/*
Disallow: /search
Disallow: /search/
Disallow: /controllers/*
Disallow: /community/members/*/joinNetwork
Disallow: /community/admin/*
Disallow: /lostpassword
Disallow: /lostpassword/
Disallow: /registrationComplete
Disallow: /registrationComplete/
Disallow: /accountActivate
Disallow: /accountActivate/
Disallow: /generate-token
Disallow: /create-widget
Disallow: /buttons
Disallow: /accountDeleted
Disallow: /accountSuspended
Disallow: /news/submit
Disallow: /news/process-submission
Disallow: /news/submit/success
Disallow: /news/errorpage
Disallow: /community/members/*/mailinglist/add
Disallow: /community/members/*/editprofile
Disallow: /community/members/*/joinNetwork
Disallow: /community/users/*/follow
Disallow: /community/users/*/unfollow
Disallow: /trustnetwork/createtoken
Disallow: /trustnetwork/applytoken
Disallow: /community/members/*/mail/*
Disallow: /community/mail/*
Disallow: /community/blogs/blogposts/unpublished/*
Disallow: /community/blogs/blogposts/reported/*
Disallow: /tools/pipeline/vote-submitted.html
Disallow: /community/classifieds/replies/*
Disallow: /report
Disallow: /thumbvote
Disallow: /like
Disallow: /dislike
Disallow: /voteRemove
Disallow: /flagAsInteresting
Disallow: /addFavorite
Disallow: /recommend/*
Disallow: /completed-recommendation.html
Disallow: /rate
Disallow: /notify
Disallow: /addTestimonial
Disallow: /comments/update/*
Disallow: /comments/great-comment
Disallow: /community/members/admin/*
Disallow: /community/blogs/blogposts/comments/moderated/*
Disallow: /community/blogs/blogposts/comments/reported/*
Disallow: /community/articles/reported/fetch/*
Disallow: /community/articles/reported/comments/moderated/*
Disallow: /community/articles/comments/reported/*
Disallow: /community/groups/*/requests
Disallow: /community/groups/*/moderation
Disallow: /community/groups/*/headlineMgmt
Disallow: /community/groups/*/pollMgmt
Disallow: /community/groups/*/join
Disallow: /community/groups/*/leave
Disallow: /community/groups/*/shout
Disallow: /community/groups/*/topics/reply/*
Disallow: /community/groups/*/banUser
Disallow: /countries/cities
Disallow: /country/schools
Disallow: /community/knowledge-center/answers/accept
Disallow: /community/knowledge-center/answers/unAccept
Disallow: /requestPassword
Disallow: /restorePassword
Disallow: /restoredPassword
Disallow: /changePassword
Disallow: /changedPassword
Disallow: /adverts/replies/mailbox
Disallow: /security/error403
Disallow: /home/error404
Disallow: /home/disabled-notification
Disallow: /home/error500
Disallow: /*?
Allow: /

Sitemap: http://www.example.com/sitemap-index.xml

It looks ok to me, but obviously, there must be something in the robots.txt which is restricting access to quite a few pages. I will be very grateful if someone can spot the offending lines in my robots.txt file

Answer 1

See you answered your own question, but as you point out, "Disallow: /*?" is the source of your problems. The "*" is a regex, or regular express pattern, which basically means any STRING of text of ANY length, "*?" means the same thing, but limits the pattern to the SHORTEST possible answer, and in the case of robots.txt, I can't think of a way that the "*?" expression would have any meaning.

Reading ROBOTS.TXT

The Disallow line lists the pages you want to block.

The User-Agent line lists the crawlers you want to block.

ERRORS_IN_YOUR_ROBOTS.TXT

(1) All the use of * in "/*/" may or may not correctly used, all the "Disallow: /INSERT_XYZ/*" are wrong, all you need is "Disallow: /INSERT_XYZ/"

(2) "Disallow: /*?" should be "Disallow: /" since the reference is to directories, not agents; with "User-Agent: *" that's correct, and "User-Agent: /" would be wrong. But since you want your site crawled in part, remove it.

(3) All the "Disallow: /INSERT_XYZ" should likely be "Disallow: /INSERT_XYZ/" if they're a reference to a directory.

Google's webpage for webmasters for Robots.txt is here.

NOTE: You should also Google these meta-tags: noindex, nofollow, noarchive, nocache

Answer 2

This is my experience with crawl errol by robots.txt. A few months ago i've found the message like "URL restricted by robots.txt", my site is Blogger (*.blogspot.com). I ask my problem to Google Webmaster and they send me the answer about my robots txt as you can read :

• rayhanzhampiet.blogspot.com :

"Google was unable to crawl the URL due to a robots.txt restriction: your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located; or it might prohibit access to the URL specifically. Often, this is not an error" can anybody tell me how can i cancel the robots.txt restriction?

• Top Contributor Webmaster Help :

Hey rayhanzhampiet, i guess we are dealing with your blog on blogger*com (blogspot*com)? If so you don't need to do anything about robots.txt restrictions.,

This is your robots.txt: User-agent: Mediapartners-Google Disallow:

User-agent: * Disallow: /search -- si only restricts duplicated stuff that not even you want to have indexed keep cool, see all URLs restricted have /search/ in their path as specified in your robots.txt?

They are restricted because they duplicate what is already seen in posts and on the homepage, for example here:

http://rayhanzhampiet.blogspot.com/search/label/Tips-Blogging You recognize having seen all this before --> on your original posts which are not restricted and thus indexed ;-)

/search/ has to be restricted to avoid duplication in the index, and, as I said, you even want to restrict them for robots for your own sake. This restriction is done by default on blogspot*com (and domains that use blogger like yours, of course) and does not need to be resolved in any way, it's not an error (messages in webmastertools are diagnostic only here).

I hope the people (blogger) who have a problem with robots.txt like me will known that they dont need to be resolved.

Thanks to Pro Webmasters,