Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a robots.txt with the following in it:

User-agent: *
Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=*
Sitemap: http://tests.compkerworld.com/sitemap.xml

My sitemap.xml, which is auto-generated, has all the URLs even those disallowed using robots.txt.

I have no problem if the search engines crawl all the pages, but I want to know if such an entry will create any kind of conflict for crawlers.

One more question: is the above syntax for disallowing correct?

share|improve this question

migrated from stackoverflow.com Feb 11 '14 at 18:55

This question came from our site for professional and enthusiast programmers.

Refer to this in regards to your syntax (you don't need the protocol and root, if placed in the root directory). – dan Feb 11 '14 at 20:09

Apparently Google will not crawl them and will report them as being blocked in Google Webmasters Tools. If that error message, which does not affect your rankings, is annoying to you then you will need to remove those pages from your sitemap.

share|improve this answer
Thanks for the response. The question came into my mind because the number of disallowed pages are 5 times the allowed pages and the disallowed pages are there in sitemap.xml. Just curious to know if it's a good practice or I should think of a way to get clean sitemap.xml. – user3104307 Feb 11 '14 at 19:15
If you can find a way to clean up your sitemap that would be good as it prevents your GWT from being cluttered up with unnecessary error messages. – John Conde Feb 11 '14 at 19:19

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.