I have a robots.txt with the following in it:

User-agent: *
Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=*
Sitemap: http://tests.compkerworld.com/sitemap.xml

My sitemap.xml, which is auto-generated, has all the URLs even those disallowed using robots.txt.

I have no problem if the search engines crawl all the pages, but I want to know if such an entry will create any kind of conflict for crawlers.

One more question: is the above syntax for disallowing correct?

  • Refer to this in regards to your syntax (you don't need the protocol and root, if placed in the root directory). – dan Feb 11 '14 at 20:09

Apparently Google will not crawl them and will report them as being blocked in Google Webmasters Tools. If that error message, which does not affect your rankings, is annoying to you then you will need to remove those pages from your sitemap.

  • Thanks for the response. The question came into my mind because the number of disallowed pages are 5 times the allowed pages and the disallowed pages are there in sitemap.xml. Just curious to know if it's a good practice or I should think of a way to get clean sitemap.xml. – user3104307 Feb 11 '14 at 19:15
  • If you can find a way to clean up your sitemap that would be good as it prevents your GWT from being cluttered up with unnecessary error messages. – John Conde Feb 11 '14 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.