1

I have a robots.txt with the following in it: 

User-agent: *
Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=*
Sitemap: http://tests.compkerworld.com/sitemap.xml

My sitemap.xml, which is auto-generated, has all the URLs even those disallowed using robots.txt.

I have no problem if the search engines crawl all the pages, but I want to know if such an entry will create any kind of conflict for crawlers.

One more question: is the above syntax for disallowing correct?

Improve this question
1
  • Refer to this in regards to your syntax (you don't need the protocol and root, if placed in the root directory).
    – dan ♦
    Feb 11, 2014 at 20:09

1 Answer 1

Reset to default
1

Apparently Google will not crawl them and will report them as being blocked in Google Webmasters Tools. If that error message, which does not affect your rankings, is annoying to you then you will need to remove those pages from your sitemap.

Improve this answer
2
  • Thanks for the response. The question came into my mind because the number of disallowed pages are 5 times the allowed pages and the disallowed pages are there in sitemap.xml. Just curious to know if it's a good practice or I should think of a way to get clean sitemap.xml.
    – Barun Kumar
    Feb 11, 2014 at 19:15
  • If you can find a way to clean up your sitemap that would be good as it prevents your GWT from being cluttered up with unnecessary error messages.
    – John Conde ♦
    Feb 11, 2014 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.