I have a robots.txt with the following in it:
User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=* Sitemap: http://tests.compkerworld.com/sitemap.xml
My sitemap.xml, which is auto-generated, has all the URLs even those disallowed using robots.txt.
I have no problem if the search engines crawl all the pages, but I want to know if such an entry will create any kind of conflict for crawlers.
One more question: is the above syntax for disallowing correct?