My robots.txt contains the line Disallow: /search.php.

Shall I nevertheless include http://example.com/search.php into my sitemap?

1 Answer 1


You should not add URLs to your sitemap unless you want them crawled and indexed. If a URL is disallowed, non-canonical, or redirecting, it doesn't belong in your XML sitemap.

Google uses your XML sitemap as a list of pages that it should crawl. When you include non-crawlable pages in your XML sitemap Google will complain about them in Google Search Console. You would get the warning "Sitemap Contains Blocked Resources."

For further reading see my answer to How should I choose which URLs I want indexed to include in my sitemap?

You are correct to disallow search because Google doesn't want to index other search results. It is bad user experience for users to click from Google search result to your site search results. Google may penalize sites that allow their site search results to be crawled and indexed. See Matt Cutts: Search results in search results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.