On a site that has a robots.txt file that disallows all crawling...

User-agent: *
Disallow: /

...does it matter what the sitemap.xml content is? Is it better, or does it make a difference at all, if requests to sitemap.xml return a 404, a 403, or an empty XML file?


From https://support.google.com/webmasters/answer/156184?hl=en

Sitemaps are a way to tell Google about pages on your site we might not otherwise discover.

This means you don't need a sitemap if you don't want Google to discover anything. Having a sitemap won't do anything so I wouldn't make one at all.

| improve this answer | |
  • 1
    Yep, just to add further clarification to this answer. A sitemap is only required if a crawler would have difficulty crawling the website - typically these can either be poorly constructed websites or websites that possess a deep and complex level of hierarchy. – zigojacko Jan 22 '14 at 15:11

In contrast to robots.txt, the sitemap doesn’t have a fixed name/location.

You could name it "s" instead of "sitemap.xml". And it doesn’t have to be placed in the host root. See http://www.sitemaps.org/protocol.html#location.

So it’s likely that search engines wouldn’t try to guess if there is a sitemap (or, if they would try it, they can’t expect to find something). There are various ways to inform search engines about your sitemap location.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.