Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

On a site that has a robots.txt file that disallows all crawling...

User-agent: *
Disallow: /

...does it matter what the sitemap.xml content is? Is it better, or does it make a difference at all, if requests to sitemap.xml return a 404, a 403, or an empty XML file?

share|improve this question
up vote 4 down vote accepted

From https://support.google.com/webmasters/answer/156184?hl=en

Sitemaps are a way to tell Google about pages on your site we might not otherwise discover.

This means you don't need a sitemap if you don't want Google to discover anything. Having a sitemap won't do anything so I wouldn't make one at all.

share|improve this answer
Yep, just to add further clarification to this answer. A sitemap is only required if a crawler would have difficulty crawling the website - typically these can either be poorly constructed websites or websites that possess a deep and complex level of hierarchy. – zigojacko Jan 22 '14 at 15:11

In contrast to robots.txt, the sitemap doesn’t have a fixed name/location.

You could name it "s" instead of "sitemap.xml". And it doesn’t have to be placed in the host root. See http://www.sitemaps.org/protocol.html#location.

So it’s likely that search engines wouldn’t try to guess if there is a sitemap (or, if they would try it, they can’t expect to find something). There are various ways to inform search engines about your sitemap location.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.