Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

On a site that has a robots.txt file that disallows all crawling...

User-agent: *
Disallow: /

...does it matter what the sitemap.xml content is? Is it better, or does it make a difference at all, if requests to sitemap.xml return a 404, a 403, or an empty XML file?

share|improve this question

2 Answers 2

up vote 4 down vote accepted

From https://support.google.com/webmasters/answer/156184?hl=en

Sitemaps are a way to tell Google about pages on your site we might not otherwise discover.

This means you don't need a sitemap if you don't want Google to discover anything. Having a sitemap won't do anything so I wouldn't make one at all.

share|improve this answer
1  
Yep, just to add further clarification to this answer. A sitemap is only required if a crawler would have difficulty crawling the website - typically these can either be poorly constructed websites or websites that possess a deep and complex level of hierarchy. –  zigojacko Jan 22 at 15:11

In contrast to robots.txt, the sitemap doesn’t have a fixed name/location.

You could name it "s" instead of "sitemap.xml". And it doesn’t have to be placed in the host root. See http://www.sitemaps.org/protocol.html#location.

So it’s likely that search engines wouldn’t try to guess if there is a sitemap (or, if they would try it, they can’t expect to find something). There are various ways to inform search engines about your sitemap location.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.