Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a website with over 1 million url's. As such, I have 1 sitemap.xml file which acts as an index to many sitemap0000.xml.gz, sitemap0001.xml.gz ... sitemap0030.xml.gz etc. files.

My question is: In my robots.txt file how do I specify the sitemap file?

Do I refernce the sitemap.xml only? As in the following:

User-agent: *
Allow: /
Sitemap: www.mysite.com/sitemap/sitemap.xml

Or do I have to specify ALL the *.xml.gz files in the sitemap too?

share|improve this question
You need to include the protocol ("http://") when specifying the Sitemap URL in the robots.txt file. – John Mueller May 15 '12 at 18:49
up vote 5 down vote accepted

Yes, only refer to the index file. (See under the "Using robots.txt" heading.) It will then lead the crawler to the others.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.