Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a website with over 1 million url's. As such, I have 1 sitemap.xml file which acts as an index to many sitemap0000.xml.gz, sitemap0001.xml.gz ... sitemap0030.xml.gz etc. files.

My question is: In my robots.txt file how do I specify the sitemap file?

Do I refernce the sitemap.xml only? As in the following:

User-agent: *
Allow: /
Sitemap: www.mysite.com/sitemap/sitemap.xml

Or do I have to specify ALL the *.xml.gz files in the sitemap too?

share|improve this question
3  
You need to include the protocol ("http://") when specifying the Sitemap URL in the robots.txt file. –  John Mueller May 15 '12 at 18:49

1 Answer 1

up vote 5 down vote accepted

Yes, only refer to the index file. (See under the "Using robots.txt" heading.) It will then lead the crawler to the others.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.