5

I have a website with over 1 million url's. As such, I have 1 sitemap.xml file which acts as an index to many sitemap0000.xml.gz, sitemap0001.xml.gz ... sitemap0030.xml.gz etc. files.

My question is: In my robots.txt file how do I specify the sitemap file?

Do I refernce the sitemap.xml only? As in the following:

User-agent: *
Allow: /
Sitemap: www.mysite.com/sitemap/sitemap.xml

Or do I have to specify ALL the *.xml.gz files in the sitemap too?

|improve this question
  • 3
    You need to include the protocol ("http://") when specifying the Sitemap URL in the robots.txt file. – John Mueller May 15 '12 at 18:49
5

Yes, only refer to the index file. (See under the "Using robots.txt" heading.) It will then lead the crawler to the others.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.