Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I've been looking at most important sites and none of them have a sitemap.xml defined. My list includes:

  • Amazon
  • Ebay
  • Yahoo
  • Microsoft
  • ... and others

But none of them include a sitemap.xml file. Why is this? It's not that important?

Note: I've been using the Google Bot user agent to check it. The only site that returned a 200 Ok was google.com

share|improve this question

closed as not constructive by John Conde Jun 4 '13 at 0:28

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

    
What? This is not constructive? Haven't you read the important information described by Stephen Ostermiller in his answer? This sucks. –  Santiago Basulto Jun 4 '13 at 1:03
    
Not sure why this has been closed, its quite an interesting questions and very good answer. –  Max Jun 4 '13 at 2:00
add comment

1 Answer 1

Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap.

  1. Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature.
  2. You can specify the name of the sitemap file in robots.txt. It doesn't have to be sitemap.xml. A robots.txt can have a line like this that says where to get the sitemap. Sitemap: http://example.com/mysitemapfile.xml
  3. It is also possible to create a sitemap file at a "secret" URL. You can tell Google about this URL through webmaster tools. That way Google will get your sitemap, but it won't be easily accessible to the public.

Looking in robots.txt, many of the sites you list do have sitemaps.

Amazon

Sitemap: http://www.amazon.com/sitemap-manual-index.xml
Sitemap: http://www.amazon.com/sitemap_vendor_videos_us.xml
Sitemap: http://www.amazon.com/sitemap_vod_index.xml
Sitemap: http://www.amazon.com/sitemaps.f3053414d236e84.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.1946f6b8171de60.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.bbb7d657c7e29fa.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.11aafed315ee654.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.c21f969b5f03d33.SitemapIndex_0.xml.gz

Ebay

Sitemap: http://www.ebay.com/lst/PDP_US_main_index.xml
Sitemap: http://www.ebay.com/lst/PDP_US_incremental_index.xml
Sitemap: http://www.ebay.com/lst/VIP_US_index.xml
Sitemap: http://www.ebay.com/lst/SRP_US_index.xml

Microsoft

...
Sitemap: http://www.microsoft.com/en-us/dynamics/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/kinectforwindows/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/sqlazure/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/cloud/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/server-cloud/sitemap.xml
...
share|improve this answer
    
Ahh!! You're right. I'm so dumb. Thank you. I'll keep checking. –  Santiago Basulto Jun 4 '13 at 0:35
    
I don't know why it was closed as not constructive. Your answer is awesome. I've learned from it, and I'm sure more people could do it too. –  Santiago Basulto Jun 4 '13 at 1:04
    
Closed doesn't mean this question will disappear, or that you did anything wrong. The way you asked this questions sounded philosophical, which can lead to answers with extended discussion and disagreement. This site is not built to handle extended discussion and disagreement. –  Stephen Ostermiller Jun 4 '13 at 2:11
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.