7

I've been looking at most important sites and none of them have a sitemap.xml defined. My list includes:

  • Amazon
  • Ebay
  • Yahoo
  • Microsoft
  • ... and others

But none of them include a sitemap.xml file. Why is this? It's not that important?

Note: I've been using the Google Bot user agent to check it. The only site that returned a 200 Ok was google.com

Improve this question
2
  • 1
    What? This is not constructive? Haven't you read the important information described by Stephen Ostermiller in his answer? This sucks.
    – Santiago Basulto
    Jun 4 '13 at 1:03
  • 1
    Not sure why this has been closed, its quite an interesting questions and very good answer.
    – Max
    Jun 4 '13 at 2:00
15

Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap.

  1. Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature.
  2. You can specify the name of the sitemap file in robots.txt. It doesn't have to be sitemap.xml. A robots.txt can have a line like this that says where to get the sitemap. Sitemap: http://example.com/mysitemapfile.xml
  3. It is also possible to create a sitemap file at a "secret" URL. You can tell Google about this URL through webmaster tools. That way Google will get your sitemap, but it won't be easily accessible to the public.

Looking in robots.txt, many of the sites you list do have sitemaps.

Amazon

Sitemap: http://www.amazon.com/sitemap-manual-index.xml
Sitemap: http://www.amazon.com/sitemap_vendor_videos_us.xml
Sitemap: http://www.amazon.com/sitemap_vod_index.xml
Sitemap: http://www.amazon.com/sitemaps.f3053414d236e84.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.1946f6b8171de60.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.bbb7d657c7e29fa.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.11aafed315ee654.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.c21f969b5f03d33.SitemapIndex_0.xml.gz

Ebay

Sitemap: http://www.ebay.com/lst/PDP_US_main_index.xml
Sitemap: http://www.ebay.com/lst/PDP_US_incremental_index.xml
Sitemap: http://www.ebay.com/lst/VIP_US_index.xml
Sitemap: http://www.ebay.com/lst/SRP_US_index.xml

Microsoft

...
Sitemap: http://www.microsoft.com/en-us/dynamics/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/kinectforwindows/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/sqlazure/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/cloud/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/server-cloud/sitemap.xml
...
Improve this answer
3
  • Ahh!! You're right. I'm so dumb. Thank you. I'll keep checking.
    – Santiago Basulto
    Jun 4 '13 at 0:35
  • 1
    I don't know why it was closed as not constructive. Your answer is awesome. I've learned from it, and I'm sure more people could do it too.
    – Santiago Basulto
    Jun 4 '13 at 1:04
  • Closed doesn't mean this question will disappear, or that you did anything wrong. The way you asked this questions sounded philosophical, which can lead to answers with extended discussion and disagreement. This site is not built to handle extended discussion and disagreement.
    – Stephen Ostermiller ♦
    Jun 4 '13 at 2:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.