4

I am looking at sitemaps of a few large websites like ebay.com and I noticed that the URLs in the sitemap point to search results rather than the actual product page and I have confirmed that all the URLs are pointing to search results of some keywords.

For example this is a sample URL in the sitemap of ebay.com, it points to search results of a set of keywords passed in the URL.

http://www.ebay.com/sch/i.html?_nkw=womens+spenco+sandals+9

Is this a best practice to not to include the actual product pages in the sitemap to reduce the sitemap size? if it is indeed the case then how do the actual product pages get discovered then?

Any guidance understanding this would be greatly apprecitated.

Improve this question

1 Answer 1

Reset to default
2

Links to product searches are often added to sitemaps because they're typically dynamically generated during a search (like on eBay) and usually contain a session ID too.

By providing a crawl-able link to products with keyword included, the search engine bot can index them because they can't perform searches themselves. The only other means to discover dynamic content is to create Snapshots of them.

Sites that rely upon AJAX and database driven content typically do the same.

Sitemap size is not an issue, providing that each one is under 50,000 links and less than 10MB in (compressed gzip) size. It will just take longer to crawl and index the site, and take up more resources in the process. This is done just to help the search bot index product searchrd based on keywords, and not to reduce the size of the sitemaps.

Improve this answer
2
  • I think the fact that most of the websites only have the dynamic urls in their sitemap ebay for example , makes it impossible to have crawler that can rely on sitemaps.xml alone , I was thinking that may be I could use the sitemap.xml to know all the possibly crawl-able pages, but it seems sitemaps.xml is not only not complete but also only includes those URLs that are pointing to search results.Relying on sitemaps.xml alone would not be a workable strategy for a crawler ? Is my understanding correct ?
    – user1965449
    Commented Jun 19, 2013 at 4:03
  • Missed your comment. Adding your URL's to your sitemap, or having them in your content somewhere, and providing snapshots is really all you can do. What some large product sites do is include URL's in their sitemap that display entire product categories (which includes multiple products). That way less direct links are needed. Hope this clarifies things.
    – dan ♦
    Commented Jun 20, 2013 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.