5

We will launch a project with over 1.000.000 products and those are in approx 150 categories. Later on you will find reviews and technical detail pages to those products. So there will be more than 2.000.000 pages at start and a growing number as soon as reviews are generated.

Content: products, product ratings/reviews, product technical details, later on maybe product questions/answers

Our prior goal is to give Google informations about important content on our portal and where were last changes.

I was a little bit confused when reading some answers to similar questions:

This one says:

The idea of a sitemap is to have this point to all pages.

And this one says:

You should not include any categories, paginated pages or sub categories in your sitemap.xml if there is no unique and distinct content on those pages.

And I thought it's a good point to put product category overview urls into XML Sitemap to give Google a good starting point to crawl my site.


However - my question is if I have to write every existing url into XML Sitemap or just the product urls from where you (or Google) reach reviews or technical details with one click?

And what happens when a product got a new review? Should I listed up the review overview URL to XML Sitemap with last modified time stamp or just give this time stamp to product url and Google will crawl it again + linked pages?

And should I put category overview pages and top product pages into XML Sitemap too?

How often will Google scan our XML Sitemap index file (because we'll have many XML Sitemaps) or will we have to upload them each time last modified time stamp has changed?

4 Answers 4

2

With a site the size of yours and especially since it seems that there are many ways to get to each product page, it makes perfect sense to make a sitemap. Create a sitemap that wraps other sitemaps with no more than 45,000 pages each wrapped sitemap. I realize the limit is 50,000. Just make sure there is headroom for error. Create an entry for each product page. Do not worry about categories. Also include home page, about, contact, privacy policy, and so forth. Include all content pages but not navigational (category) pages.

https://support.google.com/webmasters/answer/183668?hl=en and https://support.google.com/webmasters/answer/71453

Submit the sitemap using the Google Webmaster Tools. Google will read and index your site extremely fast depending upon the size of the sitemap and the speed of your site. For example, I saw 38,000+ fetches a day from Google. You may see more. The fetches will slow down as it reaches the end of your sitemap.

Google will periodically visit all of your pages and any changes will be noticed. Google uses a TTL (time to live) mechanism for each page. This TTL tells Google how often to visit the page, If the page changes often, ie. rating changes and comments, then the page will be fetched more often and indexed more quickly. It takes Google time to figure this out so be patient. No more than a month or two.

This same TTL applies to the sitemap but I warn you only to create a new sitemap when there are updates and not to try and fool Google. Google will check your sitemap file as often as it seems to be updated. Google can/will also do a HEAD request to see if your sitemap file has changed. If you create new sitemaps, also recreate the wrapper sitemap file.

Use your favorite scripting code to create the sitemap files. I created my update mechanism in about 30 minutes. You can ask the neighbors kid for help. ;-) Or solicit a developer for help. It should not cost much more than an hour of time or an X-Box game or two depending.

4
  • I assume that it is not necessary to put category sites into XML Sitemap because they have no real value content? Or is there another reason why do you say to keep them out of those XML Sitemap files?
    – Jurik
    Mar 10, 2014 at 9:41
  • Yes. Exactly. On a site years ago, I had a navigational page that looked like as much as 35,000 pages with parameters. This page had no value and diluted the overall performance of the site when all that was important was the final content pages. However, if your category pages have value or are landing pages then that would be okay to include.
    – closetnoc
    Mar 10, 2014 at 15:55
  • Okay, because I want add additional content/infos to those category pages. Looks like overview pages with "most recent" parts should be interesting too, right?
    – Jurik
    Mar 11, 2014 at 16:29
  • 1
    Right. In my case, these were just navigational pages with links that allowed someone to find what they want. No added value. I would think about adding value to the category pages or create landing pages. It may be your category pages can be landing pages. Sometimes you just have to try things and see what happens. Use your best judgement.
    – closetnoc
    Mar 12, 2014 at 2:54
1

You want to get to build an XML sitemap for all your pages, it will help to search engine to crawl as easy, but one thing search engine crawler will take time to index all of your pages and it might not crawl some pages. so you need to create xml site maps for all your pages if there are content is unique.You should update your sitemap for every weeks or every 3 three days

0

The sitemap is really more for helping Google find urls that are not easily found or possibly dynamic. Google won't scan the entire sitemap if it is too large and deep. I don't think you should worry too much about the sitemap initially. Make sure you have a webmasters account and see what is being indexed.

Also, the sitemap should be created automatically within your system, each time a page is added the sitemap should be updated. Google will check the sitemap relative to your traffic and the sitemap freq tag. It knows if the site is very popular to keep up on the index, it wants to give the best experience to searchers, and that means updated info. Good Luck.

7
  • I thought it could help Google to identify more important pages on my site and crawl less less important pages.
    – Jurik
    Mar 7, 2014 at 11:24
  • If you have a good linking structure on your pages, google will find the content without issue. It really depends on how your pages, are served. Think of it like a meta description. If the page is well defined and the content matches the intent then google will properly index the page with or without the meta description. Also importance is relative to a search term, not the site.
    – Michael
    Mar 7, 2014 at 12:50
  • "... or possibly dynamic" - What do you mean by this?
    – MrWhite
    Mar 7, 2014 at 12:55
  • A non static url. URLs that are generated from the server but don't physically exist in a folder. The url is based on a response from the server.
    – Michael
    Mar 7, 2014 at 14:05
  • The statement The sitemap is really more for helping Google find urls that are not easily found or possibly dynamic. Google won't scan the entire sitemap if it is too large and deep. is simply not true. Sitemaps help Google and Google will read them entirely and spider your site entirely, if and when Google it makes sense for Google to do so. This is often on very large sites that are deep. Google reads sitemaps with 500,000 and 1,000,000 pages and more even when the site can be easily crawled, indexed, and has been using their spider previously.
    – closetnoc
    Mar 7, 2014 at 20:59
0

You want to get to build an XML sitemap for all your pages, it will help to search engine to crawl as easy with crawl boundaries like how many pages are allowed / disallowed for crawling, so you just need to create xml, html and .gz sitemaps for all your pages

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.