sitemap.xml for huge online store

Question

I'm running online store with 30,000+ products (having unique urls) which are updated every day. I want google to index my products faster. Now crawler index ~500 pages/day (reports from google webmaster tools). Would generating sitemap xml improve indexing? Should I include in sitemap static pages like contact, shipping info etc (I have about 10 such pages)? Does google bot works independently and index pages that are not listed in sitemap?

Answer 1

Would generating sitemap xml improve indexing?

In theory, but there are no promises, either. ("Google doesn't guarantee that we'll crawl or index all of your URLs." etc.) Sitemaps can be helpful for bots to find all your pages, but they're informational; they aren't required to do anything with them. The search engines may decide to disregard some documents for whatever reasons of their own.

Should I include in sitemap static pages

There's no real reason to leave any URLs out.

Does google bot works independently and index pages that are not listed in sitemap?

Yes, all the spiders do. Sitemaps are only there to help them do their jobs, especially for documents that may not be linked yet, or not very well.

Answer 2

It sounds like your type of site was exactly what the Sitemap protocol was created for. I'd say that a sitemap file could do nothing but improve your site's indexing.

Using the <lastmod> and <changefreq> tags should allow search engine crawlers to better determine how often they re-crawl each of your product pages. Worst case scenario would be that these aren't taken into account, and you're back to square one - but you're also no worse for wear either.

Adding your static content is also not a bad idea, and you can use the <changefreq> value to let search engines know that they don't have to check that page every day - so maybe they'll spend more time indexing your product pages.

It's also worth looking into tools like Google Webmaster Tools - Google in particular have a section where you can tweak your crawl settings to see if that helps.