Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm running online store with 30,000+ products (having unique urls) which are updated every day. I want google to index my products faster. Now crawler index ~500 pages/day (reports from google webmaster tools). Would generating sitemap xml improve indexing? Should I include in sitemap static pages like contact, shipping info etc (I have about 10 such pages)? Does google bot works independently and index pages that are not listed in sitemap?

share|improve this question
add comment

2 Answers

Would generating sitemap xml improve indexing?

In theory, but there are no promises, either. ("Google doesn't guarantee that we'll crawl or index all of your URLs." etc.) Sitemaps can be helpful for bots to find all your pages, but they're informational; they aren't required to do anything with them. The search engines may decide to disregard some documents for whatever reasons of their own.

Should I include in sitemap static pages

There's no real reason to leave any URLs out.

Does google bot works independently and index pages that are not listed in sitemap?

Yes, all the spiders do. Sitemaps are only there to help them do their jobs, especially for documents that may not be linked yet, or not very well.

share|improve this answer
add comment

It sounds like your type of site was exactly what the Sitemap protocol was created for. I'd say that a sitemap file could do nothing but improve your site's indexing.

Using the <lastmod> and <changefreq> tags should allow search engine crawlers to better determine how often they re-crawl each of your product pages. Worst case scenario would be that these aren't taken into account, and you're back to square one - but you're also no worse for wear either.

Adding your static content is also not a bad idea, and you can use the <changefreq> value to let search engines know that they don't have to check that page every day - so maybe they'll spend more time indexing your product pages.

It's also worth looking into tools like Google Webmaster Tools - Google in particular have a section where you can tweak your crawl settings to see if that helps.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.