3

My team will go online with a portal that will have over 1.000.000 products. And I read here following text:

Prioritize which content you have indexed. With that much data, not all of it is going to be absolutely vital. Make a strategic decision as to what content is most important, e.g., it will be most popular, it has the best chance at ROI, it will be the most useful, etc. and make sure that that content is indexed first.

Another SEO expert told me to pick out our top 50 categories (including all products) and to put them into our XML Sitemap for Google.

Our site map for users gives already deeplinks into those categories and manufacturers too.

And if we put our top 50 categories into our XML Sitmap, should we:

a) just increase the number of categories as soon as google got almost everything crawled? b) just take out categories that do not work and put in others that work better? So that Google Bot is focused on our top 50 categories (with approx 250.000 Products)

As far as I know Google indexes everything, as long as there's no no-index set - but a XML Sitemap will help Google to find our important pages faster - is this correct?

Improve this question

3 Answers 3

Reset to default
5

a) just increase the number of categories as soon as google got almost everything crawled?

b) just take out categories that do not work and put in others that work better? So that Google Bot is focused on our top 50 categories (with approx 250.000 Products)

Not placing pages in your sitemap doesn't hide them from Google as they can find them by naturally crawling your site. So this won't spoon feed Google the pages you want indexed.

If you want to give to certain categories more weight in Google's eyes, give them more weight on your website. You can do this by linking to them more frequently and prominently in your website. Linking to these categories from your homepage or any other page that is high in the structural hierarchy of your website will give them higher precedence to Google as links from those pages will be stronger then links from deeper inner pages.

As far as I know Google indexes everything, as long as there's no no-index set - but a XML Sitemap will help Google to find our important pages faster - is this correct?

No. Sitemaps tell search engines where to find your content. It does not give weight or priority to what is to be crawled (despite their being a priority flag in the sitemap).

Improve this answer
4
  • So if I put my top 50 to XML Sitemap and the rest of my content is just crawlable by internal links, google still won't find and index my XML Sitemap content faster than rest of my content?
    – Jurik
    Commented Mar 6, 2014 at 14:20
  • 2
    More or less. Sitemaps aren't all they're cracked up to be.
    – John Conde ♦
    Commented Mar 6, 2014 at 14:21
  • 1
    See The Sitemap Paradox for more information about the limitations of sitemaps in regards to crawling and indexing.
    – Stephen Ostermiller ♦
    Commented Mar 6, 2014 at 16:27
  • 1
    Putting the priority content into a sitemap will let you measure how you are doing at getting that content indexed. Google Webmaster Tools will show you the number of pages out of that sitemap that are indexed.
    – Stephen Ostermiller ♦
    Commented Mar 6, 2014 at 16:30
1

John's answer is very good. I agree that the best way to get your priority content indexed is to feature it prominently on your site.

The other tool that you have at your disposal is robots.txt. You could use robots.txt to prevent Google from crawling large sections of lower priority content when you first launch the site. Later, when the site is better established, you could open up these areas for crawling as well.

Improve this answer
2
  • And to go sure, I should use no-follow on each link that leads into the not so important areas, right?
    – Jurik
    Commented Mar 6, 2014 at 16:44
  • 1
    There is no need to use rel=nofollow on any internal links on your site. Based on my experimentation, Google treats links that point to content that is in robots.txt very similarly to nofollowed links, even when the nofollow is not used.
    – Stephen Ostermiller ♦
    Commented Mar 6, 2014 at 18:11
1

From the GWT you can "Add to Index" upto 500 important URLs in a month and Google will crawl them in a day or two. I guess you can add site portions like sub domains and folders as separate sites and again use 500 URL submission limit for each.

The rest will be picked up by the sitemap and Google will crawl at it's own pace. The other part have already been answered by experts above.

Improve this answer
2
  • If I put those 500 URLs to "Add to Index" - will I increase the speed of indexing sub-URLs (deeplinks) of those 500 URLs? So those 500 URLs could be overviews of important products or a list of important categories where when I click deeper, come to my important products.
    – Jurik
    Commented Mar 6, 2014 at 16:52
  • I guess yes. Google Is more likely to crawl the linked pages from such URL's than from the Sitemap.
    – AgA
    Commented Mar 7, 2014 at 4:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.