1

I'm wondering the best way to handle this situation. I'm about to release a new section on a website where there are literally hundreds of millions of URL combinations that can possibly be crawled and indexed by search engines. This is because I'm giving the users the option to choose their refinements using checkboxes, similar to Zappos, and there are tons of possible combinations.

The website currently has about 100k URLs indexed by Google but releasing this new section of the website will increase that number by a very large magnitude. This could be incorrect, but I've heard websites may be penalized for having a huge sudden increase in crawled pages. Is this true? If so what's the best way to handle this? My goal isn't to "pagerank sculpt" or anything like that, but I don't want to be penalized for any changes I'm about to make.

I thought about adding a "disallow" directive to my robots.txt file, but that will block out the indexing of all of the new pages which I don't want to happen. I think it's best to have the most important ones indexed (one or two filter combinations), but I don't think it's really necessary to have the extremely long-tail combinations indexed. So I guess my question is: is there any good way of doing this or is it best to just let the spiders to their thing?

|improve this question

migrated from stackoverflow.com Sep 20 '11 at 18:55

This question came from our site for professional and enthusiast programmers.

  • This was probably the first problem ever solved by search engines. Just let them do their thing. – Greg Hewgill Sep 19 '11 at 20:32
  • Somewhat related prior question, just for cross-ref purposes. And, Jeff, was this you also? I thought this question seemed familiar. – Su' Sep 20 '11 at 19:40
  • I'm not the person who asked that question, but thanks for linking me to it. I actually know about and implement most strategies discussed by the accepted answer in other websites I manage. However, for usability purposes i do want to create all the combinations this person suggests not to. – Jeff Sep 21 '11 at 4:31
1

I see your point about the URLs being shareable in response to John's answer, and that's understandable, as a user benefit. But here you're talking about the search engines, so: Is there maybe a case for applying noindex to these granular comparison pages, or past some threshold like three selected features?

While single-product pages, and maybe some higher-level categories or whatever are obviously important to have indexed, having every single permutation of features indexed would seem like it could just dilute the site's effectiveness in results and maybe even end up dropping users on them rather than just the product they're looking for.

|improve this answer
  • Agree that noindex might be the best plan of attack for very granular search refinements. It's unfortunate that the search engines will still have to request the entire page before seeing the noindex meta tag. – Jeff Sep 21 '11 at 4:51
  • And I do know about nofollow links. Problem is I'm planning on using javascript for these and that probably can't be utilized since search engines now parse basic javascript links/redirects. – Jeff Sep 21 '11 at 5:04
4

If the same content can be reached by more then one URL then you will definitely have issues with duplicate content. You will need to specify a canonical URL for each page so Google knows which one is the "main" one and will show that URL in its search results. All other URLs that pull up the same content will be seen as the same page (which means any links to them count as a link to the main page).

There's no real problem with adding that much content at once. Just don't expect to see all of it indexed quickly. Also don't expect it to rank well right away.

Note: If this is customized content that requires a user to login then search engines will not see it as they do not create accounts nor login. So they do not have access to these URLs.

Also, if this is custom content based on user preferences, why not store the preferences in a session and avoid having mangled URLs?

|improve this answer
  • Thanks for the comment. For the most part I don't think duplicate content will be an issue. Re: sessions and mangled urls...the main reason is so the pages can be shared/linked to. I have managed sites in the past that use session-based browse/searching and for the most part I'm against it for this reason. I'm also curious about how to deal with pagination from an SEO standpoint. Is it good practice to use canonical tags for all results pages past page 1? For example, should the URL site.com/results.php?p=8 have a canonical tag pointing to site.com/results.php – Jeff Sep 20 '11 at 1:50
  • 1
    @John Conde is right about duplicate content - if you're creating additional page combinations as part of a refinement process then you are showing the same content - albiet a subset. In terms of pagination, Google has just posted two "best practice" articles on managing that issue: first second – Mike Hudson Sep 20 '11 at 2:15
  • Yes, I want it to be a refinement process and the results will be subsets. The same result will definitely appear on multiple pages using this process, but the combinations on each page and browse results will, for the most part, be different than all other pages/results. For example if we're talking cars, there would ideally be a browse page for "Lexus," and one for "Lexus IS 250s," and one for "Lexus IS 250, IS 350s, and IS Fs," etc. Just like how Zappos does it. For the most part I don't think that's a big duplicate content issue in the eyes of search engines, right? – Jeff Sep 21 '11 at 4:41
  • You have to look at how Google defines "duplicate content" which has morphed a little recently (Panda). They define it as "substantive blocks of content within or across domains that either completely match other content or are appreciably similar." (Emphasis mine). In your case, its that you're diluting the value of your content by splitting into refinements. – Mike Hudson Sep 21 '11 at 6:03
0

It all depends on what kind of content each pay will display. Google Panda (Google's ranking algorithm) will evaluate each page of a site, thus affecting the overall rank of the site.

There is a lot of dispute over how to make your site work best for Panda. But it is generally understood that Panda will rank a site lower if it comes across pages it believes are auto-generated, or generally spammy. For example, say you have www.siteinfo.xyz, and you can put any domain in (like www.siteinfo.xyz/stackoverflow.com) and the site will simply output that domain's page title, what JS engine they use, if they use Google Analytics, etc. So, each page of www.siteinfo.xyz is auto-generated and therefor each pay isn't very unique. Panda will then rank the site lower for each of these pages it finds.

There was an update to Panda recently that is said to have fixed some of the problems where valid sites were down-ranked.

All in all, Panda try to rank sites based on how valuable and valid the content is. So, as long as the pages that your site has offer good content it should be okay. To be safe it would be best to make a sitemap so Google better understands the flow.

Also check out Google's quality guidlines.

For more on panda see this.

|improve this answer
  • 3
    "Panda" is the nickname for part of Google's ranking algorithm. Specifically the part that looks for low quality pages and websites. It isn't the name of Google's ranking algorithm itself. There also isn't much of a dispute about what works best: offer quality content. – John Conde♦ Sep 19 '11 at 20:50
  • My bad O_o ... I'm not the best at SEO, but Panda quality checks should apply in this case. Not to say it's limited to that, though – Marshall Sep 19 '11 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.