2

I have an interactive page on my website that produces millions of combinations. Users can share a link to the specific combination.

Something like: example.com/tool?id1=blah&id2=blahagain

This changes a few images and words on the page. There also some more combinations that make more complex changes to the page structure, but overall it’s all the exact same tool.

I want google to mostly index the main page for the tool: example.com/tool - But I also don’t want to hurt my SEO with a potential 1 million + separate pages. The number of combinations increases over time too.

  • What is the value of any particular combination? Is it something that lots of people want to see? Does it have unique content? – Stephen Ostermiller Sep 12 at 19:40
  • Each combination is of interest I guess. The page “fuses” two characters into one and gives a image that combines both images and changes colors. Then gives a unique character description for that fusion. So the idea is if you get a particular funny or interesting combination, that has potential to get shared. – Incognito Sep 12 at 19:50
  • Can you give an example of a unique character description? How many words is it? Is it something that is automatically generated, or does somebody have to write it? – Stephen Ostermiller Sep 12 at 19:52
  • Sorry late replies, running errands. Everything is automatically generated including the descriptions. There’s base data for about 1,000 things. Then a lot of inputs the users can do to create millions of combinations. – Incognito Sep 12 at 20:18
  • I would create another page that leads to this image generation script, and tell google to not index sub pages, but only the main page. For sharing you generate the url, and the url's work, but the search engine just does not index those, only the main page. – LPChip Sep 12 at 20:49
1

If i correctly understand your goals:

  • on the one side you want to give your visitors a possibility to share a particular page url, which is created by special query string combination,
  • on the other side you don't want to index the whole gazillion of such query-string-ed urls.

If so, i think the only method to handle the stuff is a so-called PRG pattern. Read a bit further about this approach - the implementation isn't very tricky, but it is out of my experience the only way to handle this topic.

1

I would probably use robots.txt as is very simple and straight forward.

Now you could find a way to canonicalize all new generated pages (depending on the extend of on-page changes that the tool makes).

Another way would be to noindex all the generated new pages from the tool.

The 2 above cases (canonical, noindex) have 2 disadvantages in your case in my opinion, first you could end up having crawl budged issues and ending up Googlebot missing your important pages. Second depending on CMS and experience you will need to automate the process of including noindex directive for example on each case.

Robots.txt is the best solution for long term. Will you have some pages still indexed? Yes, but still wont be much i've seen sites managing their millions of combination with robots.txt and having only 50-100 of them indexed.

This is what you would need if your tool is sitting inside a directory:

Allow: /*/tool$
Disallow: /*/tool/*

And this if your tool is sitting in the root:

Allow: /tool$
Disallow: /tool/*

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.