1

I have a website for managing trading card game (TCG) collections which is built around a central search engine/filter system. This filter system is located on the cards page which lists all cards found matching the selected filters in a paginated way. If no filters are selected, all results are listed. Each result links to a detail page of the card.

Each filter is added to the URL as a query string parameter, so there might be millions URLs giving different results. Naturally, I don't want all these querystring based URLs to get indexed. In fact, I don't really care much about the content of the cards page at all. What I care about are the card detail pages to get indexed. Essentially, I want Google (or any other crawler) to index the cards page without querystring and to crawl through the pagination to index the card detail pages.

This would seem possible by excluding the querystring from the cards page in robots.txt, however, almost all backlinks (and internal links) will probably link to a specific card search result (with filters). I guess this will affect my ranking in a negatively?

My best guess to get the card detail pages crawled, would be to include <link rel="prev|next" href="/cards?page=number"> (and add an exception for the "page" querystring parameter). This would work because all cards are listed if no filters are specified.

These are just my thoughts; what would be the most appropriate way to handle this?

Update

If I decide to crawl the pagination list, what happens if a user has a cookie set (for example 48 results per page instead of the default 24)? I know crawlers will always see the same values because they do not set cookies, but does it matter if the user can visit the same URL with different content? Can Google detect this and/or penalize me for this?

  • You can use noindex which is more flexible and has better leverage, but it should also be noted that if the page is useful for users then, to be honest, it should be indexed. – Simon Hayter May 24 '17 at 12:46
  • Yes, the cards page should be indexed, but only when there are no querystring filters specified. – Egon Olieux May 24 '17 at 13:02
  • 1
    Well, ideally you would want to avoid using ?page= and use /cards/1/, /cards/2/ etc. That way you can just robots /cards? altogether. I'd recommend you keep the strings to pages that you don't want to be indexed, that way it's easier to block and administrate, but obviously this is completely optional though, just my 2 cents. – Simon Hayter May 25 '17 at 7:34
  • I agree, however the /cards/1 is the route is already used for the detail pages, so to avoid confusion I think its best to stick to a query string here. – Egon Olieux May 25 '17 at 9:28
1

I would do two things:

  1. Put a <meta name="robots" content="noindex"> tag in each page that is a list of cards. That way Google can crawl the pages and pass link juice through them, but won't include them in the search index.

  2. Create an XML sitemap that has only the card detail pages listed in it. When the pages you want indexed are in your sitemap, Google will treat those as your canonical URLs. It will prioritize crawling and indexing those.

  • A little late to respond, but that's exactly what I did :) – Egon Olieux Oct 6 '17 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.