2

I'm trying to avoid duplicate content on a website by blocking pagination via robots.txt, but it turns out the URL doesn't have a parameter passed into the URL I can simply block, it's just a number. Example:

/category/sub-category/product/

/category/sub-category/product/1

/category/sub-category/product/2

/category/sub-category/product/3

In the example above, the URLs that end with 1, 2, and 3 is pagination. Any idea on how I can block these? I don't want to just block all numbers, as this is not an ideal solution.

2 Answers 2

0

If you have possible duplicate content with your pagination, the idea is not to block pagination URLs with robots.txt but just defining the rel="canonical" tag for your different URLs.

For your four URLs, define this tag in your <head> section:

<link rel="canonical" href="http://www.example.com/category/sub-category/product/"/>
8
  • I just remembered I can also use the rel next and prev tags. This is a start then thanks. Oct 15, 2013 at 20:36
  • 1
    Yes, you also can use rel="next" and rel="prev".
    – Zistoloen
    Oct 15, 2013 at 21:00
  • 2
    I doubt the content on the paginated pages is the same as the content on the 'start' page, so the using the canonical is not the correct thing to do in this case. The canonical tag should be used on pages that have the same, or very similar content, perhaps just changing in order. So if the content on these paginate pages is not the same as the start page, this is an incorrect use of the canonical tag.
    – Max
    Oct 16, 2013 at 5:24
  • 1
    Hi, this pagination occurs for a comment box, there is a limit of 5 comments shown per page, therefore the rel next and prev, while perfectly acceptable for normal pagination, isn't ideal for this situation. I feel the rel="canonical" answer is perfect for comment pagination. I didnt' add this info in the question because I feel it needs to be simple and straightforward as possible. Thanks! Oct 16, 2013 at 15:22
  • 1
    If there are comments appearing on the paginated pages that don't appear on the main page that they are being canonicaled too, this is an incorrect use of the canonical tag. Per Google's guidlines "The rel="canonical" attribute should be used only to specify the preferred version of many pages with identical content (although minor differences, such as sort order, are okay)." So if the content is identical to the start page its ok to use, but if these other pages contain extra comments this is incorrect and Google may ignore the canonical tag altogether.
    – Max
    Oct 17, 2013 at 4:21
-2

If you wanted to block just the page's listed, this is what you would need to add to your robots.txt:

Disallow: /category/sub-category/product/1$
Disallow: /category/sub-category/product/2$
Disallow: /category/sub-category/product/3$

You could also use robots 'NOINDEX, FOLLOW' meta tag in the head section of the paginated source code. This would still allow Google to crawl these URLs (and still discover links on them), but will stop Google indexing them. If just blocking them via robots.txt, Google will not crawl the URLs any longer, and wont find other links that may be on them.

As you mentioned rel next and prev tags could also be a solution here, but using simply canonical tags would be incorrect if the content differs from one page to another.

1
  • Not an ideal solution for this particular problem. Thanks though. Oct 16, 2013 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.