Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm trying to avoid duplicate content on a website by blocking pagination via robots.txt, but it turns out the URL doesn't have a parameter passed into the URL I can simply block, it's just a number. Example:

/category/sub-category/product/

/category/sub-category/product/1

/category/sub-category/product/2

/category/sub-category/product/3

In the example above, the URLs that end with 1, 2, and 3 is pagination. Any idea on how I can block these? I don't want to just block all numbers, as this is not an ideal solution.

share|improve this question

2 Answers 2

up vote 0 down vote accepted

If you have possible duplicate content with your pagination, the idea is not to block pagination URLs with robots.txt but just defining the rel="canonical" tag for your different URLs.

For your four URLs, define this tag in your <head> section:

<link rel="canonical" href="http://www.example.com/category/sub-category/product/"/>
share|improve this answer
    
I just remembered I can also use the rel next and prev tags. This is a start then thanks. –  Edgar Quintero Oct 15 '13 at 20:36
1  
Yes, you also can use rel="next" and rel="prev". –  Zistoloen Oct 15 '13 at 21:00
2  
I doubt the content on the paginated pages is the same as the content on the 'start' page, so the using the canonical is not the correct thing to do in this case. The canonical tag should be used on pages that have the same, or very similar content, perhaps just changing in order. So if the content on these paginate pages is not the same as the start page, this is an incorrect use of the canonical tag. –  Max Oct 16 '13 at 5:24
1  
Hi, this pagination occurs for a comment box, there is a limit of 5 comments shown per page, therefore the rel next and prev, while perfectly acceptable for normal pagination, isn't ideal for this situation. I feel the rel="canonical" answer is perfect for comment pagination. I didnt' add this info in the question because I feel it needs to be simple and straightforward as possible. Thanks! –  Edgar Quintero Oct 16 '13 at 15:22
1  
If there are comments appearing on the paginated pages that don't appear on the main page that they are being canonicaled too, this is an incorrect use of the canonical tag. Per Google's guidlines "The rel="canonical" attribute should be used only to specify the preferred version of many pages with identical content (although minor differences, such as sort order, are okay)." So if the content is identical to the start page its ok to use, but if these other pages contain extra comments this is incorrect and Google may ignore the canonical tag altogether. –  Max Oct 17 '13 at 4:21

If you wanted to block just the page's listed, this is what you would need to add to your robots.txt:

Disallow: /category/sub-category/product/1$
Disallow: /category/sub-category/product/2$
Disallow: /category/sub-category/product/3$

You could also use robots 'NOINDEX, FOLLOW' meta tag in the head section of the paginated source code. This would still allow Google to crawl these URLs (and still discover links on them), but will stop Google indexing them. If just blocking them via robots.txt, Google will not crawl the URLs any longer, and wont find other links that may be on them.

As you mentioned rel next and prev tags could also be a solution here, but using simply canonical tags would be incorrect if the content differs from one page to another.

share|improve this answer
    
Not an ideal solution for this particular problem. Thanks though. –  Edgar Quintero Oct 16 '13 at 15:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.