4

I own a web site which has mainly two types of pages:

  • Content pages: they are pretty much static, and link to the previous and next content page (chronologically). The structure is that of a double linked list.

  • Listing pages: they are dynamic, and are paginated. The first one shows the most recent content. Older content is paginated via a GET parameter (e.g ?page=4). Every listing page links to several content pages.

Google is indexing some of the listing pages, which will most likely be outdated when the users find them (new content is added at a pretty fast rate). This is undesirable, of course. Also, listing pages have extracts of the content, which could be seen as repeated content from Google's point of view (which I think is also undesirable, I'm unsure on this).

The problem is, there are hundreds of thousands of content pages. Listing pages are very convenient both for visitors and for search engines to crawl the site. I am unsure if the linked list of content pages will be sufficient to allow search engines to crawl the site in a short time.

I have thought about the following options:

  • Add Disallow: /?page* to robots.txt: listing pages (other than the first one) won't be accessed.
  • Add meta nofollow to listing pages: similar to the first one, listing pages won't be accessed.
  • Add meta noindex to listing pages: they will be accesses, but not added to the index
  • Leave things as they are: eventually search engines will realize the 'page' parameter is for pagination, and won't index it.

What are your suggestions? Adding the NoIndex meta to listing pages sounds like the best solution to me, but I don't dare approach such a large task without asking first :)

Any other possible solutions I didn't think of?

Improve this question

1 Answer 1

Reset to default
1

I'd say one of the most common practices hear would be adding noindex to the listing pages, apart from the first one, so as you said they will still get crawled but will not get added to the index. I think this is preferable to blocking them in robots.txt as Google is still free to crawl through the site and can still follow links on the pages.

Adding nofollow wouldn't be much use especially if the pages are already indexed - they will remain indexed and if they aren't indexed they will still most likely get indexed in Google if they are linked to else where. Matt Cutts also mentions that you should try and avoid nofollowing internal links Should I use rel="nofollow" on internal links

There is another option, which is close to your 'let it be' solution, which is using rel=”next” and rel=”prev” html tags to indicate the relationship between component URLs in a paginated series.This helps Google accurately index your content and serve to users the most relevant page (commonly the first page). A brief explanation below:

http://www.example.com/article?story=abc&page=1
http://www.example.com/article?story=abc&page=2
http://www.example.com/article?story=abc&page=3
http://www.example.com/article?story=abc&page=4

On the first page, http://www.example.com/article?story=abc&page=1, you’d include in the <head> section:

<link rel="next" href="http://www.example.com/article?story=abc&page=2" />

On the second page, http://www.example.com/article?story=abc&page=2:

<link rel="prev" href="http://www.example.com/article?story=abc&page=1" />
<link rel="next" href="http://www.example.com/article?story=abc&page=3" />

On the third page, http://www.example.com/article?story=abc&page=3:

<link rel="prev" href="http://www.example.com/article?story=abc&page=2" />
<link rel="next" href="http://www.example.com/article?story=abc&page=4" />

And on the last page, http://www.example.com/article?story=abc&page=4:

<link rel="prev" href="http://www.example.com/article?story=abc&page=3" />

More info can be found in Google guidelines Pagination with rel=“next” and rel=“prev” and there is a good guide here Conquering Pagination – A Guide to Consolidating your Content

Improve this answer
3
  • Just wanted to add that Google does not support rel=next and rel=prev but suggests you still use it for other search engines. Google Hasn't Used rel=next and rel=prev for A While
    – Rob T
    Feb 5, 2020 at 15:53
  • It's a 6 year old answer! :)
    – Max
    Feb 27, 2020 at 5:29
  • Hey Max, since this answer ranks fairly well i wanted to add the above comment to let those seeing the post know the up to date information.
    – Rob T
    Feb 28, 2020 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.