I own a web site which has mainly two types of pages:
- Content pages: they are pretty much static, and link to the previous and next content page (chronologically). The structure is that of a double linked list.
- Listing pages: they are dynamic, and are paginated. The first one shows the most recent content. Older content is paginated via a GET parameter (e.g ?page=4). Every listing page links to several content pages.
Google is indexing some of the listing pages, which will most likely be outdated when the users find them (new content is added at a pretty fast rate). This is undesirable, of course. Also, listing pages have extracts of the content, which could be seen as repeated content from Google's point of view (which I think is also undesirable, I'm unsure on this).
The problem is, there are hundreds of thousands of content pages. Listing pages are very convenient both for visitors and for search engines to crawl the site. I am unsure if the linked list of content pages will be sufficient to allow search engines to crawl the site in a short time.
I have thought about the following options:
Disallow: /?page*to robots.txt: listing pages (other than the first one) won't be accessed.
- Add meta
nofollowto listing pages: similar to the first one, listing pages won't be accessed.
- Add meta
noindexto listing pages: they will be accesses, but not added to the index
- Leave things as they are: eventually search engines will realize the 'page' parameter is for pagination, and won't index it.
What are your suggestions? Adding the NoIndex meta to listing pages sounds like the best solution to me, but I don't dare approach such a large task without asking first :)
Any other possible solutions I didn't think of?