On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or category page included in the results. Sometimes it even ranks ahead of the article itself. Obviously, as these links aren't always going to have the article on them, this aren't the results I want people to click on.

So, I'm thinking of setting noindex on these pages. Is there any possible downside to doing so? Is this possible to do via robots.txt, or do I have to add it to all the relevant templates? All I can find for robots.txt are ways to stop the search engine crawling those pages, which isn't what I want - while I don't want them indexed, it's still the only surefire way to find all my blog posts.

2 Answers 2


No. Read this(all of it; there's a lot of useful stuff), though of particular relevance here:

Google no longer recommends blocking crawler access to duplicate content on your website, whether with a robots.txt file or other methods.


Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results.

Search engines are aware that sites(especially blogs) have pages which are basically just listings of excerpts. They're looking for more than that a far as duplicate content.


My first answer is: why would you want less exposure on Google?

Surely people will still get to the article, even if they have to go via the listing page. And that is a good thing.

Maybe rather than stopping Google from indexing certain pages, why not do some work on the articles to get them higher. Look at the text on the pages as well as the keywords, title and description (all basic SEO stuff that I'm sure you know).

  • 1. Because I have heard that google penalises pages with duplicate content, and at the moment my site has at least 4 full copies of each post (post page, chronological page, tag page, category page). Better first or second than both fourth and fifth. 2. With the exception of the post page, between the time that google indexes a page and the user clicks the link, the content may be at the bottom of the page or no longer be on that page. Will they look around to find it, or just click straight off? IMO most users will do the second.
    – Macha
    Commented Feb 11, 2011 at 18:28
  • 1
    @Macha Why are you publishing complete post content on your date/tag-based pages? Seems like that would just make them unnecessarily large. Just from a design perspective, I'd cut them down to excerpts, or even just headlines if it's that they're very short.
    – Su'
    Commented Feb 11, 2011 at 20:36
  • @macha you're getting confused with what constitutes duplicate content. Since there will be other content in your tag or category pages then it won't be an issue especially if you do what Su' suggests. I reckon headlines personally. Commented Feb 12, 2011 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.