Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or category page included in the results. Sometimes it even ranks ahead of the article itself. Obviously, as these links aren't always going to have the article on them, this aren't the results I want people to click on.

So, I'm thinking of setting noindex on these pages. Is there any possible downside to doing so? Is this possible to do via robots.txt, or do I have to add it to all the relevant templates? All I can find for robots.txt are ways to stop the search engine crawling those pages, which isn't what I want - while I don't want them indexed, it's still the only surefire way to find all my blog posts.

share|improve this question
add comment

2 Answers

up vote 4 down vote accepted

No. Read this(all of it; there's a lot of useful stuff), though of particular relevance here:

Google no longer recommends blocking crawler access to duplicate content on your website, whether with a robots.txt file or other methods.

[...]

Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results.

Search engines are aware that sites(especially blogs) have pages which are basically just listings of excerpts. They're looking for more than that a far as duplicate content.

share|improve this answer
add comment

My first answer is: why would you want less exposure on Google?

Surely people will still get to the article, even if they have to go via the listing page. And that is a good thing.

Maybe rather than stopping Google from indexing certain pages, why not do some work on the articles to get them higher. Look at the text on the pages as well as the keywords, title and description (all basic SEO stuff that I'm sure you know).

share|improve this answer
    
1. Because I have heard that google penalises pages with duplicate content, and at the moment my site has at least 4 full copies of each post (post page, chronological page, tag page, category page). Better first or second than both fourth and fifth. 2. With the exception of the post page, between the time that google indexes a page and the user clicks the link, the content may be at the bottom of the page or no longer be on that page. Will they look around to find it, or just click straight off? IMO most users will do the second. –  Macha Feb 11 '11 at 18:28
1  
@Macha Why are you publishing complete post content on your date/tag-based pages? Seems like that would just make them unnecessarily large. Just from a design perspective, I'd cut them down to excerpts, or even just headlines if it's that they're very short. –  Su' Feb 11 '11 at 20:36
    
@macha you're getting confused with what constitutes duplicate content. Since there will be other content in your tag or category pages then it won't be an issue especially if you do what Su' suggests. I reckon headlines personally. –  Piers Karsenbarg Feb 12 '11 at 10:18
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.