Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. But if I do this (in robots.txt), would it still be crawled?

I mean, the full articles would be still reached by the crawler even though I disallowed the only page that links to them?

I don't want the webcrawler not to access the page and enter the links in them, but I just don't want it to save the information (that will be repeated in the full articles).

share|improve this question
1  
Note that this wouldn't typically be a duplicate content problem. On a majority of websites the front page acts as a listing of (recent) content, presented as heading + snippet (think of all news sites, all blogs etc.). Also, the front page might be the most important page of the site. By using noindex, you might lose some opportunities. –  unor Oct 31 '12 at 22:17
    
@unor Thanks for the tip. I'll keep it in mind –  fedeetz Nov 4 '12 at 7:29
add comment

1 Answer 1

up vote 4 down vote accepted

That is what the robots meta tag is for, control per page for indexing and following.

I've come to prefer it over using robots.txt as it gives finer control.

For your page, you'd want noindex,follow for the setting. The robot will read the page, not index it, but follow all the links off the page.

<meta name="robots" content="noindex,follow" />
share|improve this answer
    
Thanks pal, this is exactly what I was looking for. I've never used the robots meta tag in the past. Thanks again! –  fedeetz Oct 29 '12 at 21:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.