1

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. But if I do this (in robots.txt), would it still be crawled?

I mean, the full articles would be still reached by the crawler even though I disallowed the only page that links to them?

I don't want the webcrawler not to access the page and enter the links in them, but I just don't want it to save the information (that will be repeated in the full articles).

Improve this question
1
  • 1
    Note that this wouldn't typically be a duplicate content problem. On a majority of websites the front page acts as a listing of (recent) content, presented as heading + snippet (think of all news sites, all blogs etc.). Also, the front page might be the most important page of the site. By using noindex, you might lose some opportunities.
    – unor
    Commented Oct 31, 2012 at 22:17

1 Answer 1

Reset to default
4

That is what the robots meta tag is for, control per page for indexing and following.

I've come to prefer it over using robots.txt as it gives finer control.

For your page, you'd want noindex,follow for the setting. The robot will read the page, not index it, but follow all the links off the page.

<meta name="robots" content="noindex,follow" />
Improve this answer
1
  • Thanks pal, this is exactly what I was looking for. I've never used the robots meta tag in the past. Thanks again!
    – federicot
    Commented Oct 29, 2012 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.