Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a web calendar, all my own work. It works fine. But I see in the logs a bunch of search indexers go round and round and round trying to chase every link. It could go forward to the year 3000 month by month and day by day. I don't think the indexers should follow this as it doesn't serve much purpose. What HTTP headers or META headers should I apply to the calendar to let them know it is a calendar, but they shouldn't follow every link on those pages?

share|improve this question
add comment

2 Answers

up vote 3 down vote accepted

Use rel="nofollow" on all links that you don't want spiders to follow or index (i.e. days, months, etc.).

Use robots.txt to specify paths that you want them to ignore.

If your site is a web application, consider using robots.txt to disallow indexing of the whole app with:

User-agent: *
Disallow: /
share|improve this answer
add comment

Probably simply <a rel='nofollow' ... on the calendar navigation links (as opposed to the events that link to specific events) would be fine.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.