Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a web calendar, all my own work. It works fine. But I see in the logs a bunch of search indexers go round and round and round trying to chase every link. It could go forward to the year 3000 month by month and day by day. I don't think the indexers should follow this as it doesn't serve much purpose. What HTTP headers or META headers should I apply to the calendar to let them know it is a calendar, but they shouldn't follow every link on those pages?

share|improve this question
up vote 3 down vote accepted

Use rel="nofollow" on all links that you don't want spiders to follow or index (i.e. days, months, etc.).

Use robots.txt to specify paths that you want them to ignore.

If your site is a web application, consider using robots.txt to disallow indexing of the whole app with:

User-agent: *
Disallow: /
share|improve this answer

Probably simply <a rel='nofollow' ... on the calendar navigation links (as opposed to the events that link to specific events) would be fine.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.