I have a web calendar, all my own work. It works fine. But I see in the logs a bunch of search indexers go round and round and round trying to chase every link. It could go forward to the year 3000 month by month and day by day. I don't think the indexers should follow this as it doesn't serve much purpose. What HTTP headers or META headers should I apply to the calendar to let them know it is a calendar, but they shouldn't follow every link on those pages?
Tell me more
×
Webmasters Stack Exchange is a question and answer site for
pro webmasters. It's 100% free, no registration required.
|
Use rel="nofollow" on all links that you don't want spiders to follow or index (i.e. days, months, etc.). Use robots.txt to specify paths that you want them to ignore. If your site is a web application, consider using robots.txt to disallow indexing of the whole app with:
|
|||
|
|
|
Probably simply |
|||
|
|