Our website is an agenda of events taking place in our city.

We have a datepicker for users to look at events scheduled in the next week, next month, etc...

We recently discovered that GoogleBot is crawling this datepicker, requesting events centuries in the future: it crawls URLs like


Is there a way for us to tell GoogleBot to not look that far in the future?

I apologize if this is a silly question, but I am a developer, not a webmaster thus I'm pretty noobie about this.


As suggested by Stephen in the comments, I should prevent both human and bot visitors from requesting events too far in the future. I can do this in code without having to rely on updating robots.txt periodically.

I have a doubt though: won't GoogleBot periodically request all the URLs in the future that it has already crawled?

If so, maybe I should complement this solution with a few rules in robots.txt to block requests further than say 10 years in the future, or even better, do this dynamically in code once again.

  • 1
    It also appears that users can click several months ahead and get an empty calendar. Limit it for users and Googlebot at the same time. Mar 21, 2017 at 16:29
  • You are correct that Googlebot is likely to try to crawl all the URLs that it has already crawled, even if you stop linking to them. Unor's robots.txt rules look good to me for preventing crawling of dates far in the future. Mar 22, 2017 at 9:27
  • Thank you Stephen Ostermiller for pointing out the non-sense of not limiting the datepicker for each and every visitor
    – Cec
    Mar 27, 2017 at 7:45

2 Answers 2


If you want to prevent crawling, you have to use robots.txt.

It would make sense to go this way if 1) you need to have the pages for these future dates and 2) you want to save your server’s and/or the search engine bot’s resources.

You can decide for which years you want to prevent crawling by specifying the beginning of the corresponding URL paths:

  • Prevent crawling of all years after 2019:

    Disallow: /202
  • Prevent crawling of all years after 2018:

    Disallow: /2019
    Disallow: /202
  • Prevent crawling of all years after 2022:

    Disallow: /2023
    Disallow: /2024
    Disallow: /2025
    Disallow: /2026
    Disallow: /2027
    Disallow: /2028
    Disallow: /2029
    Disallow: /203
  • etc.

Make sure not to forget to remove the previously blocked years when the time comes.

  • (And you have to make sure that your site doesn’t use other URLs whose path starts with 20 etc., of course. These would get blocked, too.)
    – unor
    Mar 22, 2017 at 5:22
  • Would you not need to write "Disallow: /202*" ?
    – Pit
    Mar 22, 2017 at 7:06
  • 1
    @Pit: No, Disallow values match the beginning of URL paths. So Disallow: /202 blocks /202, /202.html, /20299, /202/foo etc. In fact, in the original robots.txt specification the * has no special meaning here, so with Disallow: /202* you would only block URLs that contain literally *, e.g., /202*foo.html (Google and others extend robots.txt to give * a special meaning, but not all consumers do this, of course).
    – unor
    Mar 22, 2017 at 7:15

This is a more case specific solution, but it might be usefull in your case:

Indicate paginated content
<link rel="prev" href="http://www.example.com/article-part1.html">
<link rel="next" href="http://www.example.com/article-part3.html">

Use rel="next" and rel="prev" links to indicate the relationship between component URLs. This markup provides a strong hint to Google that you would like us to treat these pages as a logical sequence, thus consolidating their linking properties and usually sending searchers to the first page.

  • This doesn't apply to my case as I'm not dealing with articles, but this will be of use in other sites I have to develop.
    – Cec
    Mar 22, 2017 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.