4

Our website is an agenda of events taking place in our city.

We have a datepicker for users to look at events scheduled in the next week, next month, etc...

We recently discovered that GoogleBot is crawling this datepicker, requesting events centuries in the future: it crawls URLs like

https://example.com/2208-01-01/

Is there a way for us to tell GoogleBot to not look that far in the future?

I apologize if this is a silly question, but I am a developer, not a webmaster thus I'm pretty noobie about this.

UPDATE 1

As suggested by Stephen in the comments, I should prevent both human and bot visitors from requesting events too far in the future. I can do this in code without having to rely on updating robots.txt periodically.

I have a doubt though: won't GoogleBot periodically request all the URLs in the future that it has already crawled?

If so, maybe I should complement this solution with a few rules in robots.txt to block requests further than say 10 years in the future, or even better, do this dynamically in code once again.

| improve this question | |
  • 1
    It also appears that users can click several months ahead and get an empty calendar. Limit it for users and Googlebot at the same time. – Stephen Ostermiller♦ Mar 21 '17 at 16:29
  • You are correct that Googlebot is likely to try to crawl all the URLs that it has already crawled, even if you stop linking to them. Unor's robots.txt rules look good to me for preventing crawling of dates far in the future. – Stephen Ostermiller♦ Mar 22 '17 at 9:27
  • Thank you Stephen Ostermiller for pointing out the non-sense of not limiting the datepicker for each and every visitor – Cec Mar 27 '17 at 7:45
1

If you want to prevent crawling, you have to use robots.txt.

It would make sense to go this way if 1) you need to have the pages for these future dates and 2) you want to save your server’s and/or the search engine bot’s resources.

You can decide for which years you want to prevent crawling by specifying the beginning of the corresponding URL paths:

  • Prevent crawling of all years after 2019:

    Disallow: /202

  • Prevent crawling of all years after 2018:

    Disallow: /2019
Disallow: /202

  • Prevent crawling of all years after 2022:

    Disallow: /2023
Disallow: /2024
Disallow: /2025
Disallow: /2026
Disallow: /2027
Disallow: /2028
Disallow: /2029
Disallow: /203

  • etc.

Make sure not to forget to remove the previously blocked years when the time comes.

| improve this answer | |
  • (And you have to make sure that your site doesn’t use other URLs whose path starts with 20 etc., of course. These would get blocked, too.) – unor Mar 22 '17 at 5:22
  • Would you not need to write "Disallow: /202*" ? – Pit Mar 22 '17 at 7:06
  • 1
    @Pit: No, Disallow values match the beginning of URL paths. So Disallow: /202 blocks /202, /202.html, /20299, /202/foo etc. In fact, in the original robots.txt specification the * has no special meaning here, so with Disallow: /202* you would only block URLs that contain literally *, e.g., /202*foo.html (Google and others extend robots.txt to give * a special meaning, but not all consumers do this, of course). – unor Mar 22 '17 at 7:15
0

This is a more case specific solution, but it might be usefull in your case:

Indicate paginated content
<link rel="prev" href="http://www.example.com/article-part1.html">
<link rel="next" href="http://www.example.com/article-part3.html">

Use rel="next" and rel="prev" links to indicate the relationship between component URLs. This markup provides a strong hint to Google that you would like us to treat these pages as a logical sequence, thus consolidating their linking properties and usually sending searchers to the first page.

| improve this answer | |
  • This doesn't apply to my case as I'm not dealing with articles, but this will be of use in other sites I have to develop. – Cec Mar 22 '17 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.