Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down.
Does it follow Crawl-Delay from robots.txt, or not?
-
Hard to know without testing it. Their support page doesn't mention crawl delay: About Applebot - Apple Support– Stephen Ostermiller ♦Commented Jul 30, 2019 at 10:10
-
I've emailed them (without much hope), and added Crawl-delay in my robots.txt. I'll update if something is conclusive.– magnetikCommented Jul 30, 2019 at 12:47
Add a comment
|
1 Answer
To my knowledge and experience, Bing is the only bot to really observe the crawl-delay directive, but I saw that Yahoo and Yandex are supposed to also. This directive is pretty new and no standard have been defined, so its implementation varies from a bot to another as this guy says it here.
Google stated clearly it does not follow it and invites you to set this in your Search Console.
Since Apple says on its website that it respects standard robots.txt directives, I don't expect them to have implemented the crawl-delay directive in any way.
If the issue is still happening, I would block Applebot for a few days if you are sure it's the one causing too much load on your server.
One thing though, how did you identify it was Applebot the problem? Did you set up a log of all User Agents crawling your site? I did that once, and I was amazed with the high number of bots on my site in a single week.
