1

I try to get the Yandex Bot under control and I'm close to block it.

After 2 months of robots.txt adjustments, it still keeps hammering my server from multiple IPs.

User-Agent: *
Crawl-delay: 4

User-agent: Yandex
Crawl-delay: 6

Over one week the average delay is 7.6 seconds (seconds / ammount of access_log rows). But still there are peaks every day where it keeps hammering for minutes using different IPs.

Here and example: 336 page requests (not counting files/scripts etc) from 12 IPs in 276 seconds (~1.2 sec)

5.255.253.3 - - [27/Apr/2019:10:43:14 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:15 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:16 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:16 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:17 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:18 +0200] "GET  …
37.9.113.7 - - [27/Apr/2019:10:43:18 +0200] "GET  …
37.9.113.156 - - [27/Apr/2019:10:43:18 +0200] "GET  …
141.8.132.26 - - [27/Apr/2019:10:43:19 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:43:19 +0200] "GET  …
95.108.181.94 - - [27/Apr/2019:10:43:19 +0200] "GET  …
87.250.224.50 - - [27/Apr/2019:10:43:20 +0200] "GET  …
87.250.224.55 - - [27/Apr/2019:10:43:21 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:22 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:23 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:24 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:24 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:25 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:28 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:28 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:29 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:31 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:43 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:44 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:45 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:46 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:47 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:48 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:48 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:49 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:43:50 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:51 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:52 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:53 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:54 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:54 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:43:55 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:56 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:56 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:43:57 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:58 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:43:59 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:43:59 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:00 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:01 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:44:01 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:02 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:03 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:04 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:04 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:05 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:44:06 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:06 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:07 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:08 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:10 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:10 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:10 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:11 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:12 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:44:12 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:14 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:15 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:16 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:17 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:17 +0200] "GET  …
141.8.142.118 - - [27/Apr/2019:10:44:18 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:19 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:19 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:21 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:21 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:21 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:44:22 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:23 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:24 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:24 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:25 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:25 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:44:25 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:26 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:27 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:44:27 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:28 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:29 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:30 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:31 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:32 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:33 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:34 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:35 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:36 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:37 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:37 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:44:37 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:38 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:38 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:44:39 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:39 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:40 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:41 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:42 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:42 +0200] "GET  …
141.8.142.118 - - [27/Apr/2019:10:44:42 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:43 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:44:43 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:44 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:45 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:46 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:47 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:48 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:49 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:50 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:51 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:51 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:52 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:53 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:55 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:55 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:56 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:44:57 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:44:59 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:00 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:00 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:02 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:03 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:05 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:05 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:05 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:07 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:07 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:07 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:08 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:09 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:10 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:10 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:11 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:12 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:13 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:13 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:14 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:15 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:15 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:16 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:17 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:17 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:17 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:19 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:20 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:22 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:23 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:23 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:24 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:25 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:26 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:26 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:27 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:29 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:29 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:30 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:31 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:32 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:32 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:33 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:33 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:33 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:34 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:35 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:36 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:37 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:37 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:39 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:39 +0200] "GET  …
87.250.224.119 - - [27/Apr/2019:10:45:39 +0200] "GET  …
141.8.132.26 - - [27/Apr/2019:10:45:40 +0200] "GET  …
141.8.142.118 - - [27/Apr/2019:10:45:40 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:40 +0200] "GET  …
87.250.224.119 - - [27/Apr/2019:10:45:41 +0200] "GET  …
141.8.132.26 - - [27/Apr/2019:10:45:41 +0200] "GET  …
141.8.142.118 - - [27/Apr/2019:10:45:41 +0200] "GET  …
213.180.203.40 - - [27/Apr/2019:10:45:42 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:43 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:44 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:44 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:45 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:46 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:48 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:49 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:49 +0200] "GET  … "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)"
141.8.132.29 - - [27/Apr/2019:10:45:51 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:51 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:51 +0200] "GET  …
141.8.132.26 - - [27/Apr/2019:10:45:52 +0200] "GET  …
141.8.142.118 - - [27/Apr/2019:10:45:52 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:53 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:53 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:54 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:54 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:54 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:56 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:56 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:57 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:45:57 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:45:59 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:45:59 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:02 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:02 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:03 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:05 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:05 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:06 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:07 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:08 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:08 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:08 +0200] "GET  …
87.250.224.119 - - [27/Apr/2019:10:46:09 +0200] "GET  …
141.8.132.26 - - [27/Apr/2019:10:46:09 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:10 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:10 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:10 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:12 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:13 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:14 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:15 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:17 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:17 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:18 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:19 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:20 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:22 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:22 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:24 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:24 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:24 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:25 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:27 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:27 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:28 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:28 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:29 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:29 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:30 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:32 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:32 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:32 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:33 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:34 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:35 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:35 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:35 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:36 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:36 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:36 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:37 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:38 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:38 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:39 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:40 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:41 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:42 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:43 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:44 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:46 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:48 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:48 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:48 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:50 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:50 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:50 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:51 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:52 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:52 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:53 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:53 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:54 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:55 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:55 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:55 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:56 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:56 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:46:56 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:46:58 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:46:59 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:01 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:02 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:03 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:04 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:05 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:06 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:07 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:08 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:10 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:11 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:11 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:12 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:47:13 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:13 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:14 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:15 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:15 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:16 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:17 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:18 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:19 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:19 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:20 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:20 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:47:21 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:22 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:22 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:47:23 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:23 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:23 +0200] "GET  …
87.250.224.119 - - [27/Apr/2019:10:47:24 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:26 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:27 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:27 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:28 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:28 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:47:29 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:29 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:30 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:31 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:33 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:34 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:34 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:35 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:36 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:47:37 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:37 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:37 +0200] "GET  …
213.180.203.61 - - [27/Apr/2019:10:47:38 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:39 +0200] "GET  …
141.8.132.29 - - [27/Apr/2019:10:47:39 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:48 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:50 +0200] "GET  …
5.255.253.3 - - [27/Apr/2019:10:47:50 +0200] "GET  …

That's only an extract of a 40 minute "visit" with 1547 page requests by 18 IPs (~1.57 sec).

When Google and the Bing bots are active at the same time I get high CPU usage.

So is Yandex ignoring my Crawl-delay rule? How can I get it to not hammer the server like above?

3

If you think that yandex is not obeying the Crawl-delay directive try signing up for Yandex Webmasters and validate your domain. Once the validation is done you can manually set the crawl rate under Indexing -> Crawl rate

Consider reading this article on Yandex Webmasters Guide to Yandex Webmaster Tools

  • No. I suggest they respect my robots.txt. – Martin Apr 28 at 22:30
  • @Martin If you want to suggest this to Yandex, you’ll have to contact Yandex. This answer would appear to be a suitable workaround until or if Yandex correct the situation, which may be assisted by you contacting them. – grg Apr 29 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.