Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like this: "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
But when I check statcounter logs I recently saw that other Yandex related bots frequently crawl my site. They are similar to following. I took this info from my cPanel log:
Beeline (128.69.243.12)
Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2)
Referer: http://yandex.ru/yandsearch?text=example.com&lr=213
Beeline (89.178.108.247)
Referer: http://yandex.ru/yandsearch?text=example.com&lr=213
Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2)
How can I block or serve cached pages these bots?
When I check for $_SERVER['HTTP_USER_AGENT']
I can't see "yandex.ru
" in referrer. Referrer comes empty. Is it possible to find out referrer in cPanel log, but can't take it from HTTP_USER_AGENT
??
And I also I don't want to ban IPs because there exist too many IPs related with this issue and they are changed periodically. So how can I find out this bot?
Does anybody has similar issue? Thank you