Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like this:
"Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
But when I check statcounter logs I recently saw that other Yandex related bots frequently crawl my site. They are similar to following. I took this info from my cPanel log:
Beeline (188.8.131.52) Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2) Referer: http://yandex.ru/yandsearch?text=example.com&lr=213 Beeline (184.108.40.206) Referer: http://yandex.ru/yandsearch?text=example.com&lr=213 Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2)
How can I block or serve cached pages these bots?
When I check for
$_SERVER['HTTP_USER_AGENT'] I can't see "
yandex.ru" in referrer. Referrer comes empty. Is it possible to find out referrer in cPanel log, but can't take it from
And I also I don't want to ban IPs because there exist too many IPs related with this issue and they are changed periodically. So how can I find out this bot?
Does anybody has similar issue? Thank you