3

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like this: "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"

But when I check statcounter logs I recently saw that other Yandex related bots frequently crawl my site. They are similar to following. I took this info from my cPanel log:

Beeline (128.69.243.12)
Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2)
Referer: http://yandex.ru/yandsearch?text=example.com&lr=213

Beeline (89.178.108.247)
Referer: http://yandex.ru/yandsearch?text=example.com&lr=213
Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2)

How can I block or serve cached pages these bots?

When I check for $_SERVER['HTTP_USER_AGENT'] I can't see "yandex.ru" in referrer. Referrer comes empty. Is it possible to find out referrer in cPanel log, but can't take it from HTTP_USER_AGENT ??

And I also I don't want to ban IPs because there exist too many IPs related with this issue and they are changed periodically. So how can I find out this bot?

Does anybody has similar issue? Thank you

3
  • 2
    Why do you think these are bot requests? The Referer matches what a normal user of a Yandex search would send when clicking on a search result from Yandex.
    – mgkrebbs
    Jul 28, 2012 at 1:06
  • Because these types of requests come to my site with very systematically. 10-15 times every day, with similar intervals. I don't think a normal visitors would have such patience to make it every day.
    – trante
    Jul 28, 2012 at 6:33
  • Those two ip addresses are not owned by Yandex and they are not Yandex bots, they are beelinebroadband.net subscribers that are clicking links from yandex.ru Aug 3, 2012 at 22:31

1 Answer 1

2

Use a robots.txt crawl delay as described in http://help.yandex.com/search/?id=1112639

Example:

User-agent: Yandex
Crawl-delay: 2 # specifies a 2 second timeout

Before you start banning this bot, you should first verify that your logs are actually Yandex and not someone else who is spoofing the user agent to look like they are yandex. A tactic used by competitors to hussle you into blocking or delaying a bot so they can out rank you. Perform a DNS lookup : http://help.yandex.com/search/?id=1112029

You can serve a cache copy dependant on user agent in a number of ways. If you use apache you can do it with mod_rewrite rules. If you use PHP you can do it by sniffing the $_SERVER['HTTP_USER_AGENT'] variable or even use the function get_browser(). How you build a cache is also varied and can be done in 101 ways. Honestly though for the best performance you should always be using caching.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.