The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is:


then it will try and crawl it as:

/&& ?param=value  

Any idea why it is doing this?

If I remember correctly the IP that this mistake is coming from is based in California and not Russia. I believe that they crawl US sites from a US based IP address. Not sure if that helps.

More information about the request:

City: Palo Alto
State: California
Country: United States
ISP: Yandex Inc.
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

There could only be two explanations for this behavior:

  1. The crawler found a link to that malformed URL, either on your site or on some other site.
  2. The Yandex crawler has a bug.

If you aren't seeing that URL crawled by other bots or visited by real users, then I suspect it is a bug with the Yandex crawler. As for why Yandex would have that particular bug, I can't say. There is no valid reason for any user agent to make that type of substitution in a URL.


Its probably mis-translating your URL, may be if your URL was more simplified such as mypage.com/page-two

  • 4
    Do you have any backing information for this? There's absolutely nothing unusual about that example. The Yandex engineers would have to be pretty incompetent to screw it up.
    – Su'
    Aug 2 '12 at 19:15
  • help.yandex.com/webmaster/?id=1113851 It provides details as to what Yandex will and will not except as URL strings.
    – nycmixing
    Aug 2 '12 at 20:16
  • I agree with Su'. Also that links to their robot guidelines/instructions and has nothing to do with this. The link in the example is normal and valid. Yandex should not have problems with it. Aug 3 '12 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.