Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is:


then it will try and crawl it as:

/&& ?param=value  

Any idea why it is doing this?

If I remember correctly the IP that this mistake is coming from is based in California and not Russia. I believe that they crawl US sites from a US based IP address. Not sure if that helps.

More information about the request:

City: Palo Alto
State: California
Country: United States
ISP: Yandex Inc.
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
share|improve this question

There could only be two explanations for this behavior:

  1. The crawler found a link to that malformed URL, either on your site or on some other site.
  2. The Yandex crawler has a bug.

If you aren't seeing that URL crawled by other bots or visited by real users, then I suspect it is a bug with the Yandex crawler. As for why Yandex would have that particular bug, I can't say. There is no valid reason for any user agent to make that type of substitution in a URL.

share|improve this answer

Its probably mis-translating your URL, may be if your URL was more simplified such as mypage.com/page-two

share|improve this answer
Do you have any backing information for this? There's absolutely nothing unusual about that example. The Yandex engineers would have to be pretty incompetent to screw it up. – Su' Aug 2 '12 at 19:15
help.yandex.com/webmaster/?id=1113851 It provides details as to what Yandex will and will not except as URL strings. – nycmixing Aug 2 '12 at 20:16
I agree with Su'. Also that links to their robot guidelines/instructions and has nothing to do with this. The link in the example is normal and valid. Yandex should not have problems with it. – Anthony Hatzopoulos Aug 3 '12 at 2:02

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.