Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is:

/mypage.aspx?param=value 

then it will try and crawl it as:

/&& ?param=value  

Any idea why it is doing this?

EDIT:
If I remember correctly the IP that this mistake is coming from is based in California and not Russia. I believe that they crawl US sites from a US based IP address. Not sure if that helps.

More information about the request:

IP: 199.21.99.82
City: Palo Alto
State: California
Country: United States
ISP: Yandex Inc.
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
share|improve this question
add comment

2 Answers 2

There could only be two explanations for this behavior:

  1. The crawler found a link to that malformed URL, either on your site or on some other site.
  2. The Yandex crawler has a bug.

If you aren't seeing that URL crawled by other bots or visited by real users, then I suspect it is a bug with the Yandex crawler. As for why Yandex would have that particular bug, I can't say. There is no valid reason for any user agent to make that type of substitution in a URL.

share|improve this answer
add comment

Its probably mis-translating your URL, may be if your URL was more simplified such as mypage.com/page-two

share|improve this answer
4  
Do you have any backing information for this? There's absolutely nothing unusual about that example. The Yandex engineers would have to be pretty incompetent to screw it up. –  Su' Aug 2 '12 at 19:15
    
help.yandex.com/webmaster/?id=1113851 It provides details as to what Yandex will and will not except as URL strings. –  nycmixing Aug 2 '12 at 20:16
    
I agree with Su'. Also that links to their robot guidelines/instructions and has nothing to do with this. The link in the example is normal and valid. Yandex should not have problems with it. –  Anthony Hatzopoulos Aug 3 '12 at 2:02
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.