Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) /Guitars/Children%C3%C6%92%E2%E2%82%AC%C5%A1%C3%E2%80%9A%C2%B4s-Guitars/ which once was /Guitars/Children%C2%B4s-Guitars/ and perfectly works.

Others are even longer. How does this happen? And what kind of encoding is it? Apparently not UTF8...

If I could restore the original URL I could 301 them to the right address, which hopefully would stop them crawling this.

  • Not speaking specifically. I have seen double encoding where encoding is done once then the %C3 would be encoded. This is not the case in your example because you would see a pattern and it would still work. Some spider software will encode incorrectly. These are often your content scrapers and such. As well, hackers will use encoding to disguise hack patterns. But none of these things seem to apply here. I suspect that there is a database of links that has this encoded URL. Scrapers share their databases and you will see sudden access for the same bad request for a period including years. – closetnoc Oct 14 '14 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.