6

We have hashbang urls with extended utf-8 characters. For example:

https://example.com/#!каталог

Bing bot fetches the html snapshot https://example.com?_escaped_fragment_=каталог encoded URL as \xD0\xB0.... Our server responded with a 400 error for that. They only accept percent encoded URLs. %D0%B0... would get a 200 OK status.

How can I resolve this problem?

for example: enter image description here

  • Are you sure it's a real Bingbot? I have seen (in sever logs) many poorly written bots that pretend to be e.g. Googlebot. – user11153 Jan 15 '18 at 19:38
  • 1
    pic added [updated]. IP check: dig.whois.com.au/ip/40.77.167.196 – tsogtgerel.ts Jan 29 '18 at 3:31
  • 1
    could it be be a kind, how your tomcat handles urls, not? – Evgeniy Apr 27 '18 at 10:58
5
+25

You should encode the image name before giving it to crawl for any bots, i.e. the image name should be as %2F%D0%B7%D1%83%D1%80%D0%B0%D0%B3.jpg in your page source code. All modern browser decode it and for users will show the UTF-8 characters in a human-friendly way.

So, do server-side encoding and encode all these characters before including in your source code.

View the source code for this Wikipedia page and you will understand: https://ru.wikipedia.org/wiki/%D0%9F%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%B0_%D1%80%D0%B0%D0%B7%D0%B2%D0%B8%D1%82%D0%B8%D1%8F_%D0%9E%D0%9E%D0%9D

Update: Based on deeper review the website and other StackExchange users comments below. It looks that you must encode all of your URLs in your website and XML sitemap. In your website, you don't have encoded URLs, as well not encoded URLs in XML sitemap. Please use UTF-8 encoded URLs everywhere. Keep in mind, that Bingbot is way behind Googlebot and just recently started to render JS based website in more wider scale. Using non-ASCII characters could cause bugs for Bingbot.

The URLs in your source code and DOM, as well in your XML sitemap must be encoded like this based on your example:

https://example.com/#!%D0%BA%D0%B0%D1%82%D0%B0%D0%BB%D0%BE%D0%B3

Here you can read more about proper sitemap encoding: https://www.sitemaps.org/protocol.html#escaping

  • You think they are linking to зураг.jpg rather than %D0%B7ураг.jpg ? – Stephen Ostermiller May 21 '18 at 0:50
  • Yes, I assume, that in the source code the image name is: зураг.jpg – gintsg May 21 '18 at 1:28
  • 1
    @ts.tsogtgerel thanks for sharing your website. Now everything is clear. The main issue is, that your website is in JS and Bing is extremely new to rendering JS content and index JS based website. Google is pretty advanced with rendering JS files and see content pretty good, but Bing is a newbie here. Maybe you can talk with your team about an option to prerender your content or something? Here is a pretty good guide about JS rendered websites This information will give you a lot of information about the issue. – gintsg May 22 '18 at 8:11
  • 1
    @ts.tsogtgerel additional note. In your website DOM, I can see, that your URLs is not encoded as well. But in your case, Bing bot even can't see your content. – gintsg May 22 '18 at 8:18
  • 1
    "The main issue is, that your website is in JS" - The JS element could hamper Bing from crawling/indexing the site to begin with, but this should not be the reason for Bing crawling the incorrectly encoded URLs. (?) Incidentally, it seems the unencoded URLs are present in the XML sitemap as well, which I imagine is probably being submitted to Bing's Webmaster Tools? – MrWhite May 23 '18 at 0:33
0

You probably need to urlencode images src attribute in <img> html tags.

  • 2
    This is considered a low quality answer. Can you explain your thinking more and include some details? Assume the OP has little to no idea what you are talking about. Cheers!! – closetnoc May 18 '18 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.