6

We have hashbang urls with extended utf-8 characters. For example: 

https://example.com/#!каталог

Bing bot fetches the html snapshot https://example.com?_escaped_fragment_=каталог encoded URL as \xD0\xB0.... Our server responded with a 400 error for that. They only accept percent encoded URLs. %D0%B0... would get a 200 OK status.

How can I resolve this problem?

for example: enter image description here

Improve this question
3
  • Are you sure it's a real Bingbot? I have seen (in sever logs) many poorly written bots that pretend to be e.g. Googlebot.
    – user11153
    Commented Jan 15, 2018 at 19:38
  • 1
    pic added [updated]. IP check: dig.whois.com.au/ip/40.77.167.196
    – tsogtgerel.ts
    Commented Jan 29, 2018 at 3:31
  • 1
    could it be be a kind, how your tomcat handles urls, not?
    – Evgeniy
    Commented Apr 27, 2018 at 10:58

2 Answers 2

Reset to default
5
+25

You should encode the image name before giving it to crawl for any bots, i.e. the image name should be as %2F%D0%B7%D1%83%D1%80%D0%B0%D0%B3.jpg in your page source code. All modern browser decode it and for users will show the UTF-8 characters in a human-friendly way.

So, do server-side encoding and encode all these characters before including in your source code.

View the source code for this Wikipedia page and you will understand: https://ru.wikipedia.org/wiki/%D0%9F%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%B0_%D1%80%D0%B0%D0%B7%D0%B2%D0%B8%D1%82%D0%B8%D1%8F_%D0%9E%D0%9E%D0%9D

Update: Based on deeper review the website and other StackExchange users comments below. It looks that you must encode all of your URLs in your website and XML sitemap. In your website, you don't have encoded URLs, as well not encoded URLs in XML sitemap. Please use UTF-8 encoded URLs everywhere. Keep in mind, that Bingbot is way behind Googlebot and just recently started to render JS based website in more wider scale. Using non-ASCII characters could cause bugs for Bingbot.

The URLs in your source code and DOM, as well in your XML sitemap must be encoded like this based on your example:

https://example.com/#!%D0%BA%D0%B0%D1%82%D0%B0%D0%BB%D0%BE%D0%B3

Here you can read more about proper sitemap encoding: https://www.sitemaps.org/protocol.html#escaping

Improve this answer
8
  • You think they are linking to зураг.jpg rather than %D0%B7ураг.jpg ?
    – Stephen Ostermiller ♦
    Commented May 21, 2018 at 0:50
  • Yes, I assume, that in the source code the image name is: зураг.jpg
    – gintsg
    Commented May 21, 2018 at 1:28
  • 1
    @ts.tsogtgerel thanks for sharing your website. Now everything is clear. The main issue is, that your website is in JS and Bing is extremely new to rendering JS content and index JS based website. Google is pretty advanced with rendering JS files and see content pretty good, but Bing is a newbie here. Maybe you can talk with your team about an option to prerender your content or something? Here is a pretty good guide about JS rendered websites This information will give you a lot of information about the issue.
    – gintsg
    Commented May 22, 2018 at 8:11
  • 1
    @ts.tsogtgerel additional note. In your website DOM, I can see, that your URLs is not encoded as well. But in your case, Bing bot even can't see your content.
    – gintsg
    Commented May 22, 2018 at 8:18
  • 1
    "The main issue is, that your website is in JS" - The JS element could hamper Bing from crawling/indexing the site to begin with, but this should not be the reason for Bing crawling the incorrectly encoded URLs. (?) Incidentally, it seems the unencoded URLs are present in the XML sitemap as well, which I imagine is probably being submitted to Bing's Webmaster Tools?
    – MrWhite
    Commented May 23, 2018 at 0:33
0

You probably need to urlencode images src attribute in <img> html tags.

Improve this answer
1
  • 2
    This is considered a low quality answer. Can you explain your thinking more and include some details? Assume the OP has little to no idea what you are talking about. Cheers!!
    – closetnoc
    Commented May 18, 2018 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.