3

When I check BingWebmaster and use "Fetch as BingBot" then Bing is able to crawl that particular URL of my site. However, when I use SiteScan tool, then I am getting 4xx error. (SiteScan tool shows where we need on-page SEO improvements and that's why it is very important that SiteScan tool works.)

Then I tried TechnicalSEO tool to fetch&render and I did the test as follows for Bingbot useragent:

and then found that CloudFlare forbids Bingbot useragent and results into HTTP 403 response.

HTTP/2 403 
date: Thu, 06 Aug 2020 01:46:22 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d3f73e32abc0f9b1a8717b943883ee1b41596678382; expires=Sat, 05-Sep-20 01:46:22 GMT; path=/; domain=.playerauctions.com; HttpOnly; SameSite=Lax
cache-control: max-age=15
expires: Thu, 06 Aug 2020 01:46:37 GMT
x-frame-options: SAMEORIGIN
cf-request-id: 04630bbbc4000002c02a114200000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 5be51572dbc202c0-SEA
content-encoding: gzip
  • I reached out to Bing helpcenter and they said check with your Network team.
  • Network team said they couldn't find anything at CloudFlare which is blocking Bingbot.

I have checked posts on CloudFlare Forum on Bingbot not able to access to my site, Cloudflare bing bot block and Bingbot blocked however none of them works.

So what should I do to fix this problem?

13
  • 1
    I can confirm that your site is not available to Bingbot by using the following command line to fetch it: curl -A 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)' --head 'https://www.playerauctions.com/osrs-gold/' That gets a 403 error with the message "This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data." Aug 6 '20 at 9:13
  • I see.. so the client has a cloudflare firewall, so that means cloudflare itself blocking Bingbots, right? Aug 6 '20 at 9:17
  • 1
    I think it is cloudflare blocking Bingbot. It could also be the origin server, but I think that cloudflare is more likely. What rules are enabled on the firewall? I have a domain hosted on cloudflare that just redirects. I can fetch it with Bingbot just fine and get the redirect: curl -A 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)' --head 'https://attesoro.com' That domain just has the default Firewall rules -- the five "managed rules" to prevent different types of "floods" which cannot be disabled. Aug 6 '20 at 9:19
  • 1
    Curl is an open source command line application for fetching websites: curl.haxx.se It can be installed on Windows, Mac, or Linux. Aug 6 '20 at 9:23
  • 1
    It looks like Windows command line doesn't interpret single quotes. Try switching the single quotes (') to double quotes (") in that command. The user agent has to be passed as a single argument and from the error message, it appears that the parts of it are each getting passed separately. i'm on Linux and don't have Windows, so I can't test it for you on Windows. Aug 6 '20 at 10:19
3

We found out that not only CloudFlare, but also other services on CloudFlare were blocking BingBot.

Finally we allowed Bing user-agent Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) in Firewall settings and also added another rule to by-pass other default rules in WAF when the user-agent is Bingbot.

It may vary as per individual configurations, so suggesting to refer to CloudFlare FireWall Rules https://developers.cloudflare.com/firewall/cf-firewall-rules/actions/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.