When I check BingWebmaster and use "Fetch as BingBot" then Bing is able to crawl that particular URL of my site. However, when I use SiteScan tool, then I am getting 4xx error. (SiteScan tool shows where we need on-page SEO improvements and that's why it is very important that SiteScan tool works.)
Then I tried TechnicalSEO tool to fetch&render and I did the test as follows for Bingbot useragent:
- Fetch & Render Tool link: https://technicalseo.com/tools/fetch-render/
- Pre Rendering Testing Tool link: https://technicalseo.com/tools/pre-rendering/
- Check for URL: https://www.playerauctions.com/osrs-gold/
and then found that CloudFlare forbids Bingbot useragent and results into HTTP 403 response.
HTTP/2 403
date: Thu, 06 Aug 2020 01:46:22 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d3f73e32abc0f9b1a8717b943883ee1b41596678382; expires=Sat, 05-Sep-20 01:46:22 GMT; path=/; domain=.playerauctions.com; HttpOnly; SameSite=Lax
cache-control: max-age=15
expires: Thu, 06 Aug 2020 01:46:37 GMT
x-frame-options: SAMEORIGIN
cf-request-id: 04630bbbc4000002c02a114200000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 5be51572dbc202c0-SEA
content-encoding: gzip
- I reached out to Bing helpcenter and they said check with your Network team.
- Network team said they couldn't find anything at CloudFlare which is blocking Bingbot.
I have checked posts on CloudFlare Forum on Bingbot not able to access to my site, Cloudflare bing bot block and Bingbot blocked however none of them works.
So what should I do to fix this problem?
curl -A 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)' --head 'https://www.playerauctions.com/osrs-gold/'That gets a 403 error with the message "This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data."curl -A 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)' --head 'https://attesoro.com'That domain just has the default Firewall rules -- the five "managed rules" to prevent different types of "floods" which cannot be disabled.') to double quotes (") in that command. The user agent has to be passed as a single argument and from the error message, it appears that the parts of it are each getting passed separately. i'm on Linux and don't have Windows, so I can't test it for you on Windows.