When I check BingWebmaster and use "Fetch as BingBot" then Bing is able to crawl that particular URL of my site. However, when I use SiteScan tool, then I am getting 4xx error. (SiteScan tool shows where we need on-page SEO improvements and that's why it is very important that SiteScan tool works.)
Then I tried TechnicalSEO tool to fetch&render and I did the test as follows for Bingbot useragent:
- Fetch & Render Tool link: https://technicalseo.com/tools/fetch-render/
- Pre Rendering Testing Tool link: https://technicalseo.com/tools/pre-rendering/
- Check for URL: https://www.playerauctions.com/osrs-gold/
and then found that CloudFlare forbids Bingbot useragent and results into HTTP 403 response.
HTTP/2 403 date: Thu, 06 Aug 2020 01:46:22 GMT content-type: text/html; charset=UTF-8 set-cookie: __cfduid=d3f73e32abc0f9b1a8717b943883ee1b41596678382; expires=Sat, 05-Sep-20 01:46:22 GMT; path=/; domain=.playerauctions.com; HttpOnly; SameSite=Lax cache-control: max-age=15 expires: Thu, 06 Aug 2020 01:46:37 GMT x-frame-options: SAMEORIGIN cf-request-id: 04630bbbc4000002c02a114200000001 expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" server: cloudflare cf-ray: 5be51572dbc202c0-SEA content-encoding: gzip
- I reached out to Bing helpcenter and they said check with your Network team.
- Network team said they couldn't find anything at CloudFlare which is blocking Bingbot.
So what should I do to fix this problem?