Timeline for Cloudflare is blocking Bingbot
Current License: CC BY-SA 4.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 10, 2020 at 19:23 | vote | accept | Bhargav Joshi | ||
| Aug 10, 2020 at 5:23 | answer | added | Bhargav Joshi | timeline score: 3 | |
| Aug 8, 2020 at 15:30 | comment | added | Bhargav Joshi | Yep sure. That would be my pleasure. I’ll do it on Monday | |
| Aug 8, 2020 at 14:25 | comment | added | Stephen Ostermiller♦ | Can you add an answer saying exactly which settings at cloudflare got changed? It would help other people that come across this question in the future. | |
| Aug 8, 2020 at 12:39 | comment | added | Bhargav Joshi | Finally updated configuration at Cloudflare and then it worked like a charm! Thank you very much for the help :) | |
| Aug 7, 2020 at 5:16 | comment | added | Bhargav Joshi | I reached out to CloudFlare support and they suggested some solutions, that too is not working, so trying to figure out with their help documents. | |
| Aug 6, 2020 at 10:32 | comment | added | Bhargav Joshi | I see... okay. I’ll copy paste the error message from here and show it to network engineer. She is still claiming that everything is fine at firewall, so I’ll show her thi evidence. Thank you so much for your time, Stephen. This is very helpful. | |
| Aug 6, 2020 at 10:19 | comment | added | Stephen Ostermiller♦ |
It looks like Windows command line doesn't interpret single quotes. Try switching the single quotes (') to double quotes (") in that command. The user agent has to be passed as a single argument and from the error message, it appears that the parts of it are each getting passed separately. i'm on Linux and don't have Windows, so I can't test it for you on Windows.
|
|
| Aug 6, 2020 at 10:12 | comment | added | Bhargav Joshi | thanks, I tried copy-pasting that exact same command, and getting this message (uploaded screenshot on IMGUR): imgur.com/0w28EIe | |
| Aug 6, 2020 at 9:25 | comment | added | Bhargav Joshi | alright, thank you very much. | |
| Aug 6, 2020 at 9:23 | comment | added | Stephen Ostermiller♦ | Curl is an open source command line application for fetching websites: curl.haxx.se It can be installed on Windows, Mac, or Linux. | |
| Aug 6, 2020 at 9:22 | comment | added | Bhargav Joshi | Let me check with network person again. By the way, which tool to use to execute curl command? i wil study that and do some experiments. | |
| Aug 6, 2020 at 9:19 | comment | added | Stephen Ostermiller♦ |
I think it is cloudflare blocking Bingbot. It could also be the origin server, but I think that cloudflare is more likely. What rules are enabled on the firewall? I have a domain hosted on cloudflare that just redirects. I can fetch it with Bingbot just fine and get the redirect: curl -A 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)' --head 'https://attesoro.com' That domain just has the default Firewall rules -- the five "managed rules" to prevent different types of "floods" which cannot be disabled.
|
|
| Aug 6, 2020 at 9:17 | comment | added | Bhargav Joshi | I see.. so the client has a cloudflare firewall, so that means cloudflare itself blocking Bingbots, right? | |
| Aug 6, 2020 at 9:13 | comment | added | Stephen Ostermiller♦ |
I can confirm that your site is not available to Bingbot by using the following command line to fetch it: curl -A 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)' --head 'https://www.playerauctions.com/osrs-gold/' That gets a 403 error with the message "This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data."
|
|
| Aug 6, 2020 at 2:26 | history | asked | Bhargav Joshi | CC BY-SA 4.0 |