2

This is a small issue that has been ongoing for quite some time.

When I check my Apache server log for 403's, all entries are related to Bing, and all from when it requests a robots.txt, sitemap.txt, or sitemap.xml.

The problem only happens with Bing and only when they request a sitemap or robots.txt. The problem does not happen with actual Bing crawling, and even if I do an URL inspection / live test on one of the robots.txt or sitemap files, the problem does NOT happen that way.

What I'm seeing is that Apache will reject the first request with a 403, and then there will immediately be a 2nd request that will be successful. So it's pretty harmless except for junking up the logs.

Here are some examples of what I see in the logs:

[2021-06-20/15:11] (mydomain1).org 207.46.13.96 "GET /sitemap.txt HTTP/2.0" 403 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
[2021-06-20/15:11] (mydomain1).org 207.46.13.96 "GET /sitemap.txt HTTP/2.0" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

[2021-06-20/07:36] (mydomain2).net 207.46.13.45 "GET /robots.txt HTTP/2.0" 403 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
[2021-06-20/07:36] (mydomain2).net 207.46.13.45 "GET /robots.txt HTTP/2.0" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

[2021-06-20/15:13] (mydomain3).com 40.77.167.21 "GET /sitemap.xml HTTP/2.0" 403 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
[2021-06-20/15:13] (mydomain3).com 40.77.167.21 "GET /sitemap.xml HTTP/2.0" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Interestingly if I route the traffic through Cloudflare the problem goes away (or maybe the initial bad request never reaches my server and I only see the 2nd good request).

Cloudflare examples:

[2021-06-20/15:11] (mydomain4).tk 2400:cb00:28:1024::6ca2:f654 "GET /sitemap.txt HTTP/1.1" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

[2021-06-20/15:12] (mydomain4).tk 2400:cb00:28:1024::6ca2:f6c8 "GET /sitemap.xml HTTP/1.1" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

I have mod_security running but it's in DETECTION_ONLY mode and shouldn't be blocking anything on its own, just logging.

When one of these 403's happens, I get an entry in modsec_audit.log, because it logs all 403's, but it's not very helpful:

--9ea2ef5a-A--
[20/Jun/2021:15:11:29 --0500] YM@g8YvFIvJDLJuWINL3ewAAAAE 40.77.167.21 53442 (myIP) 443
--9ea2ef5a-B--
GET /sitemap.xml HTTP/2.0
Cache-Control: no-cache
Pragma: no-cache
Accept: */*
Accept-Encoding: gzip, deflate
From: bingbot(at)microsoft.com
User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Host: (mydomain1).com
--9ea2ef5a-H--
Stopwatch: 1624219889292746 4001 (- - -)
Stopwatch2: 1624219889292746 4001; combined=1982, p1=1533, p2=0, p3=0, p4=0, p5=443, sr=492, sw=6, l=0, gc=0
Producer: ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/); OWASP_CRS/3.2.0.
Server: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f
Engine-Mode: "DETECTION_ONLY"
--9ea2ef5a-Z--
3
  • You may have to turn on additional logging to figure this out. Bing could be sending bad authentication tokens. Is there anything related to these requests in the Apache error_log? That log usually has more in-depth reasons for serving error pages. – Stephen Ostermiller Jun 21 at 0:10
  • In addition to apache error log, may be mod_log_forensic can help. Also, does Bing webmasters console report these errors? – Kannan Jun 21 at 5:04
  • Nothing in the existing error logs although I'll look into turning on additional logging. Everything looks normal in webmaster tools. So this is probably harmless but I'd still like to figure out what's going on. – Displayname71 Jun 21 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.