When Bing requests a sitemap or a robots.txt, why do I see Apache reject the 1st request with a 403, followed immediately by a 2nd successful request?

Question

This is a small issue that has been ongoing for quite some time.

When I check my Apache server log for 403's, all entries are related to Bing, and all from when it requests a robots.txt, sitemap.txt, or sitemap.xml.

The problem only happens with Bing and only when they request a sitemap or robots.txt. The problem does not happen with actual Bing crawling, and even if I do an URL inspection / live test on one of the robots.txt or sitemap files, the problem does NOT happen that way.

What I'm seeing is that Apache will reject the first request with a 403, and then there will immediately be a 2nd request that will be successful. So it's pretty harmless except for junking up the logs.

Here are some examples of what I see in the logs:

[2021-06-20/15:11] (mydomain1).org 207.46.13.96 "GET /sitemap.txt HTTP/2.0" 403 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
[2021-06-20/15:11] (mydomain1).org 207.46.13.96 "GET /sitemap.txt HTTP/2.0" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

[2021-06-20/07:36] (mydomain2).net 207.46.13.45 "GET /robots.txt HTTP/2.0" 403 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
[2021-06-20/07:36] (mydomain2).net 207.46.13.45 "GET /robots.txt HTTP/2.0" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

[2021-06-20/15:13] (mydomain3).com 40.77.167.21 "GET /sitemap.xml HTTP/2.0" 403 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
[2021-06-20/15:13] (mydomain3).com 40.77.167.21 "GET /sitemap.xml HTTP/2.0" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Interestingly if I route the traffic through Cloudflare the problem goes away (or maybe the initial bad request never reaches my server and I only see the 2nd good request).

Cloudflare examples:

[2021-06-20/15:11] (mydomain4).tk 2400:cb00:28:1024::6ca2:f654 "GET /sitemap.txt HTTP/1.1" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

[2021-06-20/15:12] (mydomain4).tk 2400:cb00:28:1024::6ca2:f6c8 "GET /sitemap.xml HTTP/1.1" 200 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

I have mod_security running but it's in DETECTION_ONLY mode and shouldn't be blocking anything on its own, just logging.

When one of these 403's happens, I get an entry in modsec_audit.log, because it logs all 403's, but it's not very helpful:

--9ea2ef5a-A--
[20/Jun/2021:15:11:29 --0500] YM@g8YvFIvJDLJuWINL3ewAAAAE 40.77.167.21 53442 (myIP) 443
--9ea2ef5a-B--
GET /sitemap.xml HTTP/2.0
Cache-Control: no-cache
Pragma: no-cache
Accept: */*
Accept-Encoding: gzip, deflate
From: bingbot(at)microsoft.com
User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Host: (mydomain1).com
--9ea2ef5a-H--
Stopwatch: 1624219889292746 4001 (- - -)
Stopwatch2: 1624219889292746 4001; combined=1982, p1=1533, p2=0, p3=0, p4=0, p5=443, sr=492, sw=6, l=0, gc=0
Producer: ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/); OWASP_CRS/3.2.0.
Server: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f
Engine-Mode: "DETECTION_ONLY"
--9ea2ef5a-Z--