1

I routinely get hits from Microsoft-owned IP addresses in the 13.x.y.z ranges of AS8075 (in these ranges) that present as Bingbot in the User Agent, but these addresses are listed on several block lists, do not pass the Verify Bingbot tool, and do not have hostnames so can't pass the forward-reverse DNS check. Known legit Microsoft bots in other Microsoft IP ranges typically have bi-directional DNS records linking the IP address to a *.search.msn.com hostname and vice versa.

I want to allow legitimate Microsoft bots to crawl my sites, but by all indications these IPs seem to be sketchy.

My question is: Despite indications, are these legitimate Microsoft bot IP addresses, and if these are legitimate Microsoft bots, why wouldn't Microsoft assign hostnames, clean up the malware, and add them to its Verify tool? If these are not actually legitimate Microsoft bots, why wouldn't Microsoft clean up the machines on their network (if they control them), or enforce some rules preventing bad bots on their IPs (if they are rented out to someone else)?

0

Looking at Microsoft's public IP address space, it looks to me like Microsoft only owns 13.64.0.0/11, 13.96.0.0/13, and 13.104.0.0/14 - they don't own the entire /8 (13.X.X.X).

The 13.X.X.X block is administered by ARIN, and my guess is that it is divided up among other corporations.

So unless I am mistaken, I don't think Microsoft actually owns the IP space you think they do.

  • I am definitely only talking about IP addresses that trace to Microsoft ownership, in these first four ranges: bgp.he.net/AS8075#_prefixes (slightly broader than you posted). IP lookup of each individual IP address shows each one to be owned by Microsoft. – pseudon Jan 28 at 2:28
  • @pseudon That's very interesting. It is possible as you mentioned in your question that these IPs are handed out individually for e.g. Azure applications. However in that case, it would be nearly impossible for Microsoft to enforce any rules on Azure about bot user-agents due to a several technical hurdles, the biggest of which being the TLS encryption of the HTTP request. But I couldn't tell you what would make someone want to spoof these user agents and send their bots to random websites in the first place. – Maximillian Laumeister Jan 28 at 2:53
  • Well, there are all kinds of crawlers with all kinds of motivations (and there are a lot of dumb bots too, easily detectable). I assume spoofing a known good bot is to get past overly simplistic filtering rules. To me, a spoofed bot UA is a strong signal that I should block it. You may be right about these IPs being rented out to customers, but it's really odd to me that Miscosoft wouldn't give these addresses unique subdomains or hostnames (like googleusercontent) to distinguish that they are not officially sanctioned Microsoft-controlled IPs. – pseudon Jan 28 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.