Consequences of blocking Semrush and other bots?

Question

For typical websites, are there any disadvantages to my clients to blocking spiders like Semrush, Maxmind and the plethora of other "non search-engine" bots?

Wouldn't blocking these significantly reduce competitive analysis and provide a cheap net-benefit for my clients? Any ideas why this kind of thing is not a fairly standard service offered by web hosting providers?

Or, put another way, what benefits do Semrush, AHRef and the like give to my clients (assuming they are not clients of Semrush or AHRef)? Do "generic links" created by these sites really help SEO?

(FWIW, this is not a bandwidth thing, and Will blocking bots other than those from the major search engines affect SEO? does not really seem to be quite on point enough)

Answer 1

It depends entirely on what you want or need from these tools.

From the Search Engine's point of view, they don't rely on external tools at all. They don't consider Ahrefs, Semrush, Moz, or whatever other tool's spider gets to your site as part of their ranking algorithms.

From an SEO stand, any link from that tool is probably nofollow, so it's not a super link to help you.

Blocking them on your site has some minimal benefits such as:

• Bandwidth saving
• CPU saving, as sometimes an aggressive scan on peak hours can temporarily hurt your site
• Blind your competitors, as they won't be able to use some of the tools' intelligence gathering against you

Allowing them on your site enables you to:

• Gather insights on your sites' configuration and performance
• Leverage the tools' powers to find opportunities for improvement

Some tools are more aggressive than others. Others are downright harmful, as they try to scrape your site, duplicate your content, etc. YMMV