4

I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as

AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/7~bl; +http://www.semrush.com/bot.html
Neevabot/1.0; +https://neeva.com/neevabot

I don't use their services, and thus, it has no use for me to be included in their system.

Does crawling by those bots have any benefit to my website popularity or should I block them to save unnecessary traffic?

6

If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic search (or elsewhere). All three services respect the robots.txt file, so you could simply disallow it there.

User-agent: AhrefsBot
Disallow: /

User-agent: Neevabot
Disallow: /

User-agent: SemrushBot
Disallow: /

The above shows this as separate statements in case you wanted to block one bot but not the others (if, for example, you were using SEMRush and not the others). As noted in the comments, you could also combine these into a single statement if you wanted to block all three.

User-agent: AhrefsBot
User-agent: Neevabot 
User-agent: SemrushBot
Disallow: /
1
  • 3
    You could combine those 3 groups into a single group. ie. 3 User-agent: directives followed by a single Disallow: / directive.
    – MrWhite
    Aug 5 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.