4

I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as

AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/7~bl; +http://www.semrush.com/bot.html
Neevabot/1.0; +https://neeva.com/neevabot

I don't use their services, and thus, it has no use for me to be included in their system.

Does crawling by those bots have any benefit to my website popularity or should I block them to save unnecessary traffic?

2 Answers 2

7

If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic search (or elsewhere). All three services respect the robots.txt file, so you could simply disallow it there.

User-agent: AhrefsBot
Disallow: /

User-agent: Neevabot
Disallow: /

User-agent: SemrushBot
Disallow: /

The above shows this as separate statements in case you wanted to block one bot but not the others (if, for example, you were using SEMRush and not the others). As noted in the comments, you could also combine these into a single statement if you wanted to block all three.

User-agent: AhrefsBot
User-agent: Neevabot 
User-agent: SemrushBot
Disallow: /
1
  • 3
    You could combine those 3 groups into a single group. ie. 3 User-agent: directives followed by a single Disallow: / directive.
    – MrWhite
    Aug 5, 2021 at 22:28
3

Full disclosure: I work at Neeva.

Neevabot is used by neeva.com, a web search engine and is not an SEO analysis bot. Details at https://neeva.com/neevabot

If you're not happy with the rate or any aspect of how Neevabot is behaving with your site, please do reach out to the contacts mentioned above instead of blocking it.

Thanks very much!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.