1

I have a Magento site and I realize that the ahrefs.com crawler visits my site all the time everyday. I visit ahrefs.com and I check my backlinks (they are good).

  1. Is it good or bad, something like that? (To check all the time). Googlebot doesn't crawl my site all the time.
  2. Maybe it slow down my site? (many requests)
  3. Can I reduce the times it checks?

1 Answer 1

2

From: https://ahrefs.com/robot/

...you can control the frequency of AhrefsBot visiting your site by specifying the minimum acceptable delay between two [consecutive] requests from our bot as follows (in robots.txt):

User-Agent: AhrefsBot 
Crawl-Delay: 2

where the value of Crawl-Delay parameter is time in seconds (you can use fractional seconds too)

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.