1

I have a Magento site and I realize that the ahrefs.com crawler visits my site all the time everyday. I visit ahrefs.com and I check my backlinks (they are good).

  1. Is it good or bad, something like that? (To check all the time). Googlebot doesn't crawl my site all the time.
  2. Maybe it slow down my site? (many requests)
  3. Can I reduce the times it checks?
2

From: https://ahrefs.com/robot/

...you can control the frequency of AhrefsBot visiting your site by specifying the minimum acceptable delay between two [consecutive] requests from our bot as follows (in robots.txt):

User-Agent: AhrefsBot 
Crawl-Delay: 2

where the value of Crawl-Delay parameter is time in seconds (you can use fractional seconds too)

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.