Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a Magento site and I realize that the ahrefs.com crawler visits my site all the time everyday. I visit ahrefs.com and I check my backlinks (they are good).

  1. Is it good or bad, something like that? (To check all the time). Googlebot doesn't crawl my site all the time.
  2. Maybe it slow down my site? (many requests)
  3. Can I reduce the times it checks?
share|improve this question

From: https://ahrefs.com/robot/

...you can control the frequency of AhrefsBot visiting your site by specifying the minimum acceptable delay between two [consecutive] requests from our bot as follows (in robots.txt):

User-Agent: AhrefsBot 
Crawl-Delay: 2

where the value of Crawl-Delay parameter is time in seconds (you can use fractional seconds too)

share|improve this answer
Thank you very much for your answer! – Andrew May 26 '13 at 11:23

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.