Sign up ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a Magento site and I realize that the crawler visits my site all the time everyday. I visit and I check my backlinks (they are good).

  1. Is it good or bad, something like that? (To check all the time). Googlebot doesn't crawl my site all the time.
  2. Maybe it slow down my site? (many requests)
  3. Can I reduce the times it checks?
share|improve this question

1 Answer 1

From: can control the frequency of AhrefsBot visiting your site by specifying the minimum acceptable delay between two [consecutive] requests from our bot as follows (in robots.txt):

User-Agent: AhrefsBot 
Crawl-Delay: 2

where the value of Crawl-Delay parameter is time in seconds (you can use fractional seconds too)

share|improve this answer
Thank you very much for your answer! – Andrew May 26 '13 at 11:23

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.