Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a Magento site and I realize that the ahrefs.com crawler visits my site all the time everyday. I visit ahrefs.com and I check my backlinks (they are good).

  1. Is it good or bad, something like that? (To check all the time). Googlebot doesn't crawl my site all the time.
  2. Maybe it slow down my site? (many requests)
  3. Can I reduce the times it checks?
share|improve this question
add comment

1 Answer

From: https://ahrefs.com/robot/

...you can control the frequency of AhrefsBot visiting your site by specifying the minimum acceptable delay between two [consecutive] requests from our bot as follows (in robots.txt):

User-Agent: AhrefsBot 
Crawl-Delay: 2

where the value of Crawl-Delay parameter is time in seconds (you can use fractional seconds too)

share|improve this answer
    
Thank you very much for your answer! –  Andrew May 26 '13 at 11:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.