Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other robots that do other useful things too.

What's a quick and easy way to make ia_archiver crawl my website more slowly, in order to put less load on the server? I haven't tested the Crawl-delay directive: if you have, and it works, please tell me so. If it doesn't work, please leave a comment. If you've never tested Crawl-delay, then instead please recommend some other solution that takes fifteen minutes or less to implement. Maybe there exists some easy-to-implement software-based solution which will let me throttle too-rapid hits from ia_archiver?

Please assume my website is running on Apache 2.4.3 on Debian Linux 6.0.6 on a dedicated server which I administer.

share|improve this question
    
    
Why haven't you tested the Crawl-delay directive? It's dead simple to implement and Alexa is a responsible company and known to honor robots.txt. I'd be astonished if they didn't honor the crawl-delay directive. – Kris Jan 7 '13 at 16:57
    
@Kris: I actually asked my question to get information for another webmaster: he currently blocks ia_archiver from his massive and useful site. My website is composed of just one solitary webpage (/index.html), so I can't really test Crawl-delay. :) But anyway, the Crawl-delay directive has never been part of the official Robot Exclusion Standard. Even though ia_archiver follows the entire standard, it still might ignore Crawl-delay. – unforgettableid Jan 8 '13 at 8:01
    
How fast is too fast? I've never seen ia_archiver crawl faster than one page every few minutes. – Michael Hampton May 10 '14 at 0:37

Crawling for the Internet Archive is done both by Alex and by the Internet Archive's own crawlers. Support for the Crawl-Delay directive in the robots.txt file is vairly hap-hazard between the two due to the directive not being part of the official robots.txt standard. In addition the way both companies treat the Crawl-Delay directive when they do accept it seems to change over time in my experience. I have tried doing this in the past and have found that sometimes the Crawl-Delay directive has been respected by both, other times only one has respected it, and yet some other times neither has, and there doesn't seem to be a pattern to when it is respected or not. The only thing I can suggest that will definitely work is to add a disallow directive for both the Alexa crawler and the ia_crawler.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.