Are there any mechanisms to control what the Internet Archive archives on a site? I know to disallow all pages I could add:

User-agent: ia_archiver
Disallow: /
  1. Can I tell the bot that I want them to crawl my site once a month, or once a year?

  2. I have a site/pages that doesn't/don't get archived correctly because of assets not picked up. Is there a way to tell the Internet Archive bot what assets it needs if it's going to grab the site?


2 Answers 2


Note: This answer is increasingly out-of-date.

The largest contributor to the Internet Archive's web collection has been Alexa Internet. Material that Alexa crawls for its purposes has been donated to IA a few months later. Adding the disallow rule mentioned in the question does not affect those crawls, but the Wayback will 'retroactively' honor them (denying access, the material will still be in the archive - you should exclude Alexa's robot if you really want to keep your material out of the Internet Archive).

There may be ways to affect Alexa's crawls, but I'm not familiar with that.

Since IA developed its own crawler (Heritrix) they have started doing their own crawls, but those tend to be targeted crawls (they do election crawls for Library of Congress and have done national crawls for France and Australia etc.). They do not engage in the kind of sustained world scale crawls that Google and Alexa conduct. IA's largest crawl was a special project to crawl 2 billion pages.

As these crawls are operated on schedules that derive from project specific factors, you can not affect how often they visit your site or if they visit your site.

The only way to directly affect how and when IA crawls your site is to use their Archive-It service. That service allows you to specify custom crawls. The resultant data will (eventually) be incorporated into IA's web collection. This is however a paid subscription service.

  • 3
    Your comment about IA doing their own crawls was true in 2011, and is no longer true in 2016: we do a lot of crawling on our own, now. Commented Jun 20, 2016 at 20:52
  • @GregLindahl you are welcome to add an updated answer to this question Commented Jun 18, 2017 at 10:59

Most search engines support the "Crawl-delay" directive, but I don't know if IA does. You could try it though:

User-agent: ia_archiver
Crawl-delay: 3600

This would limit the delay between requests to 3600 seconds (i.e. 1 hour), or ~700 requests per month.

I don't think #2 is possible - the IA bot grabs the assets as and when it sees fit. It may have a file size limit to avoid using too much storage.

  • @Kris: Setting a crawl-delay should do that by proxy. If you have 30 pages and you tell the crawler to only hit once a day, each page is likely to be refreshed roughly every 30 days. (Not a guarantee, obviously.) Commented Jul 14, 2010 at 9:30
  • In theory yes, however, if you are doing archival crawling you'd never obey such a rule. Crawling a site a document per day means that you can not get a good capture of the site at a specific moment in time. If this attribute was to be respected it would be with an upper limit of 1-5 minutes in any archival crawl.
    – Kris
    Commented Jul 15, 2010 at 22:42
  • Ah OK, I see your point. Commented Jul 15, 2010 at 23:43
  • I've just been looking at the recently released Heritrix 3 and I see that they have added handling of the crawl-delay directive, but it defaults to only respecting a maximum of 300 seconds (5 minutes).
    – Kris
    Commented Jul 21, 2010 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.