Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Are there any mechanisms to control what the Internet Archive archives on a site? I know to disallow all pages I could add:

User-agent: ia_archiver
Disallow: /
  1. Can I tell the bot that I want them to crawl my site once a month, or once a year?

  2. I have a site/pages that doesn't/don't get archived correctly because of assets not picked up. Is there a way to tell the Internet Archive bot what assets it needs if it's going to grab the site?

share|improve this question
    
I'm also very interested in answers to this. +1 :) –  Tim Post Jul 11 '10 at 19:08
1  
add comment

2 Answers

up vote 6 down vote accepted

The largest contributor to the Internet Archive's web collection has been Alexa Internet. Material that Alexa crawls for its purposes has been donated to IA a few months later. Adding the disallow rule mentioned in the question does not affect those crawls, but the Wayback will 'retroactively' honor them (denying access, the material will still be in the archive - you should exclude Alexa's robot if you really want to keep your material out of the Internet Archive).

There may be ways to affect Alexa's crawls, but I'm not familiar with that.

Since IA developed its own crawler (Heritrix) they have started doing their own crawls, but those tend to be targeted crawls (they do election crawls for Library of Congress and have done national crawls for France and Australia etc.). They do not engage in the kind of sustained world scale crawls that Google and Alexa conduct. IA's largest crawl was a special project to crawl 2 billion pages.

As these crawls are operated on schedules that derive from project specific factors, you can not affect how often they visit your site or if they visit your site.

The only way to directly affect how and when IA crawls your site is to use their Archive-It service. That service allows you to specify custom crawls. The resultant data will (eventually) be incorporated into IA's web collection. This is however a payed subscription service.

share|improve this answer
add comment

Most search engines support the "Crawl-delay" directive, but I don't know if IA does. You could try it though:

User-agent: ia_archiver
Crawl-delay: 3600

This would limit the delay between requests to 3600 seconds (i.e. 1 hour), or ~700 requests per month.

I don't think #2 is possible - the IA bot grabs the assets as and when it sees fit. It may have a file size limit to avoid using too much storage.

share|improve this answer
    
@Kris: Setting a crawl-delay should do that by proxy. If you have 30 pages and you tell the crawler to only hit once a day, each page is likely to be refreshed roughly every 30 days. (Not a guarantee, obviously.) –  DisgruntledGoat Jul 14 '10 at 9:30
    
In theory yes, however, if you are doing archival crawling you'd never obey such a rule. Crawling a site a document per day means that you can not get a good capture of the site at a specific moment in time. If this attribute was to be respected it would be with an upper limit of 1-5 minutes in any archival crawl. –  Kris Jul 15 '10 at 22:42
    
Ah OK, I see your point. –  DisgruntledGoat Jul 15 '10 at 23:43
    
I've just been looking at the recently released Heritrix 3 and I see that they have added handling of the crawl-delay directive, but it defaults to only respecting a maximum of 300 seconds (5 minutes). –  Kris Jul 21 '10 at 10:18
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.