10

Update: As @KevinFegan notes in the comments, their documentation changed. The below part describes how the Internet Archive handled it in the past (at least in 2014). Their FAQ How can I have my site's pages excluded from the Wayback Machine? refers to Removing Documents From the Wayback Machine, which documents that their bot is called ia_archiver. So ...


7

The Wayback Machine archive is a combination of data from a large number of different crawls: Alexa crawls, which appear after a 6 month delay Our own crawls, which are seeded from the Alexa top million list and others ArchiveTeam crawls, done by volunteers ArchiveIt crawls, done by our 400+ partners, mostly libraries, many of which allow their data to be ...


6

No. In fact the Internet Archive is not a CDN. Following your own link we see A content delivery network, or content distribution network (CDN), is a geographically distributed network of proxy servers and their data centers. The goal is to provide high availability and performance by distributing the service spatially relative to end users But while the ...


5

There are really 2 issues here: Will the robots.txt on your site Disallow (block) Wayback from crawling your site. Will Wayback crawl your site. For point #1: As others have said, the correct entry for robots.txt is: User-agent: ia_archiver Disallow: Keep in mind that it might take a while (perhaps a good long while), for Wayback to notice any changes ...


4

The robots.txt ia_archiver Disallow entry (with the "/") should be fine for the need you describe (to "preserve for eternity", but not yet publicly). I just did a quick test, commenting out the ia_archiver Disallow entry for a site that had it for at least the past 10 years. Then I looked the site up on archive.org/web, and it showed up grabs it had ...


4

Update 2017 Archive bot now does not care about your robots.txt. If you really want to block it, send them a email according to this page, or block their IP address via htaccess.


3

I tried the robots.txt method and it didn't work. So I contacted the website on their email info@archive.org: Hello, Can you please remove my personal website dimitarnestorov.com from your archive? Thanks! Dimitar And I got the following answer: Hello, The Internet Archive can exclude websites from the Wayback Machine (web....


3

"ia_archiver" is now (ab)used by Alexa, some sources say: 1, 2. Archive.org now (2018) does NOT respect the "robots.txt" any more at all.3 Not only for mil/gov pages, but for all pages. As experienced with my own private website, which has and had an ia-excluding robots.txt since 2012; and now I suddenly found out it has been crawled and saved by them all ...


2

Use a robots.txt file to prevent spiders from crawling and indexing your content. From Alexa's "for webmasters" page: User-agent: ia_archiver Disallow: / From Googe's robots.txt help page: User-Agent: Googlebot Disallow: / If you want to block all robots and prevent all spiders from crawling and indexing your content: User-Agent: * Disallow: / Alexa ...


1

There's some information about this on Wikipedia, Snapshots usually become available more than 6 months after they are archived or in some cases even later, 24 months or longer. The frequency of snapshots is variable, so not all tracked web site updates are recorded. There are sometimes intervals of several weeks or years between snapshots. https://en....


1

Crawling for the Internet Archive is done both by Alex and by the Internet Archive's own crawlers. Support for the Crawl-Delay directive in the robots.txt file is vairly hap-hazard between the two due to the directive not being part of the official robots.txt standard. In addition the way both companies treat the Crawl-Delay directive when they do accept it ...


Only top voted, non community-wiki answers of a minimum length are eligible