I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:

User-agent: *
Disallow: /

Today, according to archive.org I have to add the following in my robots.txt to allow their bots:

User-agent: ia_archiver
Disallow:

But, I already had done what they indicated a couple of years ago, at least, I added the following:

User-agent: archive.org_bot
Disallow:

Then there's another source claiming that you have to add the two above Disallows, plus another one:

User-agent: ia_archiver-web.archive.org 
Disallow:

Note that you need to put Disallow: / if you don't want the bot to archive your site.

Has there been a change with the IA bot? If so, when?

What is the recommended way? Should I just allow all three for now and hope that IA will not change their bot name again in the future?

  • I was only aware of ia_archiver. The others are a surprise to me. Do you have links for this? The reason I ask is that archive.org visits my site anyway and I had to block them by IP address. You also mention that you Do want to allow archive.org but then talk about blocking it. I just want to be clearer on this and the links can help us all. Thanks in Advance! – closetnoc Oct 25 '14 at 15:42
  • Updated the question. Hope it's clearer now. The tiny version: I don't want search engines bots on this site, I do want archive.org bots. But maybe I should reverse the question since that's what most people are looking for? – guaka Oct 25 '14 at 15:51
  • Actually, if you do not use any of these in, you are allowing archive.org providing that you are not blocking with a blanket statement. – closetnoc Oct 25 '14 at 16:09
  • Using just "ia_archiver" should also block "ia_archiver-web.archive.org", so the later would seem to be unnecessary (providing this bot follows the standard). – MrWhite Oct 26 '14 at 0:26
  • Do you see the ia-archiver (or archive.org_bot) bot in your access logs? – MrWhite Oct 26 '14 at 0:38

Update: As @KevinFegan notes in the comments, their documentation changed. The below part describes how the Internet Archive handled it in the past (at least in 2014).


Their FAQ How can I have my site's pages excluded from the Wayback Machine? refers to Removing Documents From the Wayback Machine, which documents that their bot is called ia_archiver.

So this record should allow their bot to crawl your entire site:

User-agent: ia_archiver
Disallow:
  • The order of the groups should not matter. The most specific (ie. longest) user agent that matches is the one that wins. The * group only matches when no other group has matched. – MrWhite Oct 26 '14 at 0:18
  • @w3d: You’re right, I removed this part. Thanks for the info :) – unor Oct 26 '14 at 14:27
  • 1
    Apparently, this has changed over time. I can't find "ia_archiver" on the FAQ Page you provided, and on this Archive.org Blog page from April 25, 2017, Mark Graham says: The “ia_archiver” User Agent is used by Alexa Internet, not the Internet Archive. – Kevin Fegan May 17 '17 at 14:28
  • @KevinFegan: Thanks for your notice! I updated my answer to link to archived versions of the documentation that contained the name. – unor Jun 15 '17 at 23:03

There are really 2 issues here:

  1. Will the robots.txt on your site Disallow (block) Wayback from crawling your site.
  2. Will Wayback crawl your site.

For point #1:
As others have said, the correct entry for robots.txt is:

User-agent: ia_archiver
Disallow:

Keep in mind that it might take a while (perhaps a good long while), for Wayback to notice any changes you have made to robots.txt.

To check if the robots.txt on your site will allow Wayback to crawl your site:

  1. Go to this URL: https://archive.org/web/
  2. In the box at the TOP of the page, enter the URL of a page on your site, and click the "Browse History" button.
  3. Or, in the box under "Save Page Now" (currently near the bottom on the right), and enter the URL of a page on your site, and click the "Save Page" button.

At this point, you should see 1 of 3 things:

  1. You will see an error message indicating that Wayback can't access pages on that site due to "robots.txt".
  2. You will see the "calendar" of historical save points for the page on your site. In this case, you know that Wayback is NOT blocked from crawling your site.
  3. Or, you will see a message indicating that Wayback doesn't have an archive of that page, and an offer to click a link to add the page to Wayback. In this case also, you know that Wayback is NOT blocked from crawling your site.


Now, for point #2:

Will Wayback crawl your site?

Just because you Allow Wayback to crawl your site, doesn't mean that they (ever) will crawl your site.

According to the Wayback FAQ (emphasis added):

How can I get my site included in the Wayback Machine?

Much of our archived web data comes from our own crawls or from Alexa Internet's crawls. Neither organization has a "crawl my site now!" submission process. Internet Archive's crawls tend to find sites that are well linked from other sites. The best way to ensure that we find your web site is to make sure it is included in online directories and that similar/related sites link to you.

Alexa Internet uses its own methods to discover sites to crawl. It may be helpful to install the free Alexa toolbar and visit the site you want crawled to make sure they know about it.

Regardless of who is crawling the site, you should ensure that your site's 'robots.txt' rules and in-page META robots directives do not tell crawlers to avoid your site.


Update: 09-May-2017

Others have left comments/answers indicating that Archive.org no longer honors robots.txt. Perhaps this is a "work-in-progress" and it will eventually be the case, but I have not seen this new behavior yet.

The case for this seems to come from this article: Robots.txt: ROBOTS.TXT IS A SUICIDE NOTE by archiveteam.org. While that page has little if anything good to say about "Robots.txt", it doesn't mention anywhere that Archive.org will no longer honor robots.txt.

Also of note: that article is hosted on archiveteam.org, which is most definitely not archive.org, and I'm not sure there is any (official) relationship between archive.org and archiveteam.org.

In fact, this page on archive.org about Archive Team, seems to declare a distinction between archive.org and archiveteam.org (emphasis added):

Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. ...

In any case, I decided to give this a try, and I found that, at least at this time, Archive.org STILL honors robots.txt:

  1. I found a random item on eBay: Item #: 131795294232
  2. Click to view the sold items:

eBay items sold

  1. The "Items sold" page opens: http://offer.ebay.com/ws/eBayISAPI.dll?ViewBidsLogin&item=131795294232 Copy the link to the clipboard.
  2. Goto web.archive.org, and paste the link from eBay.
  3. You will see that archive.org indicates that the "Page cannot be displayed due to robots.txt."

Wayback/Archive.org "error" message

So, at this time, I remain unconvinced, but I would love to be proven wrong... it would be great if it were true.

  • Locking out archive.org with robots.txt won't work anymore: – wortwart May 8 '17 at 15:06
  • @wortwart - That would be great if it were so (see the update I added to my answer). Do you have links to info about this? – Kevin Fegan May 10 '17 at 5:28
  • Sure: blog.archive.org/2017/04/17/… "A few months ago we stopped referring to robots.txt files on U.S. government and military web sites (...) We are now looking to do this more broadly." – wortwart May 10 '17 at 8:04
  • Archive bot does not render all pages like Googlebot and other crawler do, because it's require too much resource power. They have simple bot, which stop crawling when they see some pages which look like auto generated and may require too much CPU resource and may stuck into loop, so such a pages is not render by Archive and hence they don't index it. – Goyllo Jun 19 '17 at 6:32

The robots.txt ia_archiver Disallow entry (with the "/") should be fine for the need you describe (to "preserve for eternity", but not yet publicly).

I just did a quick test, commenting out the ia_archiver Disallow entry for a site that had it for at least the past 10 years. Then I looked the site up on archive.org/web, and it showed up grabs it had collected in 2007, 2008, 2009, 2011, 2012, 2013, 2014, 2015, 2016 and 2017! This means that Archive.org never strictly honored what others thought to be a "do not archive" statement during these years, it was merely not exposing the archived copies.

Update 2017

Archive bot now does not care about your robots.txt.

If you really want to block it, send them a email according to this page, or block their IP address via htaccess.

  • 2
    See the May 2017 update to my answer: How to properly (dis)allow the archive.org bot?.... The Archive bot still cares about the robots.txt file, except for government websites. Notice the article you mentioned was from www.archiveteam.org, which is not related to Archive.org. ---> – Kevin Fegan Jun 18 '17 at 23:25
  • ---> While that page has little if anything good to say about "Robots.txt", it doesn't mention anywhere that Archive.org will no longer honor robots.txt. The relevant Archive.org article is: Robots.txt meant for search engines don’t work well for web archives. "A few months ago we stopped referring to robots.txt files on U.S. government and military web sites (...) We are now looking to do this more broadly." – Kevin Fegan Jun 18 '17 at 23:32
  1. "ia_archiver" is now (ab)used by Alexa, some sources say: 1, 2.
  2. Archive.org now (2018) does NOT respect the "robots.txt" any more at all.3 Not only for mil/gov pages, but for all pages. As experienced with my own private website, which has and had an ia-excluding robots.txt since 2012; and now I suddenly found out it has been crawled and saved by them all the years and now the whole history is visible. It's a feeling of being betrayed. >:-(

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.