who.is is a service that gives people whois information of websites and archive.org automatically saves people's website. How can I block them using htaccess from getting access with my website?

2 Answers 2


Whois does not crawl your website to gather the info, therefore there is nothing to block. However, you can set your info to private with your registrar.

For the Internet Archive, you can do this:

in .htaccess, you can use this to block bots from accessing your site:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (archive.org_bot) [NC]
RewriteRule .* - [R=403,L]

This will return a 403. I have found the bot name here. The problem with this is that if the bot name is wrong, it will obviously not work.

Blocking crawlers is a job better suited for robots.txt like this:

User-agent: archive.org_bot
Disallow: /

However, Internet Archive says that robots.txt are meant for search engines and might very well be ignored by the Internet Archive.

A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to [email protected]). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.

Source: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/

Your best bet would be to send them an email to ask them not to list your site.

  • 1
    If archive.org is really disobeying robots.txt then they're becoming more and more of a malicious site. I agree with blocking them through htaccess if they are crawling your site too frequently. I think the internet archive is a good idea though I have found issues with their crawling service as well.
    – Michael d
    Feb 19, 2018 at 21:28
  • 2
    @michaeld Archive.org has always been rogue. They ignore take down requests and robots.txt file. They talk a good game, but do not do what's right. I fought with them for years. Clear copyright infringement. Hired a lawyer. Put them on notice so that they stay off my site. So far so good. But they will screw up again later.
    – closetnoc
    Feb 19, 2018 at 21:35
  • I get what you're saying and I am not arguring with you. Their point of view is that the archive is useful: "We feel is it imperative to preserve web data for future generations. Unlike print media, web pages can and do disappear into thin air." Also, they say that you cannot claim that something is private if you make it public on the internet. Again, I totally see your point and I am not trying to spark a debate; I am only trying to provide the right solution to the OP. The point is whether we think it's right or not, they are doing it anyways. Feb 19, 2018 at 21:38
  • If you want your personal information aatached to your domain name not be visible through a whois request you need to talk to your registrar not registrant. Note that this has nothing to do with website access and crawling. Feb 19, 2018 at 21:51
  • @Patrick Mevzek thank you, I have changed the word in the answer. Feb 19, 2018 at 22:05

You can block archive.org from crawling your site in your robots.txt file with

User-agent: ia_archiver
Disallow: /

I believe this will also block archive.org from accessing your site by putting this in your htaccess file:

SetEnvIfNoCase User-Agent "^ia_archiver" bad_bot

Order Allow,Deny
Allow from all
Deny from env=bad_bot
  • Do you have an official / up-to-date reference for the use of the ia_archiver user-agent (and robots.txt)? I believe this used to be correct, however, the relevant page on the archive.org website appears to have been deleted sometime between 31 Oct 2015 and 14 Nov 2015. Ironically, the page is available in the "wayback machine": web.archive.org/web/20151031123632/https://archive.org/about/…
    – MrWhite
    Feb 20, 2018 at 0:04
  • It may be User-agent: archive.org_bot as another post suggested
    – Michael d
    Feb 20, 2018 at 0:24
  • It looks like they document the new User agent here: archive.org/details/archive.org_bot Feb 21, 2018 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.