0

who.is is a service that gives people whois information of websites and archive.org automatically saves people's website. How can I block them using htaccess from getting access with my website?

2

Whois does not crawl your website to gather the info, therefore there is nothing to block. However, you can set your info to private with your registrar.

For the Internet Archive, you can do this:

in .htaccess, you can use this to block bots from accessing your site:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (archive.org_bot) [NC]
RewriteRule .* - [R=403,L]

This will return a 403. I have found the bot name here. The problem with this is that if the bot name is wrong, it will obviously not work.

Blocking crawlers is a job better suited for robots.txt like this:

User-agent: archive.org_bot
Disallow: /

However, Internet Archive says that robots.txt are meant for search engines and might very well be ignored by the Internet Archive.

A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.

Source: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/

Your best bet would be to send them an email to ask them not to list your site.

  • 1
    If archive.org is really disobeying robots.txt then they're becoming more and more of a malicious site. I agree with blocking them through htaccess if they are crawling your site too frequently. I think the internet archive is a good idea though I have found issues with their crawling service as well. – Michael d Feb 19 '18 at 21:28
  • 2
    @michaeld Archive.org has always been rogue. They ignore take down requests and robots.txt file. They talk a good game, but do not do what's right. I fought with them for years. Clear copyright infringement. Hired a lawyer. Put them on notice so that they stay off my site. So far so good. But they will screw up again later. – closetnoc Feb 19 '18 at 21:35
  • I get what you're saying and I am not arguring with you. Their point of view is that the archive is useful: "We feel is it imperative to preserve web data for future generations. Unlike print media, web pages can and do disappear into thin air." Also, they say that you cannot claim that something is private if you make it public on the internet. Again, I totally see your point and I am not trying to spark a debate; I am only trying to provide the right solution to the OP. The point is whether we think it's right or not, they are doing it anyways. – BenoitLussier Feb 19 '18 at 21:38
  • If you want your personal information aatached to your domain name not be visible through a whois request you need to talk to your registrar not registrant. Note that this has nothing to do with website access and crawling. – Patrick Mevzek Feb 19 '18 at 21:51
  • @Patrick Mevzek thank you, I have changed the word in the answer. – BenoitLussier Feb 19 '18 at 22:05
0

You can block archive.org from crawling your site in your robots.txt file with

User-agent: ia_archiver
Disallow: /

I believe this will also block archive.org from accessing your site by putting this in your htaccess file:

SetEnvIfNoCase User-Agent "^ia_archiver" bad_bot

<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.