Questions tagged [internet-archive]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
11
votes
6answers
17k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
14
votes
2answers
487 views

Control over the Internet Archive besides just “Disallow /”?

Are there any mechanisms to control what the Internet Archive archives on a site? I know to disallow all pages I could add: User-agent: ia_archiver Disallow: / Can I tell the bot that I want them to ...
3
votes
1answer
809 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...