Questions tagged [internet-archive]
The internet-archive tag has no usage guidance.
6
questions
1
vote
0answers
79 views
Using the Internet Archive’s Wayback Machine as a CDN
Can the Internet Archive’s Wayback Machine be used as a content-distribution network (CDN)?
Aside from the moral problem of stealing bandwidth from an organization I love, would serving a unique file ...
6
votes
2answers
2k views
What determines the frequency the Wayback Machine crawls one's website?
I understand archive.org takes its website list to crawl from Alexa, but I don't understand how it decides the snapshot frequencies for each website. We can see some websites are crawled multiple ...
10
votes
5answers
14k views
How to properly (dis)allow the archive.org bot? Did things change, if so when?
I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:
User-agent: *
Disallow: /
...
1
vote
3answers
1k views
How to prevent Alexa and Google from indexing our stuff?
My traffic is not from search engine and I want to prevent bots or others from displaying or accessing it.
What would be a good .htaccess code to do so?
I've heard
User-agent: ia_archiver
Disallow:...
3
votes
1answer
770 views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
13
votes
2answers
418 views
Control over the Internet Archive besides just “Disallow /”?
Are there any mechanisms to control what the Internet Archive archives on a site? I know to disallow all pages I could add:
User-agent: ia_archiver
Disallow: /
Can I tell the bot that I want them to ...
