All Questions
2 questions
15
votes
7
answers
26k
views
How to properly (dis)allow the archive.org bot? Did things change, if so when?
I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:
User-agent: *
Disallow: /
...
- 551
3
votes
1
answer
917
views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
- 99.4k