Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [internet-archive]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
3 votes
1 answer

Can's Wayback Machine ignore some query terms?

I'd have a single page web application that uses both query terms to control content shown to users as well as target-style hash fragments, which prevents encoding the query terms in the hash. An ...
Mike Godin's user avatar
1 vote
1 answer

Can I tell who is accessing archived documents from my site from my server logs?

I currently host a website with a collection of some of my projects, where a few of them are currently undergoing a patent review process. I don't want these documents to be public (though they were ...
user avatar
1 vote
0 answers

Is there any way to keep a link unmodified when crawled by Archive-It?

We are using Archive-It's "Standard" seed type (see, and wish to keep some links, particularly to ...
Mike Godin's user avatar
4 votes
2 answers

Is there any SEO benefit in having a site archived by's "Wayback Machine"?

Is there any SEO benefit of having a site frequently archived by the Wayback Machine vs. being completely excluded?
Michael Rogers's user avatar
2 votes
0 answers

Why does Google AdSense see content taken from my old site on the Internet Archive as plagiarized?

Some years back I hosted a blog but unfortunately the blog wen off, so later I hosted another blog with a different domain because the older one has been taken. So I grabbed all my content from http:/...
Akorede Erisan's user avatar
2 votes
1 answer

Using the Internet Archive’s Wayback Machine as a CDN

Can the Internet Archive’s Wayback Machine be used as a content-distribution network (CDN)? Aside from the moral problem of stealing bandwidth from an organization I love, would serving a unique file ...
Lucas's user avatar
  • 197
11 votes
2 answers

What determines the frequency the Wayback Machine crawls one's website?

I understand takes its website list to crawl from Alexa, but I don't understand how it decides the snapshot frequencies for each website. We can see some websites are crawled multiple ...
Manu H's user avatar
  • 213
13 votes
6 answers

How to properly (dis)allow the bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on So my robots.txt starts with this: User-agent: * Disallow: / ...
the's user avatar
  • 1,570
1 vote
3 answers

How to prevent Alexa and Google from indexing our stuff?

My traffic is not from search engine and I want to prevent bots or others from displaying or accessing it. What would be a good .htaccess code to do so? I've heard User-agent: ia_archiver Disallow:...
user4951's user avatar
  • 385
3 votes
1 answer

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
unforgettableidSupportsMonica's user avatar
14 votes
2 answers

Control over the Internet Archive besides just "Disallow /"?

Are there any mechanisms to control what the Internet Archive archives on a site? I know to disallow all pages I could add: User-agent: ia_archiver Disallow: / Can I tell the bot that I want them to ...
artlung's user avatar
  • 8,648