The Internet Archive's Wayback Machine thinks all my Web site's pages are unreachable: why?

Question

The Internet Archive experienced an attack roughly a month ago and brought all its services offline. Roughly last week, they brought the Wayback Machine's “save page now” service back online. I have since then used it successfully to save a number of Web pages, so it does see to be working.

But for some reason, ever since then [edit: see update below], the Wayback Machine thinks my personal Web site's pages are all “unreachable”: if I go to the save page now page of the Archive and enter a valid URL from my site, say http://www.madore.org/~david/?1731283103 (the final number here is just to make sure that the result does not get cached anywhere), the Wayback Machine spends roughly a dozen seconds spinning and finally tells me:

Saving page http://www.madore.org/~david/?1731283103

Save Page Now could not capture this URL because it was unreachable.

Return to Save Page Now

But the address is indeed reachable (if you wish to test, please use a different number). In fact, the Internet archive did access it according to my server logs:

207.241.225.61 - - [11/Nov/2024:00:59:44 +0100] "GET /~david/?1731283103 HTTP/1.1" 200 23461 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/605.1.15"

The same thing seems to happen for every single page of my site. Other sites have no such problem, and I have no idea what the difference might be.

Google seems to have no problem crawling my site, for example. The problem is specific to the Internet Archive.

What might possibly be going on here? How could I try to debug this?

I have tried contacting staff at the Internet Archive, but with no success (I imagine they are understaffed and overworked, so I can't blame them).

---

✱ Edit (2024-11-11): I have now performed further experiments which help explain the problem.

A tcpdump of a conversation between my server and the Wayback Machine when I request a save page now goes like this:

• one of the Archive's machines performs performs an HTTP GET request on port 80 of my server,

• my server starts sending data, and the Archive's machine ACKs the beginning of it,

• roughly 260ms after that connection was initiated, and with typically about 14kb ACKed (of about 22kb sent), the Archive's machine suddenly sends a TCP RST, aborting the connection;

• independently of this, several seconds before or after the connection to port 80 was made (and aborted by the Archive), another of one of the Archive's machines attempts to contact my server on the HTTPS port (443), and fails as the latter is not opened.

^{(I uploaded a typical full tcpdump output here: my server is 163.172.82.209 and the Archive machine doing the HTTP connection is 207.241.225.229 in this example; also, in this example, an HTTPS connection was attempted 3s after the HTTP one, but in other cases it comes before.)}

So it seems to be a timeout issue. To confirm that this is so, I created a very small text file http://www.madore.org/~david/.tmp/test.txt and I was indeed able to archive it. It appears that the Wayback Machine just gives up on the request after roughly a quarter of a second, and closes the connection.

But a quarter of a second seems very harsh for a timeout on an HTTP request, doesn't it? Wouldn't many other sites be affected?

My working hypothesis at this stage is something like this: the Archive dramatically decreased some timeout following the attack, and either they re-increased it later but only for HTTPS, or else for some reason this timeout only affects HTTP, and since nearly everyone uses HTTPS now, nobody noticed that this nearly-but-not-completely breaks archiving of HTTP pages (all those that do not respond within ~250ms).

Short of getting the Archive administrators' attention (if someone knows how I might do this, please comment!), I think there's very little I can do at this point except pivot to HTTPS and hope the timeout issue doesn't affect it. (I'll try to dive into TCP options to see if any might help, though.)

---

✱ Update (2024-11-17): One week later (and just as I was preparing experiments to compare the effect of HTTP and HTTPS), the behavior is different: the Wayback Machine no longer reports my pages as “unreachable” — it appears to save them properly. However, the saved pages still don't appear in the archive's itself, so there is still something broken. The change is probably an improvement, though, and leads me to suppose that someone is aware of problems and working on them (and it's certainly not “my fault” or something I can do anything about). I will edit this question again if and when it appears that everything is back to normal.

Answer 1

I'm not sure, but may be it's because the Internet Archive is trying to access your site with https. I've tried another http site, and it has been transformed to https. And as your site is not reachable in https, it could be the reason.

Answer 2

Short-version, I couldn't find anything conclusive, but maybe you could check your robots.txt file(s)

What I did find is that your default robots.txt blocks /~david// (yes, with two slashes), but this shouldn't block your website because of the double slashes.

User-Agent: *
# other Disallows
Disallow: /~david//
Disallow: /%7edavid//

However, I can't know if it changed recently. So maybe it had only one slash before.

But, your robots.txt at www4 blocks / for UserAgent *, so this could explain why using www4 sub-domain is not working.

User-Agent: *
Disallow: /

My recommendation would be to 'soften' your robots.txt rules and test again.

Also, I don't know if there's any rules that could be serving different robots.txt other than those I could access directly from my browser (something like Rewrite Rules), so that's worth checking out too.

Bottom-line, I'd say to investigate if it could be related to robots.txt