Unfortunately, our hosting provider experienced 100% data loss, so I've lost all content for two hosted blog websites:

(Yes, yes, I absolutely should have done complete offsite backups. Unfortunately, all my backups were on the server itself. So save the lecture; you're 100% absolutely right, but that doesn't help me at the moment. Let's stay focused on the question here!)

I am beginning the slow, painful process of recovering the website from web crawler caches.

There are a few automated tools for recovering a website from internet web spider (Yahoo, Bing, Google, etc.) caches, like Warrick, but I had some bad results using this:

  • My IP address was quickly banned from Google for using it
  • I get lots of 500 and 503 errors and "waiting 5 minutes…"
  • Ultimately, I can recover the text content faster by hand

I've had much better luck by using a list of all blog posts, clicking through to the Google cache and saving each individual file as HTML. While there are a lot of blog posts, there aren't that many, and I figure I deserve some self-flagellation for not having a better backup strategy. Anyway, the important thing is that I've had good luck getting the blog post text this way, and I am definitely able to get the text of the web pages out of the Internet caches. Based on what I've done so far, I am confident I can recover all the lost blog post text and comments.

However, the images that go with each blog post are proving…more difficult.

Any general tips for recovering website pages from Internet caches, and in particular, places to recover archived images from website pages?

(And, again, please, no backup lectures. You're totally, completely, utterly right! But being right isn't solving my immediate problem… Unless you have a time machine…)

  • 100
    When somebody like Jeff Atwood himself can lose two entire websites in one fell swoop... Well. I'm going to review my own backup procedures, for one :P
    – Phoshi
    Dec 11 '09 at 21:10
  • 243
    @Phoshi: Jeff has some good articles on Coding Horror on backup. You should give them a quick read.
    – joshhunt
    Dec 11 '09 at 21:31
  • 36
    joshhunt wins one (1) internet. This offer may not be combined with other offers, exchanged, or substituted. No rainchecks.
    – Adam Davis
    Dec 11 '09 at 21:50
  • 30
    The lengths some people will go to, to earn rep on SU...
    – Antony
    Dec 12 '09 at 0:11
  • 28
    Please don't refer to what you did as "backups" - if those files are on the same server, they're in no way "backups."
    – wahnfrieden
    Dec 14 '09 at 22:35

43 Answers 43


This is my python script, it will scrape though google cache and download the content of your webiste, and it can run without trouble with 503 504 404 error (Google blocks IP that send many request): https://gist.github.com/3787790


Your images, ask SUN microsystems to give them back to you, they have made "an entire internet backup" ... in a shipping container

"The Internet Archive offers long-term digital preservation to the ephemeral Internet," said Brewster Kahle, founder, the Internet Archive organization. "As more of the world's most valuable information moves online and data grows exponentially, the Internet Archive will serve as a living history to ensure future generations can access and continue to preserve these important documents over time."

Founded in 1996 by Brewster Kahle, the Internet Archive is a non-profit organization that has built a library of Internet sites and other cultural artifacts in digital form that include moving images, live audio, audio and text formats. The Archive offers free access to researchers, historians, scholars, and the general public; and also features "The Wayback Machine" -- a digital time capsule that allows users to see archived versions of Web pages across time. At the end of 2008, the Internet Archive housed over three petabtyes of information, which is roughly equivalent to about 150 times the information contained in the Library of Congress. Going forward, the Archive is expected to grow at approximately 100 terabytes a month.

alt text
(source: gawker.com)

more here and here


I was going to suggest Warrick because it was written by one of my CS professors. I'm sorry to hear that you had a bad experience with it. Maybe you can at least send him a note with some bug reports.


Have you tried doing a Google Image search, with the syntax site:codinghorror.com?


I do have full text entries for Codinghorror in my RSS reader back to June 30, 2009, if that will help at all. E-mail me at jake (at) orty (dot) com. I'll see if I can get them dumped out of Newsgator Inbox in any sort of usable format. I might have them back further (I'll need to dig up my archived PST files). Can't help with images, but it's a start (shrug).

(Nevermind: Looks like you have plenty more options above than I could provide. Sorry about the noise, feel free to flag to delete.)


I can read old posts on my Google Reader account. Maybe that helps: relating to your horror.


Maybe you could crowd-source it asking us to look in our browser caches. I generally read Coding Horror via Google Reader, so my Firefox cache doesn't seem to have anything from codinghorror.com in it.

Others can look in their own Firefox cache by browsing to: about:cache?device=disk .


Just another shot at retrieving the content.

I was subscribed using feed burner. So might have some archives in my mail! You can ask others, who might be able to forward you those posts.


This happened to me once and I had to rebuild my WordPress blog. I was able to recover all of the text from search engine caches just like you are doing. However, when you recreate the posts you can really screw up your inbound links if you don't give them the original permalinks. Images weren't much of a problem for me because I tend to store them locally.


You could try to get the broken HDD from hosting company and give it to a hdd recovery service, I think you could find one. At least the backup images would probably be restored form there. Also this disk could be part of some mirror/RAID system and there is somewhere a mirror image?


Most solutions use a combination of blog reader assistance, archive.org, and Google caching. Consider turning this data crisis into a blog recovery tool specification. Several features listed in the question and answers look ready to automate, given knowledge an owner would have of their root site.

  1. Restore pages from archive.org, Google cache, or local cache using web spider that avoids bannable techniques
  2. Check local cache, Google image search, and imageshack for matching file names
  3. After initial recovery, make list of site's missing images and other URLs (e.g., return 304 code for images)
  4. Add upload or contribution form for readers who have cached versions
  5. Site owner previews and validates contributions
  6. Resubmit recovered pages to search engines, if desired

Owners that derive a lot of value from quick recovery might offer a bounty for missing files or other outside assistance.


Just automate grabbing the individual Google page cache files.

Here's a Ruby script I used in the past.


My script doesn't appear to have any sleeps. I didn't get IP banned for some reason, but I'd recommend adding one.

  • google page caches don't save images, just text
    – icco
    Dec 11 '09 at 23:07

The website is still cached and indexed in Google so in terms of front end content, if you're quick, you can still grab the content on your website's pages and use your browsers dev tools to get the site's source code and stylesheets (before Google starts dropping your pages from it's index).

Other than that, the only way of retrieving data is on the server itself and if your host can't do that for you then you're probably out of luck. It's incredibly poor for any hosting provider to not be able to recover any form of data - someone at the company must have made a royal cock up.


Not the answer you're looking for? Browse other questions tagged or ask your own question.