Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Unfortunately, our hosting provider experienced 100% data loss, so I've lost all content for two hosted blog websites:

(Yes, yes, I absolutely should have done complete offsite backups. Unfortunately, all my backups were on the server itself. So save the lecture; you're 100% absolutely right, but that doesn't help me at the moment. Let's stay focused on the question here!)

I am beginning the slow, painful process of recovering the website from web crawler caches.

There are a few automated tools for recovering a website from internet web spider (Yahoo, Bing, Google, etc.) caches, like Warrick, but I had some bad results using this:

  • My IP address was quickly banned from Google for using it
  • I get lots of 500 and 503 errors and "waiting 5 minutes…"
  • Ultimately, I can recover the text content faster by hand

I've had much better luck by using a list of all blog posts, clicking through to the Google cache and saving each individual file as HTML. While there are a lot of blog posts, there aren't that many, and I figure I deserve some self-flagellation for not having a better backup strategy. Anyway, the important thing is that I've had good luck getting the blog post text this way, and I am definitely able to get the text of the web pages out of the Internet caches. Based on what I've done so far, I am confident I can recover all the lost blog post text and comments.

However, the images that go with each blog post are proving…more difficult.

Any general tips for recovering website pages from Internet caches, and in particular, places to recover archived images from website pages?

(And, again, please, no backup lectures. You're totally, completely, utterly right! But being right isn't solving my immediate problem… Unless you have a time machine…)

share|improve this question

migrated from webapps.stackexchange.com Aug 14 '12 at 15:05

This question came from our site for power users of web applications.

86  
When somebody like Jeff Atwood himself can lose two entire websites in one fell swoop... Well. I'm going to review my own backup procedures, for one :P –  Phoshi Dec 11 '09 at 21:10
225  
@Phoshi: Jeff has some good articles on Coding Horror on backup. You should give them a quick read. –  joshhunt Dec 11 '09 at 21:31
32  
joshhunt wins one (1) internet. This offer may not be combined with other offers, exchanged, or substituted. No rainchecks. –  Adam Davis Dec 11 '09 at 21:50
25  
The lengths some people will go to, to earn rep on SU... –  Antony Dec 12 '09 at 0:11
22  
Please don't refer to what you did as "backups" - if those files are on the same server, they're in no way "backups." –  wahnfrieden Dec 14 '09 at 22:35

42 Answers 42

This is my python script, it will scrape though google cache and download the content of your webiste, and it can run without trouble with 503 504 404 error (Google blocks IP that send many request): https://gist.github.com/3787790

share|improve this answer

If your images were stored on an external service such as Flickr or a CDN (as mentioned in one of your podcasts), you may still have the image resources there.

Some of the images could be found searching on Google Images and click on "Find similar images", maybe there are copies on other sites.

share|improve this answer

Jeff, I have written something for you here

In short what I propose you do is:

  1. Configure the web server to return 304 for every image request. 304 means that the file is not modified and this means that the browser will fetch the file from its cache if it is present there. (credit: this SuperUser answer)

  2. In every page in the website, add a small script to capture the image data and send it to the server.

  3. Save the image data in the server.

  4. Voila!

You can get the scripts from the given link.

share|improve this answer

By going to Google Image search and typing site:codinghorror.com you can at least find the thumbnailed versions of all of your images. No, it doesn't necessarily help, but it gives you a starting point for retrieving those thousands of images.

Codinghorror images

It looks like Google stores a larger thumbnail in some cases:

Google vs. Bing

Google is on the left, Bing on the right.

share|improve this answer
2  
yeah, worst case, we'll have to scale up the thumbnails from Google. I hear Bing stores larger thumbnails, though? –  Jeff Atwood Dec 11 '09 at 20:59
17  
I don't know if this is you. But Imageshack seems to have many of your blog images. profile.imageshack.us/user/codinghorror –  Nick Berardi Dec 11 '09 at 21:04
24  
Use the Google thumbnails as a start, then use tineye.com to see if anyone is hosting a copy. –  sep332 Dec 11 '09 at 21:48

I can read old posts on my Google Reader account. Maybe that helps: relating to your horror.

share|improve this answer

I do have full text entries for Codinghorror in my RSS reader back to June 30, 2009, if that will help at all. E-mail me at jake (at) orty (dot) com. I'll see if I can get them dumped out of Newsgator Inbox in any sort of usable format. I might have them back further (I'll need to dig up my archived PST files). Can't help with images, but it's a start (shrug).

(Nevermind: Looks like you have plenty more options above than I could provide. Sorry about the noise, feel free to flag to delete.)

share|improve this answer

In the past I've used http://www.archive.org/ to pull up cached images. It's kind of hit or miss but it has worked for me.
Also, when trying to recover stock photos that I've used on an old site, www.tineye.com is great when I only have the thumbnails and I need the full size images.

I hope this helps you. Good Luck.

share|improve this answer

I've had great experiences with archive.org. Even if you aren't able to extract all of your blog posts from the site, they keep periodical snapshots:

alt text

This way you can check out each page and see the blog posts you made. With the names of all the posts you can easily find them in Google's cache if archive.org doesn't have it. Archive tries to keep images, Google cache will have images, and I haven't emptied my cache recently so I can help you with the more recent blog posts :)

share|improve this answer

Have you tried doing a Google Image search, with the syntax site:codinghorror.com?

share|improve this answer

I suggest the combination of archive.org and a request anonymizer like [Tor][2]. I suggest using anonymizer because that way each of your requests will have a random IP and location and that way you can avoid getting banned by a archive.org (like Google did) for unusually high number of requests.

Good Luck, there are a lot of gems in that blog.

share|improve this answer

Just automate grabbing the individual Google page cache files.

Here's a Ruby script I used in the past.

http://pastie.org/739757

My script doesn't appear to have any sleeps. I didn't get IP banned for some reason, but I'd recommend adding one.

share|improve this answer

I've managed to recover these files from my Safari cache on Snow Leopard:

bad-code-offset-back.jpg
bad-code-offset-front.jpg
code-whitespace-invisible.png
code-whitespace-visible.png
coding-horror-official-logo-small.png
coding-horror-text.png
codinghorror-search-logo1.png
crucial-ssd-128gb-ct128m225.jpg
google-microformat-results-forum.png
google-microformat-results-review.png
kraken-cthulhu.jpg
mail.png
powered-by-crystaltech-web-hosting.png
ssd-vs-magnetic-graph.png

If anyone else wants to try, I've written a Python script to extract them to ~/codinghorror/filename, which I've put online here.

I hope this helps.

share|improve this answer

Try this query on the Wayback Machine:

http://web.archive.org/web/*sa_re_im_/http://codinghorror.com/*

This will get you all the images from codinghorror.com archived by archive.org. This returns 3878 images, some of which are duplicates. It will not be complete, but a good start none the less.

For the remaining images, you can use the thumbnails from a search engine cache, and then do a reverse look-up using these at http://www.tineye.com/ . You give it the thumbnail image, and it will give you a preview and a pointer to closely matching images found on the web.

share|improve this answer

Your images, ask SUN microsystems to give them back to you, they have made "an entire internet backup" ... in a shipping container

"The Internet Archive offers long-term digital preservation to the ephemeral Internet," said Brewster Kahle, founder, the Internet Archive organization. "As more of the world's most valuable information moves online and data grows exponentially, the Internet Archive will serve as a living history to ensure future generations can access and continue to preserve these important documents over time."

Founded in 1996 by Brewster Kahle, the Internet Archive is a non-profit organization that has built a library of Internet sites and other cultural artifacts in digital form that include moving images, live audio, audio and text formats. The Archive offers free access to researchers, historians, scholars, and the general public; and also features "The Wayback Machine" -- a digital time capsule that allows users to see archived versions of Web pages across time. At the end of 2008, the Internet Archive housed over three petabtyes of information, which is roughly equivalent to about 150 times the information contained in the Library of Congress. Going forward, the Archive is expected to grow at approximately 100 terabytes a month.

alt text

more here and here

share|improve this answer

I was going to suggest Warrick because it was written by one of my CS professors. I'm sorry to hear that you had a bad experience with it. Maybe you can at least send him a note with some bug reports.

share|improve this answer

Most solutions use a combination of blog reader assistance, archive.org, and Google caching. Consider turning this data crisis into a blog recovery tool specification. Several features listed in the question and answers look ready to automate, given knowledge an owner would have of their root site.

  1. Restore pages from archive.org, Google cache, or local cache using web spider that avoids bannable techniques
  2. Check local cache, Google image search, and imageshack for matching file names
  3. After initial recovery, make list of site's missing images and other URLs (e.g., return 304 code for images)
  4. Add upload or contribution form for readers who have cached versions
  5. Site owner previews and validates contributions
  6. Resubmit recovered pages to search engines, if desired

Owners that derive a lot of value from quick recovery might offer a bounty for missing files or other outside assistance.

share|improve this answer

Here's my wild stab in the dark: configure your web server to return 304 for every image request, then crowd-source the recovery by posting a list of URLs somewhere and asking on the podcast for all your readers to load each URL and harvest any images that load from their local caches. (This can only work after you restore the HTML pages themselves, complete with the <img ...> tags, which your question seems to imply that you will be able to do.)

This is basically a fancy way of saying, "get it from your readers' web browser caches." You have many readers and podcast listeners, so you can effectively mobilize a large number of people who are likely to have viewed your web site recently. But manually finding and extracting images from various web browsers' caches is difficult, and the entire approach works best if it's easy enough that many people will try it and be successful. Thus the 304 approach. All it requires of readers is that they click on a series of links and drag off any images that do load in their web browser (or right-click and save-as, etc.) and then email them to you or upload them to a central location you set up, or whatever. The main drawback of this approach is that web browser caches don't go back that far in time. But it only takes one reader who happened to load a post from 2006 in the past few days to rescue even a very old image. With a big enough audience, anything is possible.

share|improve this answer
46  
+1 for the most creative approach. Could actually work since CH has some many readers. –  delux247 Dec 11 '09 at 22:17
15  
implemented here? diovo.com/2009/12/… –  Jeff Atwood Dec 14 '09 at 21:18
3  
I think you could crawl your static files for the image tags and copy all of those into one giant page of images, instead of having everybody click each link. The diovo.com implementation looks very impressive, hope it works out for you. –  phloopy Dec 15 '09 at 6:00
1  
WOW that's a wicked wizardy shit I <3 –  Ahmad Alfy Mar 21 '13 at 10:27

At the risk of pointing out the obvious, try mining your own computer's backups for the images. I know my backup strategy is haphazard enough that I have multiple copies of a lot of files hanging around on external drives, burned discs, and in zip/tar files. Good luck!

share|improve this answer

You could try to get the broken HDD from hosting company and give it to a hdd recovery service, I think you could find one. At least the backup images would probably be restored form there. Also this disk could be part of some mirror/RAID system and there is somewhere a mirror image?

share|improve this answer

Luckily, future generations will be ok.

Even with only some of this big rock, scientists/linguiststs figured out a lot.

Rosetta Stone

If a few pictures are missing, leave it to someone to figure out in a couple thousand years.

Hopefully, you're laughing a little. :)

share|improve this answer
4  
Ok, you got a giggle from me at least ;-) –  Ogre Psalm33 Dec 16 '09 at 17:09

You could use TinEye to find duplicates of your images by searching the thumbnails with google cache. This will help only with images you've taken from others site, though.

share|improve this answer
1  
No, it would help with images others have taken from CH. –  DisgruntledGoat Dec 24 '09 at 23:24

+1 on the dd recommendation if (1) the raw disk is available somewhere; and (2) the images were simple files. Then you can use a forensic 'data-carving' tool to (for example) pull out all credible ranges that appear to be JPGs/PNGs/GIFs. I've recovered 95%+ of the photos on an iPhone that was wiped this way.

The open source tools 'foremost' and its successor 'scalpel' can be used for this:

http://foremost.sourceforge.net/

http://www.digitalforensicssolutions.com/Scalpel/

share|improve this answer
2  
wow -- great tips! –  Jeff Atwood Dec 11 '09 at 22:04
2  
Photorec may also be of use once you get dd images. –  Broam Dec 11 '09 at 22:28

The web archive caches the images. It's under heavy load right now, you should be ok until 2008 or so.

http://web.archive.org/web/20080618014552rn%5F2/www.codinghorror.com/blog/

share|improve this answer

A suggestion for the future: I use Windows Live Writer for blogging and it saves local copies of posts on my machine, in addition to publishing them out to the blog.

share|improve this answer

If you're hoping to try to scrape users' caches, you may want to set the server to respond 304 Not Modified to all conditional-GET ('If-Modified-Since' or 'If-None-Match') requests, which browsers use to revalidate their cached material.

If your initial caching headers on static content like images were pretty liberal -- allowing things to be cached for days or months -- you could keep getting revalidate requests for a while. Set a cookie on those requests, and appeal to those users to run a script against their cache to extract the images they still have.

Beware, though: the moment you start putting up any textual content with inline resources that aren't yet present, you could be wiping out those cached versions as revalidators hit 404s.

share|improve this answer

It is a long shot, but you could consider:

  • Posting the exact list of picture you are missing
  • crowd-sourcing the retrieval process through all your readers's internet cache.

For instance, see the Nirsoft Mozilla Cache Viewer:

alt text

It can quickly dig up any "blog.stackoverflow.com" picture one might still have through a simple command line:

MozillaCacheView.exe -folder "C:\Documents and Settings\Administrator\Local Settings\Application Data\Mozilla\Firefox\Profiles\acf2c3u2.default\Cache" 
/copycache "http://blog.stackoverflow.com" "image" /CopyFilesFolder "c:\temp\blogso" /UseWebSiteDirStructure 0


Note: they have the same cache explorer for Chrome.

alt text

(I must have 15 days worth of blog.stackoverflow.com pictures in it)

And Internet Explorer, or Opera.


Then update the public list to reflect what the readers report finding in their cache.

share|improve this answer

About five years ago, an early incarnation of an external hard drive on which I was storing all my digital photos failed badly. I made an image of the hard drive using dd and wrote a rudimentary tool to recover anything that looked like a JPEG image. Got most of my photos out of that.

So, the question is, can you get a copy of the virtual machine disk image which held the images?

share|improve this answer

Very sorry to hear this and I am very annoyed for you, and the timing - I wanted an offline copy of a few of your posts and did HTTrack on your entire site but had to go out (this was a couple of weeks ago) and I stopped it.

If the host is half descent - and by the fact I am guessing you are a good customer... I would ask them to either send you the hard drives (as I am guessing they should be using RAID) or do some recovery themselves.

Whilst this may not be a fast process, I did this with one host for a client and was able to recover entire databases intact (... basically, the host tried an upgrade for the control panel they were using and messed it up.. but nothing was overwritten).

Whatever happens - Good luck from all your fans on the SO sites!

share|improve this answer

Have you tried your own local browser cache? Pretty good chance some of the more recent stuff is still there. http://lifehacker.com/385883/resurrect-images-from-my-web-browser-cache

(Or you could compile a list of all missing images and everyone could check their cache to see if we can fill in the blanks)

share|improve this answer

(1) Extract a list of the filenames of all missing images from the HTML backups. You'll be left with something like:

  • stay-puft-marshmallow-man.jpg
  • internet-properties-dialog.png
  • yahoo-homepage-small.png
  • password-show-animated.gif
  • tivo2.jpg
  • michael-abrash-graphics-program

(2) Do a Google Image Search for those filenames. It seems like MANY of them have been, um, "mirrored" by other bloggers and are ripe for the taking because they have the same filename.

(3) You could do this in an automated fashion if it proves successful for, say, 10+ images.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.