0

We're about to have a major site infrastructure migration (new OS, upgraded PHP, etc).

We currently have a live test mirror of the new environment running, using the production file system.

We're expecting there to be some issues due to both the version change in PHP (let's just say PHP was very out of date on the old server and leave it at that) and changes in environment variables. Are there any good automated tools to quickly spider both sites so we can pinpoint issues faster than manually checking each page? I've already found a few manually, but anything which can double check in an automated fashion--especially across the entire site--would be lovely. I've searched around a bit but haven't found anything that seems to fit what we need. Maybe I'm just running the wrong keywords.

Multi step processes would be fine too, like spidering the site with a batch script/mass downloader and then running a folder diff. I'm coding from a windows environment but 'nix tools are fine too.

Improve this question

1 Answer 1

Reset to default
1

Its pretty easy to do with a little bit of shell scripting. Here is a script uses bash, wget, and diff to download two urls (specified on the command line) and print out the differences.

#!/bin/bash
set -e
url1=$1
url2=$2
if [[ "$url1" != http* ]]
then
   echo "Bad url: $url1"
fi
if [[ "$url2" != http* ]]
then
   echo "Bad url: $url2"
fi
file1=`wget -x $url1 2>&1 | grep 'Saving to' | sed "s/^Saving to: .//g;s/.$//g"`
file2=`wget -x $url2 2>&1 | grep 'Saving to' | sed "s/^Saving to: .//g;s/.$//g"`
diff -u $file1 $file2

You could call this with a list of pages on the site.

Alternately, you could use wget to download both sites entirely and then do a diff of the entire directory structure created.

wget -rx http://old.example.com/
wget -rx http://new.example.com/
diff -u old.example.com new.example.com

I tested this on a machine running Ubuntu Linux. It should be trivial to get to work on other Linux variants. You much just have to install wget or diff. Under Windows you could run it under Cygwin. Should be able to get it to work under OS X as well, but I don't know the details.

Improve this answer
5
  • It's an enterprise (higher ed) level site with a number of "child"/department sub-sites, so it definitely needs to be recursive/spidering. For your second wget example, should that be a lower case r, and isn't wget's recursion limited to a depth of 5 layers (5 link jumps)? That would get everything fairly top level, hopefully, so I may pop Cygwin on here and try it a bit later today =)
    – taswyn
    Commented Feb 4, 2013 at 20:02
  • Corrected the -R to -r. There is a -l for specifying the retrieval depth. The default depth is set to 5, but you can use -l to set it higher.
    – Stephen Ostermiller ♦
    Commented Feb 4, 2013 at 22:37
  • I'm fairly sure the maximum depth is 5, and -l can be used to restrict to lower (going by the documentation), but in either case thank you, I'm waiting for a couple fixes to go through first but will give it a try and see if it works tomorrow hopefully!
    – taswyn
    Commented Feb 4, 2013 at 23:34
  • /etc/wgetrc can have a maximum recursion level in it (reclevel). It appears to me that the -l switch CAN override it, but the documentation isn't explicit. You could always edit the wgetrc file to change it.
    – Stephen Ostermiller ♦
    Commented Feb 5, 2013 at 11:58
  • Just a note, make sure to set the recursion flag for the diff. This MOSTLY seemed to work, although there are a few oddities I haven't had time to isolate (for some reason it completed far earlier on our test server)... and the wget needs to get tuned if possible to only pull html/text files, and not random 200MB mp4s someone left up =P about 10k files later, lol... More importantly it's managed to catch a number of minor issues I wouldn't have been able to find manually and might have missed the attention of the individual web admins for those departments, so thank you for the help Stephen!
    – taswyn
    Commented Feb 7, 2013 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.