Tool to generate list of redirections (sitemap for .htaccess)

Question

Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site?

I'm building new version of some site, but I don't want to get 404 errors for the old pages. Instead I want to return 301 redirection, but first of all I need the list of all the old links, then redirect to new ones.

The only one I've found so far is xenu webcrawler, but it's for Windows.

Answer 1

You don't have to use software, you can use .htaccess redirects that follow patterns like if the old link is to /software-for-learning/softwareitem-21, you could have it read the directory structure and forward everything inside of that (in this case /softwareitem-21 is ignored, since you matched the pattern based on the top level directory) redirect to the new path pattern (eg should now go to /products/software/learning/.

This is a bit tougher, I won't lie, but it might save you a few headaches and knowing how to write these rules out to follow certain patterns (like finding the numbers inside the title, or ignoring the title and just redirecting the page as long as its inside the directory /A/ to /new/directory/structure/a/ or whatever).

more info here: http://www.javascriptkit.com/howto/htaccess7.shtml

Answer 2

Highly recommend Xenu if you can run a virtual windows system - it's very quick and very simple.

That said this would appear to be an extensive list, I'm told harvest is the best of the bunch, but can be complicated to set up.

You could also use platform independant software like http://www.xml-sitemaps.com/.