Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site?

I'm building new version of some site, but I don't want to get 404 errors for the old pages. Instead I want to return 301 redirection, but first of all I need the list of all the old links, then redirect to new ones.

The only one I've found so far is xenu webcrawler, but it's for Windows.

  • Does the site use JS to generate URLs? What about forms? If it's all accessible just by following links wget will do the job. Commented Jan 24, 2012 at 13:07
  • @Peter, How can I use wget to generate such a (recursive) list?
    – takeshin
    Commented Jan 24, 2012 at 14:19
  • --spider More details will have to wait until I'm back at a Linux box. Commented Jan 24, 2012 at 15:48

2 Answers 2


You don't have to use software, you can use .htaccess redirects that follow patterns like if the old link is to /software-for-learning/softwareitem-21, you could have it read the directory structure and forward everything inside of that (in this case /softwareitem-21 is ignored, since you matched the pattern based on the top level directory) redirect to the new path pattern (eg should now go to /products/software/learning/.

This is a bit tougher, I won't lie, but it might save you a few headaches and knowing how to write these rules out to follow certain patterns (like finding the numbers inside the title, or ignoring the title and just redirecting the page as long as its inside the directory /A/ to /new/directory/structure/a/ or whatever).

more info here: http://www.javascriptkit.com/howto/htaccess7.shtml

  • That website, JavaScriptKit, is excellent! It has all sorts of good things. Thank you. (You are everywhere, 609south... well, on SE, at least!) Commented Jan 24, 2012 at 20:25

Highly recommend Xenu if you can run a virtual windows system - it's very quick and very simple.

That said this would appear to be an extensive list, I'm told harvest is the best of the bunch, but can be complicated to set up.

You could also use platform independant software like http://www.xml-sitemaps.com/.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.