Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site?

I'm building new version of some site, but I don't want to get 404 errors for the old pages. Instead I want to return 301 redirection, but first of all I need the list of all the old links, then redirect to new ones.

The only one I've found so far is xenu webcrawler, but it's for Windows.

share|improve this question
Does the site use JS to generate URLs? What about forms? If it's all accessible just by following links wget will do the job. – Peter Taylor Jan 24 '12 at 13:07
@Peter, How can I use wget to generate such a (recursive) list? – takeshin Jan 24 '12 at 14:19
--spider More details will have to wait until I'm back at a Linux box. – Peter Taylor Jan 24 '12 at 15:48

You don't have to use software, you can use .htaccess redirects that follow patterns like if the old link is to /software-for-learning/softwareitem-21, you could have it read the directory structure and forward everything inside of that (in this case /softwareitem-21 is ignored, since you matched the pattern based on the top level directory) redirect to the new path pattern (eg should now go to /products/software/learning/.

This is a bit tougher, I won't lie, but it might save you a few headaches and knowing how to write these rules out to follow certain patterns (like finding the numbers inside the title, or ignoring the title and just redirecting the page as long as its inside the directory /A/ to /new/directory/structure/a/ or whatever).

more info here: http://www.javascriptkit.com/howto/htaccess7.shtml

share|improve this answer
That website, JavaScriptKit, is excellent! It has all sorts of good things. Thank you. (You are everywhere, 609south... well, on SE, at least!) – Ellie Kesselman Jan 24 '12 at 20:25

Highly recommend Xenu if you can run a virtual windows system - it's very quick and very simple.

That said this would appear to be an extensive list, I'm told harvest is the best of the bunch, but can be complicated to set up.

You could also use platform independant software like http://www.xml-sitemaps.com/.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.