Suppose you're re-designing an existing large site. The first thing to do is to figure out how the site is structured, how many links, pages, sub domains, etc.

Is there a tool you can use to crawl a site and have an easy-to-understand site map generated?

3 Answers 3


I use Xenu Link Sleuth, it will take a long time to run on a big site, but it will tell you how many levels each page is from the home page as well as how many internal links are pointing to the page and out from the page. It also gives data on broken links, redirects etc. and it's free.


You could try http://www.xml-sitemaps.com/ to create a sitemap, but I think it is limit to 500 links.


If you are using windows try IIS SEO toolkit IISSEO, you can tell it to crawl your site. It stores all the data in a SQL database so you can then slice the data how you please. It also gives some hints as to potential problems your site has. It also exports the data to CSV.

I use Xenu also to cross check any URL changes after they are made. You don't want any broken 301's.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.