Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Suppose you're re-designing an existing large site. The first thing to do is to figure out how the site is structured, how many links, pages, sub domains, etc.

Is there a tool you can use to crawl a site and have an easy-to-understand site map generated?

share|improve this question

3 Answers 3

You could try http://www.xml-sitemaps.com/ to create a sitemap, but I think it is limit to 500 links.

share|improve this answer

I use Xenu Link Sleuth, it will take a long time to run on a big site, but it will tell you how many levels each page is from the home page as well as how many internal links are pointing to the page and out from the page. It also gives data on broken links, redirects etc. and it's free.

share|improve this answer

If you are using windows try IIS SEO toolkit IISSEO, you can tell it to crawl your site. It stores all the data in a SQL database so you can then slice the data how you please. It also gives some hints as to potential problems your site has. It also exports the data to CSV.

I use Xenu also to cross check any URL changes after they are made. You don't want any broken 301's.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.