Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 18 down vote favorite
9

I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the site.

I have written a php crawler to follow all the links on my site, and then I am comparing that against a dump of the directory structure, but is there something easier?

share|improve this question
See also webmasters.stackexchange.com/questions/13310/… who asks about spell-checking as well. – James Skemp Jun 29 '11 at 22:44

6 Answers

up vote 14 down vote accepted

I've used Xenu's Link Sleuth. It works pretty well, just be sure not to DOS yourself!

share|improve this answer
1  
Check the 'Orphan files' option in the options and it will prompt you for an ftp login to your site. – pelms Jul 9 '10 at 10:27
Does this handle sites you have to login too, to get to the page? – Donny V. Jul 18 '10 at 20:49
@Jim How can I see what page the broken link is on? – Rob Jul 24 '12 at 12:58
Once the crawl is complete there is a report that will tell you the information. Depending on the size of the site and number of broken links, it can be hard to parse. I usually adjust the reporting options before the crawl, and I open the report as a text file (since it is .htm) and delete the stuff that isn't pertinent to make it more manageable. Hope this helps. – Jim Jul 24 '12 at 13:54
up vote 6 down vote

If you are using windows 7 the best tool is IIS7's SEO Toolkit 1.0. It is free and you can download it for free.

The tool will scan any site and tell you where all of the dead links are, what pages take to long to load, what pages have missing titles, duplicate titles, same for keywords and descriptions, and what pages have broken HTML.

share|improve this answer
1  
I used this personally, and it is excellent for scanning and of course SEO, but the FTP check in Xenu just edges this out. – MrChrister Jul 10 '10 at 16:10
up vote 3 down vote

Try W3C's open source tool Link Checker. You can use it online or install it locally

share|improve this answer
up vote 1 down vote

I'm a big fan of linklint for linkchecking large static sites, if you have a unix command line around (I've used on linux, MacOS, and FreeBSD). See their site for installation instructions. Once installed, I create a file called check.ll and do:

linklint @check.ll

Here's what my check.ll file looks like

# linklint
-doc .
-delay 0
-http
-htmlonly
-limit 4000
-net
-host www.example.com
-timeout 10

That does a crawl of www.example.com and generates HTML files with cross-referenced reports for what is broken, missing, etc.

share|improve this answer
up vote 1 down vote

There are several products from Microsys, especially their A1 Sitemap Generator and A1 Website Analyzer that will crawl your website and report everything you can possibly imagine about it.

That includes broken links, but also a table view of all your pages so you can compare things like identical <title> and meta description tags, nofollow links, meta noindex on webpages, and a whole lot of diseases that just need a sharp eye and a quick hand to fix.

share|improve this answer
up vote 1 down vote

Link Examiner is a really good freeware too for your need.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.