21

I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the site.

I have written a php crawler to follow all the links on my site, and then I am comparing that against a dump of the directory structure, but is there something easier?

15

I've used Xenu's Link Sleuth. It works pretty well, just be sure not to DOS yourself!

  • 1
    Check the 'Orphan files' option in the options and it will prompt you for an ftp login to your site. – pelms Jul 9 '10 at 10:27
  • Does this handle sites you have to login too, to get to the page? – Donny V. Jul 18 '10 at 20:49
  • @Jim How can I see what page the broken link is on? – Rob Jul 24 '12 at 12:58
  • Once the crawl is complete there is a report that will tell you the information. Depending on the size of the site and number of broken links, it can be hard to parse. I usually adjust the reporting options before the crawl, and I open the report as a text file (since it is .htm) and delete the stuff that isn't pertinent to make it more manageable. Hope this helps. – plntxt Jul 24 '12 at 13:54
6

If you are using windows 7 the best tool is IIS7's SEO Toolkit 1.0. It is free and you can download it for free.

The tool will scan any site and tell you where all of the dead links are, what pages take to long to load, what pages have missing titles, duplicate titles, same for keywords and descriptions, and what pages have broken HTML.

  • 1
    I used this personally, and it is excellent for scanning and of course SEO, but the FTP check in Xenu just edges this out. – MrChrister Jul 10 '10 at 16:10
4

Try W3C's open source tool Link Checker. You can use it online or install it locally

1

I'm a big fan of linklint for linkchecking large static sites, if you have a unix command line around (I've used on linux, MacOS, and FreeBSD). See their site for installation instructions. Once installed, I create a file called check.ll and do:

linklint @check.ll

Here's what my check.ll file looks like

# linklint
-doc .
-delay 0
-http
-htmlonly
-limit 4000
-net
-host www.example.com
-timeout 10

That does a crawl of www.example.com and generates HTML files with cross-referenced reports for what is broken, missing, etc.

1

There are several products from Microsys, especially their A1 Sitemap Generator and A1 Website Analyzer that will crawl your website and report everything you can possibly imagine about it.

That includes broken links, but also a table view of all your pages so you can compare things like identical <title> and meta description tags, nofollow links, meta noindex on webpages, and a whole lot of diseases that just need a sharp eye and a quick hand to fix.

1

Link Examiner is a really good freeware too for your need.

Not the answer you're looking for? Browse other questions tagged or ask your own question.