Tag Info

Hot answers tagged

12

For Windows users: Xenu For *nix/Mac: wget (usually installed by default) Update: Typical wget usage: wget -r --spider -b -o /var/tmp/wget.log http://target.site.com/ -r - Recursive download --spider - Check downloaded content for new links, then discard -b - Run in the background -o /var/tmp/wget.log - Target log file http://target.site.com/ - ...


6

If you are using windows 7 the best tool is IIS7's SEO Toolkit 1.0. It is free and you can download it for free. The tool will scan any site and tell you where all of the dead links are, what pages take to long to load, what pages have missing titles, duplicate titles, same for keywords and descriptions, and what pages have broken HTML.


3

Web apps that offer scheduled link checking with emailed test results: Nodeworks No tags, no glory! LinkAlarm All offer free trial versions or limited accounts with paid upgrade options. The W3C offers a free online link checker tool, the source code for which is available here, but this isn't an automated service -- you'd have to run the link ...


3

A1WebsiteAnalyzer is easy to use. You can automate via Windows scheduled tasks. I haven't used InSpyder but it's the same price and does the same thing.


3

I use a piece of software called Xenu Link Sleuth. I first used it about 10 years ago and the interface hasn't changed much in that time, but it works great for small to medium size websites. It will still work on large websites, but it will take an age to crawl tens of thousands of dynamically generated content pages.


1

Yes, there are crawlers that can crawl a site which requires login. This requires that you log into your site using your web browser and export your cookies. Then you start the crawler with the cookies that you had used to log in and the the crawler crawls the site as your logged in user. To export your cookies, use Firefox with the Export Cookies ...


1

However, the change would have broken existing links. So, nicely, they stuck in 302 redirects. Well, the important bit here is that the change would have broken existing links. /commerce/product_info.php?products_id=9266 isn't the location of the product page anymore, /products/9266 is. It's not at all unreasonable to expect the old-style links to stop ...


1

Note: You'll get a better answer to this by emailing team@stackoverflow.com, they handle questions and issues with the validator. IMHO sparkfun have done the wrong thing SEO wise by using a 302, it should be a 301 (see "SEO: ecommerce item deleted by user, 301 rediret to HOME PAGE or 404 not found?"). At a complete guess the stack bot regards a 302 as ...


1

There are several products from Microsys, especially their A1 Sitemap Generator and A1 Website Analyzer that will crawl your website and report everything you can possibly imagine about it. That includes broken links, but also a table view of all your pages so you can compare things like identical <title> and meta description tags, nofollow links, meta ...


1

I'm a big fan of linklint for linkchecking large static sites, if you have a unix command line around (I've used on linux, MacOS, and FreeBSD). See their site for installation instructions. Once installed, I create a file called check.ll and do: linklint @check.ll Here's what my check.ll file looks like # linklint -doc . -delay 0 -http -htmlonly -limit ...


1

Google Webmaster tools will report on broken links. It's a simple tool, free to install to a website.....and platform agnostic. If you're not already using it, it can help you boost your SEO and useability as well. https://www.google.com/webmasters/tools/home?hl=en



Only top voted, non community-wiki answers of a minimum length are eligible