6

I have a bit of a situation coming up. Due to a complete website structure redesign that is basically inevitable, I expect to have the following:

  • Our sitemap of about 12,000 url's have about 90-95% of them change
  • Out of those 12,000, I expect around 5000-6000 internal links to go dead in the process.
  • No external links to this site yet, as it is still in development.

Is there a tool out there that can do the following:

  • I can feed the sitemap.xml after the restructuring
  • have it parse each pages links for 404 errors on that page
  • only report the pages/errors, preferably with just the url it is on, the url of the error, and the anchor text

I have found a few tools, but all of them seem to be limited to 100 pages.

Any advice for an intermediate webmaster to help this situation? 301 redirects are not viable in this situation.

8
  • 1
    I am writing you a PHP script right now, which will let you upload a sitemap, press go, and then check each link for a 404 (obviously I can't test it for you since it's a local site, you said.) It will output (one per line) the URL to the dead pages only. I could also make it show dead ones in RED and good links in GREEN. Let me know soon, before I finish the script. I'll post it as an answer after I'm done. If possible, can you upload your sitemap to Pastebin or something? It will help greatly.
    – ionFish
    Aug 22, 2012 at 18:09
  • that would really help! the site is actually up on an accessible server, however what I need to do is provide a sitemap.xml, and then it would need to check each page from that sitemap for dead links within that page (1 page to another) - and then just note the url of the page that contains a dead anchor, and the anchor text if possible. sitemap can be found at msap.com/xmlsitemap.php
    – NRGdallas
    Aug 22, 2012 at 18:17
  • also, if you want, you can simply provide the php code and I can paste it into a text file and upload - it sounds like this would be really helpful for alot of webmasters to be able to edit slightly as well to do the same thing! I don't know much PHP, but if there is anything you need or that I can help with, let me know!
    – NRGdallas
    Aug 22, 2012 at 18:22
  • I am going to make the source public for the community when it's done. Shouldn't be too much longer. Wait, check each page from that sitemap for dead links within that page (1 page to another) like open each page FROM the sitemap, scan the whole page for dead links? Or just find dead links from the sitemap itself?
    – ionFish
    Aug 22, 2012 at 18:28
  • the sitemap is dynamically generated and will be 100% correct, the problem is the individual pages within that sitemap will then have tons of dead links, so it would need to recursively scan each page within the sitemap - tried to clarify the question a bit, as I was worried you had interpreted it the other way. Basically whatever tool would need to spider all anchors on each page and scan those, probably ALOT more complex :P - I understand if you don't want to try to reach for that one :)
    – NRGdallas
    Aug 22, 2012 at 18:35

2 Answers 2

5

I've finished what I can on the script so far (read comments of original question for detail and context). Source:

http://www.ionfish.org/projects/xml-spider/

Features:

  1. ability to start from any point (resume crashed attempt?) since it tells you which number it's processing.
  2. scans any sitemap public accessible
  3. finds all the links it has to scan BEFORE actually scanning (uses nearly zero memory or CPU)
  4. orders output by page scanned, with the links in order in which they appear in the HTML
  5. filters out bad links and removes duplicates
  6. uses cURL to get the headers of links ONLY (not wasting time/bandwidth)
  7. tells you exactly what type of error it encounters (if not 200) and highlights the output RED, and puts it in a <span> with the class being the error code
  8. has no output buffering (most servers) so the output is "live" unless your browser likes to chunk it
  9. technically could run forever, so VERY large (50k+ links) are possible
  10. option to show ONLY errors and hide the "successful" links
  11. option to debug, dumping the arrays of first every link in the sitemap, second all the cURL variables for each link in the sitemap, and third, an array of every link on every page in the sitemap

This is distributed (officially) under the MIT license. (Included in source).

3
  • Nice job ionFish, I'll check that out as it looks (from the code) like a handy quick checker. You should throw it up on github or something. Aug 24, 2012 at 18:23
  • @AnthonyHatzopoulos - One other colleague and I are writing an interface, titled "XML Sitemap Tools" which will have MANY features and be set in a sexy interface. I might branch it to GitHub later on, but I can't do any more work until NRGdallas contacts me about what he wants.
    – ionFish
    Aug 24, 2012 at 18:25
  • just as an update, this is now complete with a ton of really amazing features: running using VERY little processing power, and is very clean and working great - going to give it a bit of time to make sure no other bugs are missed, then ill get with ionFish to see if he wants to publicly list or how he wants to make it available. Great work ionFish!
    – NRGdallas
    Sep 7, 2012 at 20:55
1

Might be too much for your purposes but I would suggest using one of my favorite tool for this is the opensource webcheck. Written in python and currently maintained by Arthur de Jong. Check out the source and download at http://arthurdejong.org/webcheck/

It is very versitile tool which can crawl any given website, which runs via the command line, has a ton of switches and options (see the man page after installing) and then when finished it will generate reports like this: http://arthurdejong.org/webcheck/demo/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.