How do I diagnose 408 crawl errors?

Question

Somebody sent me a message about my website: "I'm getting a 408 error when I try to crawl it."

How do I resolve the 408 errors that are causing problems for this crawler?

Answer 1

You can use a tool like Xenu's Link Sleuth to find crawl errors on your website. (It works on Windows only).

Answer 2

According to the documentation a 408 error is a "Request Timeout".

The client did not produce a request within the time that the server was prepared to wait. The client MAY repeat the request without modifications at any later time.

This page has helpful advice for diagnosing 408 errors. Specifically:

408 errors are often difficult to resolve. They typically involve one-off variations in system workload or operations.

If you see persistent 408 errors, the first thing to consider is the workload on the Web server - particularly around the time the 408 errors were generated. If this is light, then you also need to consider workload on the client system. If the computer systems on both ends of the socket connection seem to be running normally, then temporary Internet surges may be to blame.