I'm building a link checker to check a huge amount of URLs for uptime and I'm currently thinking about timeout times. Do you know any statistics which provide the average header response time to have an approximation for setting the timeouts? Thanks!
Analyze HTTPArchive.org's dump of web performance data for thousands of sites to get the average response time.
HTTPArchive.org tracks web performance information such as size of pages, failed requests, and technologies utilized.
While a lot of interesting aggregated statistics drawn from frequently fetched data is presented through charts, response time is not specifically shown. So you may have to dig into the data yourself.
30 seconds timeout is my recommendation. And I recommend you not download the entire web page html and instead should be getting just the HEAD (php example) or just the http response (php example) and then bailing. Otherwise you could be downloading megabytes of long pages which increase your time.
At the most I would recommend a max 60 seconds timeout if you trying servers across the world but it should be sufficient to use a 30 second try with a retry every 60 seconds. Like trying for 30 seconds up to a max of 3 times at 60 second intervals to not hammer the server.
There really isn't any stats I can pull for you. I would use a similar value of most web browsers or retriever type commands (like curl, wget). Example:
php max-execution-time: 30 seconds http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
Mozilla's old setting was 30 seconds http://kb.mozillazine.org/Network.http.connect.timeout
Internet explorer is something like an hour, see this stackoverflow question http://stackoverflow.com/questions/1342310/where-can-i-find-the-default-timeout-settings-for-all-browsers
CURL uses the system's default socket timeout http://curl.haxx.se/mail/lib-2003-05/0097.html http://stackoverflow.com/questions/10308915/php-default-curl-timeout-value
Wget doesn't have a default except for a 900 second read timeout http://www.gnu.org/software/wget/manual/html_node/Download-Options.html