Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm building a link checker to check a huge amount of URLs for uptime and I'm currently thinking about timeout times. Do you know any statistics which provide the average header response time to have an approximation for setting the timeouts? Thanks!

share|improve this question

2 Answers 2

up vote 0 down vote accepted

Analyze HTTPArchive.org's dump of web performance data for thousands of sites to get the average response time.

HTTPArchive.org tracks web performance information such as size of pages, failed requests, and technologies utilized.

While a lot of interesting aggregated statistics drawn from frequently fetched data is presented through charts, response time is not specifically shown. So you may have to dig into the data yourself.

share|improve this answer

30 seconds timeout is my recommendation. And I recommend you not download the entire web page html and instead should be getting just the HEAD (php example) or just the http response (php example) and then bailing. Otherwise you could be downloading megabytes of long pages which increase your time.

At the most I would recommend a max 60 seconds timeout if you trying servers across the world but it should be sufficient to use a 30 second try with a retry every 60 seconds. Like trying for 30 seconds up to a max of 3 times at 60 second intervals to not hammer the server.

There really isn't any stats I can pull for you. I would use a similar value of most web browsers or retriever type commands (like curl, wget). Example:

php max-execution-time: 30 seconds http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time

Mozilla's old setting was 30 seconds http://kb.mozillazine.org/Network.http.connect.timeout

Internet explorer is something like an hour, see this stackoverflow question http://stackoverflow.com/questions/1342310/where-can-i-find-the-default-timeout-settings-for-all-browsers

CURL uses the system's default socket timeout http://curl.haxx.se/mail/lib-2003-05/0097.html http://stackoverflow.com/questions/10308915/php-default-curl-timeout-value

Wget doesn't have a default except for a 900 second read timeout http://www.gnu.org/software/wget/manual/html_node/Download-Options.html

share|improve this answer
    
I would also check the server header response for 200, 301, and 302. I wouldn't waste bandwidth on a site giving a 404 or any other response than a valid page. –  Anagio Jul 23 '12 at 20:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.