How do large sites like SO handle their access logs?

Question

i'm always wondering how large sites like SO handle their access logs. A write to the disk on every request seems a little bit uneconomical, but is Google Analytics that reliable to use it as your only information resource?

Thank you

Answer 1

There is much information that web server logs contain which will never be available to Google Analytics, two things I can think of:

• Errors, like 404, etc ...
• Access on media files, like pictures, etc ... (including external websites linking to your images)
• IP Addresses, although an answer to another question pointed out that it can be set as a user variable
• Full length referral urls, for example google sends links from product search, web search, etc ... each search has keywords but google analytics does not display the other variables like &source=products, etc.

Must be more stuff I just can't think of right now.

And there are also the error logs ... must be important for a website to run smoothly, in my opinion. Not something you would ignore.

Answer 2

On a *nix system you could use syslog-ng to store log messages on a dedicated log server for your load-balanced cluster(s) and then use a log analysis solution like Splunk to keep tabs on things - as for what the StackExchange sites actually run, may be a good question for StackOverflow Meta.

Answer 3

I dont really look at these logs and end up deleting them on a monthly basis. I only look at them for trouble shooting. As for as monitoring application use GoogleAnalytics, CrazyEgg and others do a great job.

Before such services existed these logs were very valuable. Now, they are good developer tools, but I dont know of any of my colleagues or friends that actively archive these logs or parse them for data.