Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

What are the best options for parsing Amazon S3 (Simple Storage) log files? I've turned on logging and now I have log files that look like this:

858e709ba90996df37d6f5152650086acb6db14a67d9aaae7a0f3620fdefb88f files.example.com [08/Jul/2010:10:31:42 +0000] 68.114.21.105 65a011a29cdf8ec533ec3d1ccaae921c 13880FBC9839395C REST.GET.OBJECT example.com/blog/wp-content/uploads/2006/10/kitties_we_cant_stop_here_this_is_bat_country.jpg "GET /example.com/blog/wp-content/uploads/2006/10/kitties_we_cant_stop_here_this_is_bat_country.jpg HTTP/1.1" 200 - 32957 32957 12 10 "http://atlanta.craigslist.org/forums/?act=Q&ID=163218891" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.19) Gecko/2010031422 Firefox/3.0.19" -

What are the best options for automating the log files? I'm not using any other Amazon services other than S3.

share|improve this question
    
Automating in what way? What information do you want to extract? –  delete Jul 9 '10 at 1:01
    
Something like Webalizer or AWStats or the like. Basic breakdowns by url, date, url, referrer. –  artlung Jul 9 '10 at 1:09
add comment

2 Answers 2

up vote 2 down vote accepted

Amazon recommends using AWStats and includes instructions on how to set up an AWStats configuration file. You will also need to set up some method to automate downloading the logs to where your AWStats installation is.

Personally I use cron to set off a Python script that uses boto to grab the files.

"""s3logDL.py

Downloads S3 logs and deletes old logs

Usage: python s3logDL.py"""

import sys
import os
from boto.s3 import Connection
import boto.s3

########################


########################

def main(argv):
    s3 = Connection('ACCESS_ID', 'PASSWORD')
    logsBucket = s3.get_bucket('logbucket')

    for b in logsBucket:
        #Download log
        print b.name
        key = logsBucket.get_key(b.name)
        fp = open("/PATH/TO/SAVE/POINT/" + b.name, "w")
        key.get_file(fp)

        #Delete log
        key.delete()

    exit(0)


########################

if __name__ == '__main__':
    main(sys.argv)
share|improve this answer
    
This looks great. Thanks! –  artlung Jul 9 '10 at 4:50
add comment

We are about to launch Qloudstat which will give you analytics for your S3 account as a service.

share|improve this answer
1  
It appears as though you are the registrant for the qloudstat.com domain - per our FAQ, it would be appreciated if you would edit your answer to disclose your affiliation to avoid downvotes or flags on your answer as spam. –  danlefree Nov 29 '11 at 11:38
    
No one likes spam Kocher! –  Anagio Nov 6 '12 at 22:50
    
+1, its an alternative, I will try it :) –  fnkr Jun 1 '13 at 11:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.