The regex is easy for 404s in a typical Apache log:
grep ' 404 ' access_log
But many are coming from bots, which I don't care about. I'm looking for actual people hitting 404s. So I'm trying to negate the term 'bot', which comes later in the line, but nothing's working. Any ideas?
This can be done using the -v (for --invert-match) option of grep:
grep -v "excluded_word" access_log | grep ' 404 '
grep -v "excluded_word" access_log will return all the lines that don't have the unwanted word and then it is piped to grep ' 404 ' to list only lines with that pattern.
Since an access_log can be pretty big, a faster way is to use awk
awk '/404/ && !/bot/' access_log will find 404 and but not bot
I find that one of the most useful ways to analyze 404 errors from a log file is to look at the referrer. You want to know where the 404 error was clicked, and the referrer tells you that. In addition, bots almost never send the referrer string. When you look at only records that contain the referrer, it gives you more information and excludes bots.
When no referrer is sent, Apache logs a quoted - in that field instead. So you can use grep -v to exclude lines that have only a dash in a field:
grep ' 404 ' access_log | grep -v ' "-" '
