1

The regex is easy for 404s in a typical Apache log:

grep ' 404 ' access_log

But many are coming from bots, which I don't care about. I'm looking for actual people hitting 404s. So I'm trying to negate the term 'bot', which comes later in the line, but nothing's working. Any ideas?

2

This can be done using the -v (for --invert-match) option of grep:

grep -v "excluded_word" access_log | grep ' 404 '

grep -v "excluded_word" access_log will return all the lines that don't have the unwanted word and then it is piped to grep ' 404 ' to list only lines with that pattern.

Since an access_log can be pretty big, a faster way is to use awk

awk '/404/ && !/bot/' access_log will find 404 and but not bot

1

I find that one of the most useful ways to analyze 404 errors from a log file is to look at the referrer. You want to know where the 404 error was clicked, and the referrer tells you that. In addition, bots almost never send the referrer string. When you look at only records that contain the referrer, it gives you more information and excludes bots.

When no referrer is sent, Apache logs a quoted - in that field instead. So you can use grep -v to exclude lines that have only a dash in a field:

grep ' 404 ' access_log | grep -v ' "-" '
  • This is a great idea, but my log syntax contains several instances of " - ". e.g., x.x.x.x - - [16/Jun/2020:16:16:55 -0400]. I'll see about surer ways of finding referrer. – jphansen Jun 16 '20 at 20:28
  • Here's one that excludes bots and only includes lines with a referrer: grep -v "bot" access_log | grep ' 404 [0-9]* "http.*"' You can define more of the "http" part if you only want to include your domain. That is, to find broken links within your site. – jphansen Jun 16 '20 at 20:37
  • I forgot the quotes that I now edited it. The referrer is typically quoted in Apache logs, and it usually the only empty quoted field. There are more complete ways to parse Apache logs, but this is 99% effective and easy enough to remember. – Stephen Ostermiller Jun 16 '20 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.