Why Am I Getting a Large Number of 404 Errors From One IP Address

Question

I am getting a large number of 404 errors from a single IP address. Should I be worried, what can I do about it?

Answer 1

First things first, not all 404 errors are bad, the fact that a 404 error is coming up means that your server is doing the right thing and returning a 404 page not found error for the request.

Why does this happen...

Old Links Still Pointing To Your Site
The internet is like an elephant, long memory, and many sites will still have links to your site even after your site no longer exists and the domain is not registered anywhere. 404 errors from someone clicking on one of these links is very common and nothing to be worried about at all. If the content has just been moved to another location you should try and add a 301 permanently moved header and redirect the user to the new location so as to maintain the page rank from that link as well as to maintain the user experience.

Mis-Spelt URL's
Once again this is very common, the internet is by and large used by humans and humans are fallible, we often spell things incorrectly, and so if a URL has been typed it incorrectly it will return a 404 error to the end user which in many cases will encourage the user to check the address and try again making sure to spell it right or use the site search feature if there is one to try and find the page they where looking for. Once again nothing to worry about if you see these sorts of 404 records in your server log.

Large Number of Requests to the Same Page
This is often the result of a mis-configured web crawler. This can either be a genuine crawler from a legitimate search engine website such as Google, or it can be a private crawler used by a private party working on their own search private search engine. If the load it is imparting on your server is of a concern then you can block the user agent string or IP address but often this is not needed as returning a 404 error to the end user does not impart a huge load on the server and many web servers are designed to handle a larger number of connections that those which are imparted by a mis-configured innocent crawler.

Attempts to Breach the Website
Now this is where you are more likely to need to sit up and maybe do something. If your server logs indicate traffic which you know should not exist, and it appears as though the connections resemble a penetration test of your website (such as URL's which vary by only a small amount and are increment an ID number of small number of letters each time). In this case it is a possibility that whoever is controlling the connections on the other end is trying to see if they can break into your website, sometimes just for the crowing rights of saying they managed it, sometimes for more nefarious purposes. The first step when you see this would be to verify, check the IP address and do a lookup, does the IP belong to a business that may have a genuine need or is unlikely to be doing this sort of thing intentionally, if so try and let them know as they may have malware on their network or something may be mis-configured. If however the IP address appears to be coming from an internet service provider then there is a real chance that there is no reason for that to be happening, in which case you can temporarily block the address space from connecting to your server and notify the ISP's abuse email about the attempted breaches on your server and the IP address which was causing the breaches. No guarantee that anything will be done but it is a possibility.

Basically the point is that excepting extreme circumstances or protracted issues that may be occurring there is no need to be concerned from an initial small or medium number of 404 errors, mainly when the errors get into a large number of they are starting to affect your sites performance do you need to dig deeper and find out what is going on.