Is it OK to block requests that contain no user agent? I am getting random 404 errors on URLs I have never used. These requests have no user agent. If this is the case, will it affect any of my SEO?
-
1Isn't the 404 error already indicating to these agents that they are making a mistake? What error status are you proposing that you would serve instead?– Stephen Ostermiller ♦Commented Jul 10, 2014 at 1:17
-
1I agree with @Stephen Ostermiller. You will be just trading one error for another and wasting resources doing it. Let it go brother- let it go.– closetnocCommented Jul 10, 2014 at 1:22
-
Ever heard of saving server resources by dropping the request????– Mike FlynnCommented Jul 10, 2014 at 1:24
-
@Mike Flynn Dropping the request in this case would require more resources. Using .htaccess, for example, processes the request twice internally within Apache. You would be better off blocking them in a firewall or outside of the server.– closetnocCommented Jul 10, 2014 at 2:04
-
1BTW- I have been where you are. I was getting 100,000 requests per hour minimum from China a few years back. Dropping the requests at the firewall, if you have one, saves a lot of resources!– closetnocCommented Jul 10, 2014 at 2:11
Add a comment
|
1 Answer
Blocking No User Agent
Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search engine crawlers use user agents.
404's is normal
The correct code for a page that doesn't exist is 404... returning anything else would be incorrect and if your having issues with bots and concerned that a lot of resources are being consumed then using block on user agent would be incorrect through it would work but why when there's a simpler way...
Slowing down requests
You could block IP's, User Agents and so on but there's a much easier way... simply by using status 429 Too Many Requests (RFC 6585) this tackles all bots, all users and so on that request too much data too often within a certain period. It's common to see that some of the worlds most visited sites use this method and its effective. If you want to spend your time going through IPs from logs and then ban them then the status code you should return 403 Forbidden
answered Jul 10, 2014 at 2:45
-
I want to reinforce something @bybe has said. Blocking using user agents is a futile effort. A user agent can be changed easily most of the time. As well, blocking by user agents is unreliable as user agent names are so numerous, can change rapidly, can be forged, may not exist, and so on. In my security research, the one most unrelaible data element I have is user agent names. Bybe has given you an excellent answer. Commented Jul 10, 2014 at 3:24