4

Is it OK to block requests that contain no user agent? I am getting random 404 errors on URLs I have never used. These requests have no user agent. If this is the case, will it affect any of my SEO?

  • 1
    Isn't the 404 error already indicating to these agents that they are making a mistake? What error status are you proposing that you would serve instead? – Stephen Ostermiller Jul 10 '14 at 1:17
  • 1
    I agree with @Stephen Ostermiller. You will be just trading one error for another and wasting resources doing it. Let it go brother- let it go. – closetnoc Jul 10 '14 at 1:22
  • Ever heard of saving server resources by dropping the request???? – Mike Flynn Jul 10 '14 at 1:24
  • @Mike Flynn Dropping the request in this case would require more resources. Using .htaccess, for example, processes the request twice internally within Apache. You would be better off blocking them in a firewall or outside of the server. – closetnoc Jul 10 '14 at 2:04
  • 1
    BTW- I have been where you are. I was getting 100,000 requests per hour minimum from China a few years back. Dropping the requests at the firewall, if you have one, saves a lot of resources! – closetnoc Jul 10 '14 at 2:11
10

Blocking No User Agent

Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search engine crawlers use user agents.

404's is normal

The correct code for a page that doesn't exist is 404... returning anything else would be incorrect and if your having issues with bots and concerned that a lot of resources are being consumed then using block on user agent would be incorrect through it would work but why when there's a simpler way...

Slowing down requests

You could block IP's, User Agents and so on but there's a much easier way... simply by using status 429 Too Many Requests (RFC 6585) this tackles all bots, all users and so on that request too much data too often within a certain period. It's common to see that some of the worlds most visited sites use this method and its effective. If you want to spend your time going through IPs from logs and then ban them then the status code you should return 403 Forbidden

| improve this answer | |
  • I want to reinforce something @bybe has said. Blocking using user agents is a futile effort. A user agent can be changed easily most of the time. As well, blocking by user agents is unreliable as user agent names are so numerous, can change rapidly, can be forged, may not exist, and so on. In my security research, the one most unrelaible data element I have is user agent names. Bybe has given you an excellent answer. – closetnoc Jul 10 '14 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.