2
votes
3answers
2k views

Interpretation of empty User-agent

How should I interpret a empty User-agent? I have some custom analytics code and that code has to analyze only human traffic. I have got a working list of User-agents denoting human traffic, and bot ...
3
votes
1answer
510 views

Disallow all user agents except one using .htaccess?

I've been struggling to get this .htaccess working. The aim is to disallow all user agents besides my app. The app sends a GET request with a user agent of lets say 'AcmeUpdater'. Whenever I try to ...
8
votes
3answers
591 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
2
votes
1answer
466 views

Apache logging based on user agent

I'd like to update my apache logging so that its destination is based on the user agent string or file type. Specifically, I would like to send my googlebot requests to their own log file, and I ...