How should I interpret a empty User-agent? I have some custom analytics code and that code has to analyze only human traffic. I have got a working list of User-agents denoting human traffic, and bot ...
I've been struggling to get this .htaccess working. The aim is to disallow all user agents besides my app. The app sends a GET request with a user agent of lets say 'AcmeUpdater'. Whenever I try to ...
I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
I'd like to update my apache logging so that its destination is based on the user agent string or file type. Specifically, I would like to send my googlebot requests to their own log file, and I ...