I get huge bandwidth transfer from some unknown bot Unknown robot (identified by 'bot*').There are ways to block it via htaccess or other means. The question is whether to block it for website safety or it is the good that search engines are crawing the website.

I am very new to website designing so don't know the basics. Please guide me if i have to block them or let it be like this


The first step would be to figure out if this bot reads robots.txt. You can define a crawl-delay in there for this bot. This question goes in more detail about what you have to do. You can disallow this bot to crawl your site via there too. Please note that rogue bots do not read, or simply ignore, robots.txt and that changes to this file might not work.

If you want to block a rogue bot the hard way, figure out it's ip. You can use mod_authz_host in .htaccess to hard-block this ip. You can do this with the following code:

<Directory />
  Order Deny,Allow
  Deny from
  Allow from all

The xml-like-tags around this code say that these rules are valid for the / directory (root). If a request is processed, it will process all rules for that directory and all rules for any parent directory that directory has. (so: A request to /asdf/ will eventually parse these rules too.

Order Deny,Allow tells Apache to process all 'deny' rules first, then all allow rules. Deny from, will block all requests to Needless to tell that you should change this ip to the real ip. Allow from all will allow everyone else access. This will present that bot with a Forbidden error I believe.

More information about mod_authz_host can be found here and more information about the <directory> directive can be found here.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy