I have some problem, when I try to write robots.txt for my site ...

I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it with .htaccess or other way ?

  • 1
    This question has an .htaccess file you can use webmasters.stackexchange.com/questions/29883/… Dec 19, 2012 at 10:21
  • I would like to see your robot file first,It would not be appropriate to give a one sided decision without invigilating the second side. Dec 19, 2012 at 11:01
  • No it's appropriate. They want to prevent a robot that doesn't honor robots.txt. Pretty simple if the robot isn't following your rules, band the sob by IP and cross your fingers.
    – Anagio
    Dec 19, 2012 at 11:19

2 Answers 2


If there are crawlers not following your robots.txt rules you will need to ban them by IP. Placing their user agent's into your robots.txt to ban does nothing if they aren't following it's rules.


Simple: Ban them all! With PHP and Regex. For example:

if (preg_match('/(?i)badbot1|badbot2|badbot3/',$_SERVER['HTTP_USER_AGENT'])){ 

header ('HTTP/1.1 403 Forbidden'); 

The header statement is optional

Be careful, never close the last "badbot" with a pipe "|". If you do, you ban all your traffic! So, use "badbot1|badbot2|badbot3".

Never "|badbot1|badbot2|badbot3" and Never "badbot1|badbot2|badbot3|"

Good luck

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.