I have some problem, when I try to write robots.txt for my site ...
I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it with .htaccess or other way ?
-
1This question has an .htaccess file you can use webmasters.stackexchange.com/questions/29883/…– paulmorrissDec 19, 2012 at 10:21
-
I would like to see your robot file first,It would not be appropriate to give a one sided decision without invigilating the second side.– sachin chauhanDec 19, 2012 at 11:01
-
No it's appropriate. They want to prevent a robot that doesn't honor robots.txt. Pretty simple if the robot isn't following your rules, band the sob by IP and cross your fingers.– AnagioDec 19, 2012 at 11:19
Add a comment
|
2 Answers
If there are crawlers not following your robots.txt rules you will need to ban them by IP. Placing their user agent's into your robots.txt to ban does nothing if they aren't following it's rules.
Simple: Ban them all! With PHP and Regex. For example:
if (preg_match('/(?i)badbot1|badbot2|badbot3/',$_SERVER['HTTP_USER_AGENT'])){
header ('HTTP/1.1 403 Forbidden');
exit();
}
The header statement is optional
Be careful, never close the last "badbot" with a pipe "|". If you do, you ban all your traffic! So, use "badbot1|badbot2|badbot3".
Never "|badbot1|badbot2|badbot3" and Never "badbot1|badbot2|badbot3|"
Good luck
