Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 2 down vote favorite

I have some problem, when I try to write robots.txt for my site ...

I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it with .htaccess or other way ?

share|improve this question
1  
This question has an .htaccess file you can use webmasters.stackexchange.com/questions/29883/… – paulmorriss Dec 19 '12 at 10:21
I would like to see your robot file first,It would not be appropriate to give a one sided decision without invigilating the second side. – sachin chauhan Dec 19 '12 at 11:01
No it's appropriate. They want to prevent a robot that doesn't honor robots.txt. Pretty simple if the robot isn't following your rules, band the sob by IP and cross your fingers. – Anagio Dec 19 '12 at 11:19

2 Answers

up vote 2 down vote accepted

If there are crawlers not following your robots.txt rules you will need to ban them by IP. Placing their user agent's into your robots.txt to ban does nothing if they aren't following it's rules.

share|improve this answer
up vote 0 down vote

Simple: Ban them all! With PHP and Regex. For example:

if (preg_match('/(?i)badbot1|badbot2|badbot3/',$_SERVER['HTTP_USER_AGENT'])){ 

header ('HTTP/1.1 403 Forbidden'); 
exit(); 
}

The header statement is optional

Be careful, never close the last "badbot" with a pipe "|". If you do, you ban all your traffic! So, use "badbot1|badbot2|badbot3".

Never "|badbot1|badbot2|badbot3" and Never "badbot1|badbot2|badbot3|"

Good luck

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.