Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Are there any search engines that do not care about robots.txt?

share|improve this question
    
Any reason you want to know? –  John Conde May 21 '11 at 16:27
add comment

1 Answer 1

up vote 3 down vote accepted

All of the major, and minor, search engines respect robots.txt. Generally speaking only robots with bad intentions, or are poorly written, ignore robots.txt. You can find a list of bad bots here.

share|improve this answer
    
Thanks! The list is pretty old though, no entries later than 2007.. –  Kim May 21 '11 at 16:44
    
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.