Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Are there any search engines that do not care about robots.txt?

share|improve this question
Any reason you want to know? – John Conde May 21 '11 at 16:27
up vote 3 down vote accepted

All of the major, and minor, search engines respect robots.txt. Generally speaking only robots with bad intentions, or are poorly written, ignore robots.txt. You can find a list of bad bots here.

share|improve this answer
Thanks! The list is pretty old though, no entries later than 2007.. – Kim May 21 '11 at 16:44

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.