Are there any search engines that do not care about robots.txt?
Tell me more
×
Webmasters Stack Exchange is a question and answer site for
pro webmasters. It's 100% free, no registration required.
|
|
All of the major, and minor, search engines respect robots.txt. Generally speaking only robots with bad intentions, or are poorly written, ignore robots.txt. You can find a list of bad bots here. |
|||||
|
|