Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines"

enter image description here

So that leaves me wondering, that rewording must be that some search engines disregard your decision about indexing or not your website.

Is there a list of those somewhere?

share|improve this question
    
All this does is change your robots.txt file, which instructs search engines which pages they can crawl, and which they cannot. Whether or not they obey those instructions is up the bot. –  Brendon Dec 17 '13 at 0:06
1  
And that's the question... what search engine bots disregard the robots.txt and still index your site? –  Fredy31 Dec 17 '13 at 0:11

2 Answers 2

up vote 3 down vote accepted

Even Google will occasionally include a site in its search index, despite the fact that the site disallows all crawling with robots.txt. Google views robots.txt rules as preventing crawling, but not necessarily inclusion in the search index.

When Googlebot can't crawl a site, but the site has lots of external links, Google may include pages from the site in the index and use the anchor text of the links to determine what the page is about.

Google's Matt Cutts blogged about this. He gave an example of the Department of Motor Vehicles in California. Its website disallowed crawling, but it would be ridiculous if people couldn't find it on Google. So they came up with a way of including sites like that without being able to crawl them.

share|improve this answer

I believe that the robot.txt file is just like asking the search engine not to index you. There is no actual disabling of indexing, it is up to the search engines not to index you.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.