1

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines"

enter image description here

So that leaves me wondering, that rewording must be that some search engines disregard your decision about indexing or not your website.

Is there a list of those somewhere?

  • All this does is change your robots.txt file, which instructs search engines which pages they can crawl, and which they cannot. Whether or not they obey those instructions is up the bot. – Brendon Dec 17 '13 at 0:06
  • 1
    And that's the question... what search engine bots disregard the robots.txt and still index your site? – Fredy31 Dec 17 '13 at 0:11
3

Even Google will occasionally include a site in its search index, despite the fact that the site disallows all crawling with robots.txt. Google views robots.txt rules as preventing crawling, but not necessarily inclusion in the search index.

When Googlebot can't crawl a site, but the site has lots of external links, Google may include pages from the site in the index and use the anchor text of the links to determine what the page is about.

Google's Matt Cutts blogged about this. He gave an example of the Department of Motor Vehicles in California. Its website disallowed crawling, but it would be ridiculous if people couldn't find it on Google. So they came up with a way of including sites like that without being able to crawl them.

0

I believe that the robot.txt file is just like asking the search engine not to index you. There is no actual disabling of indexing, it is up to the search engines not to index you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.