content -- Does this work to block search engines to crawl me?


The age-old standard for managing robots is /robots.txt. robots.txt asks robots not to crawl or index certain pages on your site. Your specific question seems to relate more to the Robots <META> tag, which belongs in the <head> of your document, and cannot be specified within a <div> tag somewhere in the body of your page.

As it is, your markup will not be interpreted by robots as any sort of request, and will invalidate your HTML.

| improve this answer | |

If you want to block search engines from crawling and/or indexing your pages you can use a few methods:

1) Use robots.txt

2) Use meta tags

<meta name="robots" content="noindex, nofollow">

3) Use http headers

Header set x-robots-tag: noindex

4) Use rel="nofollow"

<a href="http://www.example.com/sample.html" rel="nofollow">Link to page I don't want indexed</a>

5) Place the content behind a login. Search engines (generally) do not submit forms nor create accounts at websites.

6) Block the bots of all of the major search engines using .htaccess but that would be tedious and prone to error if they change whatever it is you would use to identify them (e.g. ip address, user agent).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy