<div id="papers" NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> content -- Does this work to block search engines to crawl me? </div>
migrated from superuser.com Jan 21 '11 at 1:08
This question came from our site for computer enthusiasts and power users.
The age-old standard for managing robots is /robots.txt.
As it is, your markup will not be interpreted by robots as any sort of request, and will invalidate your HTML.
If you want to block search engines from crawling and/or indexing your pages you can use a few methods:
1) Use robots.txt
2) Use meta tags
3) Use http headers
4) Use rel="nofollow"
5) Place the content behind a login. Search engines (generally) do not submit forms nor create accounts at websites.
6) Block the bots of all of the major search engines using .htaccess but that would be tedious and prone to error if they change whatever it is you would use to identify them (e.g. ip address, user agent).