Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 1 down vote favorite 
<div id="papers" NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

content -- Does this work to block search engines to crawl me?

</div>
share|improve this question
It does not block, it is a request to not crawl your site, it can be ignored if they wish to, most do not ignore this request. You have a low accept rate, you need to accept more answers. – Moab Jan 20 '11 at 23:14

migrated from superuser.com Jan 21 '11 at 1:08

2 Answers

up vote 2 down vote accepted

The age-old standard for managing robots is /robots.txt. robots.txt asks robots not to crawl or index certain pages on your site. Your specific question seems to relate more to the Robots <META> tag, which belongs in the <head> of your document, and cannot be specified within a <div> tag somewhere in the body of your page.

As it is, your markup will not be interpreted by robots as any sort of request, and will invalidate your HTML.

share|improve this answer
up vote 0 down vote

If you want to block search engines from crawling and/or indexing your pages you can use a few methods:

1) Use robots.txt

2) Use meta tags

<meta name="robots" content="noindex, nofollow">

3) Use http headers

Header set x-robots-tag: noindex

4) Use rel="nofollow"

<a href="http://www.example.com/sample.html" rel="nofollow">Link to page I don't want indexed</a>

5) Place the content behind a login. Search engines (generally) do not submit forms nor create accounts at websites.

6) Block the bots of all of the major search engines using .htaccess but that would be tedious and prone to error if they change whatever it is you would use to identify them (e.g. ip address, user agent).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.