Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

content -- Does this work to block search engines to crawl me?

share|improve this question

migrated from superuser.com Jan 21 '11 at 1:08

This question came from our site for computer enthusiasts and power users.

up vote 3 down vote accepted

The age-old standard for managing robots is /robots.txt. robots.txt asks robots not to crawl or index certain pages on your site. Your specific question seems to relate more to the Robots <META> tag, which belongs in the <head> of your document, and cannot be specified within a <div> tag somewhere in the body of your page.

As it is, your markup will not be interpreted by robots as any sort of request, and will invalidate your HTML.

share|improve this answer

If you want to block search engines from crawling and/or indexing your pages you can use a few methods:

1) Use robots.txt

2) Use meta tags

<meta name="robots" content="noindex, nofollow">

3) Use http headers

Header set x-robots-tag: noindex

4) Use rel="nofollow"

<a href="http://www.example.com/sample.html" rel="nofollow">Link to page I don't want indexed</a>

5) Place the content behind a login. Search engines (generally) do not submit forms nor create accounts at websites.

6) Block the bots of all of the major search engines using .htaccess but that would be tedious and prone to error if they change whatever it is you would use to identify them (e.g. ip address, user agent).

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.