Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 3 down vote favorite

I have some web pages that google has indexed. Now I added a robots.txt file to exclude them but should I also add a “noindex” tag and if so then how can I best add this?

share|improve this question

migrated from stackoverflow.com Jun 28 '11 at 17:32

4 Answers

up vote 4 down vote

The robots.txt should be enough But Google will not make the change until the next time they index your website. Which if your site is low traffic may be a while.

share|improve this answer
up vote 2 down vote

You would use code similar to this:

<html>
<head>
 <meta name="robots" content="noindex" />
 <title>Your Page Title</title>
</head>
share|improve this answer
Is there a fancy way to do this with MVC? – MIMI Jun 28 '11 at 17:03
up vote 2 down vote

If you want to block search engines from crawling and/or indexing your pages you can use a few methods:

1) Use robots.txt. Google will honor it and remove those pages automatically although there is no timetable for them to get it done

2) Use meta tags

<meta name="robots" content="noindex, nofollow">

3) Use http headers

Header set x-robots-tag: noindex

4) Use rel="nofollow" on any links that point to those pages.

<a href="http://www.example.com/sample.html" rel="nofollow">Link to page I don't want indexed</a>

You can also tell Google to remove those pages from their index in Google Webmaster Tools.

share|improve this answer
up vote 1 down vote

Google will respect your robot.txt directives. The noindex is not needed for Google bots.

See this page for more info about ways to prevent bots from indexing your content.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.