Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have some web pages that google has indexed. Now I added a robots.txt file to exclude them but should I also add a “noindex” tag and if so then how can I best add this?

share|improve this question
add comment

migrated from stackoverflow.com Jun 28 '11 at 17:32

This question came from our site for professional and enthusiast programmers.

6 Answers

The robots.txt should be enough but Google will not make the change until the next time they index your website. Which if your website is low traffic may be a while.

share|improve this answer
add comment

You would use code similar to this:

<html>
<head>
 <meta name="robots" content="noindex" />
 <title>Your Page Title</title>
</head>
share|improve this answer
    
Is there a fancy way to do this with MVC? –  MIMI Jun 28 '11 at 17:03
add comment

If you want to block search engines from crawling and/or indexing your pages you can use a few methods:

1) Use robots.txt. Google will honor it and remove those pages automatically although there is no timetable for them to get it done

2) Use meta tags

<meta name="robots" content="noindex, nofollow">

3) Use http headers

Header set x-robots-tag: noindex

4) Use rel="nofollow" on any links that point to those pages.

<a href="http://www.example.com/sample.html" rel="nofollow">Link to page I don't want indexed</a>

You can also tell Google to remove those pages from their index in Google Webmaster Tools.

share|improve this answer
add comment

Not being indexed is nothing difficult if you understand how crawlers work, and by that I don't mean understanding their algorithm, just basic mechanics. Also, there is no need to consider how a specific crawler works at an specific moment since they may change their behaviour at any given time.

They check ips and links, doesn't matter how they get their first resource, but once they got one, they follow links from there and try some basic combinations, which means that if is reachable, it will be indexed.

So, if you want to avoid being indexed, you want to "not be there". How can you achieve that?

  1. No being there. Don't publish
  2. Hide behind a closed door. Like a protected system with user and password.
  3. Use HTTP codes like 307 or 302.

Option 1 is is obvious, so it doesn't require much details.

Option 2 is very good one while you are testing on live environment, showing to a client, doing some maintenance, etc. It requires that you implement some user/password system and so, most people won't do this unless they are using a prepackaged system that includes such mechanism. Still is advisable. You can combine this with method 3 if you use IP restrictions, for instance to allow your self or a client to look at the website but nobody else.

Option 3 is easier to implement than option 2, but may not be the right one on all situations. If you need limited access, remember to craft the rules allowing some IPs. This option is recommended a lot of times using 5xx codes. Don't do that. There is nothing wrong with your server, so don't give wrong information. If the request is made with version 1.1, or newer, of the HTTP protocol, the answer should be 307. If the request is made with version 1.0, the answer should be 302. Both options imply that you should provide an alternate URL, it may be a page explaining the reasons (ideally), or at least a page on your site.

This options don't assume that you had content and moved it, which is the case for a 301 redirect. Assume the situation that you are publishing new content and you don't want it to be indexed.

share|improve this answer
add comment

Google will respect your robots.txt directives. The noindex is not needed for Google bots.

See this page for more info about ways to prevent bots from indexing your content.

share|improve this answer
add comment

Important update

Google may not respect your robots.txt directives, as stated here: https://developers.google.com/webmasters/control-crawl-index/docs/faq#h17

However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.

Don't know when Google changed this, but is how it works right now.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.