I have some web pages that Google has indexed. I have now added a robots.txt file to exclude them, but should I also add a
noindex tag? If so, then how do I do so?
Google may not respect your robots.txt directives, as stated here: https://developers.google.com/webmasters/control-crawl-index/docs/faq#h17
However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.
Don't know when Google changed this, but is how it works right now.
If you want to block search engines from crawling and/or indexing your pages you can use a few methods:
1) Use robots.txt. Google will honor it and remove those pages automatically although there is no timetable for them to get it done
2) Use meta tags
<meta name="robots" content="noindex, nofollow">
3) Use http headers
Header set x-robots-tag: noindex
rel="nofollow" on any links that point to those pages.
<a href="http://www.example.com/sample.html" rel="nofollow">Link to page I don't want indexed</a>
Not being indexed is nothing difficult if you understand how crawlers work, and by that I don't mean understanding their algorithm, just basic mechanics. Also, there is no need to consider how a specific crawler works at an specific moment since they may change their behaviour at any given time.
They check ips and links, doesn't matter how they get their first resource, but once they got one, they follow links from there and try some basic combinations, which means that if is reachable, it will be indexed.
So, if you want to avoid being indexed, you want to "not be there". How can you achieve that?
- No being there. Don't publish
- Hide behind a closed door. Like a protected system with user and password.
- Use HTTP codes like 307 or 302.
Option 1 is is obvious, so it doesn't require much details.
Option 2 is very good one while you are testing on live environment, showing to a client, doing some maintenance, etc. It requires that you implement some user/password system and so, most people won't do this unless they are using a prepackaged system that includes such mechanism. Still is advisable. You can combine this with method 3 if you use IP restrictions, for instance to allow your self or a client to look at the website but nobody else.
Option 3 is easier to implement than option 2, but may not be the right one on all situations. If you need limited access, remember to craft the rules allowing some IPs. This option is recommended a lot of times using 5xx codes. Don't do that. There is nothing wrong with your server, so don't give wrong information. If the request is made with version 1.1, or newer, of the HTTP protocol, the answer should be 307. If the request is made with version 1.0, the answer should be 302. Both options imply that you should provide an alternate URL, it may be a page explaining the reasons (ideally), or at least a page on your site.
This options don't assume that you had content and moved it, which is the case for a 301 redirect. Assume the situation that you are publishing new content and you don't want it to be indexed.