5

I have some web pages that Google has indexed. I have now added a robots.txt file to exclude them, but should I also add a noindex tag? If so, then how do I do so?

Improve this question
1
  • I would suggest to rephrase your title to make it very clear which of the 2 you are enquiring: 1. Asking about removing URLs already indexed in google search index 2. preventing URLs from being crawled and indexed in the first place. The answers may differ. For the latter robots.txt may suffice, for the former it may likely not.
    – Sybille Peters
    Aug 21, 2019 at 8:54

6 Answers 6

Reset to default
4

Important update

Google may not respect your robots.txt directives, as stated here: https://developers.google.com/webmasters/control-crawl-index/docs/faq#h17

However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.

Don't know when Google changed this, but is how it works right now.

Improve this answer
3

The robots.txt should be enough but Google will not make the change until the next time they index your website. Which if your website is low traffic may be a while.

Improve this answer
3

If you want to block search engines from crawling and/or indexing your pages you can use a few methods:

1) Use robots.txt. Google will honor it and remove those pages automatically although there is no timetable for them to get it done

2) Use meta tags

<meta name="robots" content="noindex, nofollow">

3) Use http headers

Header set x-robots-tag: noindex

4) Use rel="nofollow" on any links that point to those pages.

<a href="http://www.example.com/sample.html" rel="nofollow">Link to page I don't want indexed</a>

You can also tell Google to remove those pages from their index in Google Webmaster Tools.

Improve this answer
1
1

You would use code similar to this:

<html>
<head>
 <meta name="robots" content="noindex" />
 <title>Your Page Title</title>
</head>
Improve this answer
0
0

Not being indexed is nothing difficult if you understand how crawlers work, and by that I don't mean understanding their algorithm, just basic mechanics. Also, there is no need to consider how a specific crawler works at an specific moment since they may change their behaviour at any given time.

They check ips and links, doesn't matter how they get their first resource, but once they got one, they follow links from there and try some basic combinations, which means that if is reachable, it will be indexed.

So, if you want to avoid being indexed, you want to "not be there". How can you achieve that?

  1. No being there. Don't publish
  2. Hide behind a closed door. Like a protected system with user and password.
  3. Use HTTP codes like 307 or 302.

Option 1 is is obvious, so it doesn't require much details.

Option 2 is very good one while you are testing on live environment, showing to a client, doing some maintenance, etc. It requires that you implement some user/password system and so, most people won't do this unless they are using a prepackaged system that includes such mechanism. Still is advisable. You can combine this with method 3 if you use IP restrictions, for instance to allow your self or a client to look at the website but nobody else.

Option 3 is easier to implement than option 2, but may not be the right one on all situations. If you need limited access, remember to craft the rules allowing some IPs. This option is recommended a lot of times using 5xx codes. Don't do that. There is nothing wrong with your server, so don't give wrong information. If the request is made with version 1.1, or newer, of the HTTP protocol, the answer should be 307. If the request is made with version 1.0, the answer should be 302. Both options imply that you should provide an alternate URL, it may be a page explaining the reasons (ideally), or at least a page on your site.

This options don't assume that you had content and moved it, which is the case for a 301 redirect. Assume the situation that you are publishing new content and you don't want it to be indexed.

Improve this answer
0

Google will respect your robots.txt directives. The noindex is not needed for Google bots.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.