10

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt.

Disallow: /page-one.html
Noindex: /page-two.html

Seems like it would prevent search engines from crawling page one, and prevent them from indexing page two.

Is this robots.txt directive supported by Google and other search engines? Does it work? Is it documented?

Improve this question
5
  • I dunno... bu this could be very helpful for some. As well, I would like to see the sitemap expanded to be more communicative. Between robots and sitemaps, this should be a golden opportunity to communicate back to search engines and others about a site. I am also in favor of a text based opportunity to inform search engines about the site in a different way separate from being available in search such as the about page as if you got the chance to talk directly to a Google site reviewer. It could save some heart-aches and misunderstandings. A chance to say Ooopppsss, I goofed- Sorry.
    – closetnoc
    Sep 10, 2015 at 14:55
  • I'm under the impression that disallow stops Google discovering links on the parent and child pages if any exist. While noindex simply stops the page being listed, it doesn't stop discovery while disallow does.
    – Simon Hayter
    Sep 10, 2015 at 16:41
  • @SimonHayter I know that is how nofollow works for meta tags. It would be nice to know if that is also the case for robots.txt.
    – Stephen Ostermiller ♦
    Sep 10, 2015 at 17:56
  • Hey @StephenOstermiller not only meta tags, its the same for <a rel="no-follow"> too. I see no reason why it would be treated any different. Obviously this isn't official and its even recommended by John Muller not to use it in the robots.txt but other than his tweet I haven't managed to find much information about it.
    – Simon Hayter
    Sep 10, 2015 at 18:18
  • It's worth noting that Google no longer flags the Noindex directive within robots.txt files as an error.
    – Aran
    Apr 12, 2019 at 9:29

1 Answer 1

Reset to default
9

Google used to unofficially support a Noindex directive in robots.txt, however in 2019, they announced that the directive will no longer work.

Here is what Google's John Mueller says about Noindex: in robots.txt:

We used to support the no-index directive in robots.txt as an experimental feature. But it's something that I wouldn't rely on. And I don't think other search engines are using that at all.

Before Google announced the feature was discontinued, deepcrawl.com did some testing of the feature and discovered that:

  • Before 2019, it still worked with Google
  • It prevented URLs from appearing in the search index
  • URLs that have been noindexed in robots.txt were marked as such in Google Search Console

Given that Google discontinued the feature, it shouldn't be used anymore.

Instead, use robots meta tags that are well supported and documented to prevent indexing:

<meta name="robots" content="noindex" />
Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.