3

In my robots.txt file (http://www.tutorvista.com/robots.txt), I'm using Noindex: /content/... to disallow indexing:

enter image description here

This should mean that http://www.tutorvista.com/content/ and anything below this URL shouldn't be indexed. But in the image of my search results below, you can see that pages under this URL are being indexed:

enter image description here

Additionally, I'm using Disallow: /biology/ which means that http://www.tutorvista.com/biology/ and anything below this shouldn't be crawled. But in the image of my search results, you can see that pages under this URL are being crawled and indexed.

enter image description here

So can anyone tell me what's wrong with my robots.txt directives?

Improve this question
3
  • Google's [webmaster tools](HTTPS://google.com/webmasters/tools) (select a site and go to Crawl | Robots tester) allows you to test your robots.txt against various paths on your site. Doing that with your robots.txt shows that (for example) Allow: /biology/ is the justification for /biology/abdominal-cavity-organs being allowed rather than the root, and because these rules come before the exact disallow rule.
    – Zhaph - Ben Duguid
    Sep 12, 2017 at 7:14
  • And with reference to your search result shot, you have the following allow rule: Allow: /biology/animations/ above the disallow block, so that will take precedence.
    – Zhaph - Ben Duguid
    Sep 12, 2017 at 7:17
  • @Zhaph-BenDuguid Thank you for your reply. Yes you are right! Now I understand.
    – Goutham Jabez GJ
    Sep 13, 2017 at 10:53

2 Answers 2

Reset to default
7

"noindex" directives should not be used in your robots.txt file, instead a noindex meta tag should be added to any pages that you don't want indexed in Google.

A NOINDEX tag looks like the below and it should be placed in the section of any page you do not want indexed:

<meta name="robots" content="noindex">

More information can be found here.

In the second example while you do have "Disallow: /biology/" in your robots.txt file, a few lines above this you also have "Allow: /biology/animations/" hence why this page in indexed in your example.

Hope this helps!

Improve this answer
3
  • 2
    Actually Noindex is supposed to work in robots.txt: How does “Noindex:” in robots.txt work? but Google has said they may remove support for it, which may be what happened here. If you are going to use it, use it in conjunction with the noindex meta tag.
    – Stephen Ostermiller ♦
    Sep 1, 2017 at 10:21
  • @matthew Thanks for the reply. In the Second example image, check the fourth result it has /biology. but i have actually disallowed using Disallow: /biology/ . how did it show up?
    – Goutham Jabez GJ
    Sep 5, 2017 at 9:36
  • 1
    Yeah, always have a backup plan, like @StephenOstermiller suggested :). Unofficial extensions may or may not stick around, and behavior will be different across search engines anyway.
    – John Mueller
    Sep 6, 2017 at 9:16
6

Note that Noindex is not part of the original robots.txt specification. Google supported it as experimental feature (see: How does “Noindex:” in robots.txt work?), but it’s not clear if that is still the case (as they didn’t document it to begin with). But let’s assume it is.

Your robots.txt has two problems.

Empty lines

A record must not contain empty lines. Empty lines are used to separate records.

A conforming bot (which doesn’t identify as Googlebot-Image/Adsbot-Google/Mediapartners-Google) uses this record:

User-agent: *
Allow: /

So none of the following Disallow/Allow/Noindex lines apply.

Of course a bot may try to "fix" this and interpret the following lines to be part of this record (i.e., ignoring the blank lines), but the robots.txt spec doesn’t define this, so I wouldn’t count on it.

... in Noindex values

If Noindex works like Disallow (which we don’t know for sure, as Noindex is not specified/documented, but I guess it wouldn’t make sense to specify it differently), the ... you appended to the values mean that ... must appear in the URLs you want to noindex.

The line

Noindex: /content/biology/...

would apply to a URL like /content/biology/.../foobar, but not to a URL like /content/biology/foobar nor /content/biology/.

So if you want every URL whose paths starts with /content/biology/ to be noindexed, you would have to specify:

Noindex: /content/biology/
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.