2

I have my website, say www.example.com, and I want it to be indexed. Anyway I also have one section, www.example.com/section that I don't want to be indexed.

I tried adding the following headers to the pages under that section:

X-Robot-Tag: noindex,nofollow

Moreover, my robots.txt file says:

User-agent: *
Disallow: /section/

I waited about 24h but, searching for "example.com section" on the google the page is still there, at the first place!

What am I doing wrong? Isn't there any way to prevent indexing only a specific section?

Improve this question
3
  • 2
    "X-Robot-Tag: noindex,nofollow" - It should be X-Robots-Tag (with an s).
    – MrWhite
    Commented Feb 23, 2020 at 14:47
  • "I waited about 24h" - It can take longer than 24h for pages to be removed from the index. It depends how often Google is crawling your site.
    – MrWhite
    Commented Feb 23, 2020 at 14:51
  • "can longer than 24h" should say "can take months". The first pages could be removed in 24 hours, but if you are trying to remove thousands of pages, it could will likely take months until Googlebot gets around to re-crawling every single one of them to see the tags you have put in.
    – Stephen Ostermiller ♦
    Commented Feb 23, 2020 at 15:25

1 Answer 1

Reset to default
4

You are sending Google two conflicting commands: you are telling them to NOINDEX some pages but then you are also preventing them from reading the NOINDEX header by blocking them from crawling these pages (in the robots.txt) so they can't see the NOINDEX command.

You should only use the NOINDEX tag and let their bots crawl your website in order to see it;

remove the:

Disallow: /section/

from your robots.txt file and next time they crawl your website, they will see the NOINDEX tag and will know not to index these pages (of course, they can still decide to INDEX these pages based on other signals, such as backlinks, but this is the correct, technical implementation).

Improve this answer
4
  • "they can still decide to INDEX these pages based on other signals, such as backlinks" - not if X-Robots-Tag: noindex is set. That would apply if the page was blocked by robots.txt.
    – MrWhite
    Commented Feb 23, 2020 at 14:50
  • The robots file can only prevent crawling, so yes, backlinks can make them indexed but the NOINDEX tag is also not a guarantee, I have a few cases from different websites that even with the tag in place, Google decided to index the page (rare but still happens)
    – Andre Guelmann
    Commented Feb 23, 2020 at 15:08
  • So robots.txt is useless? If I remove robots.txt and just leave the noindex would solve my problem?
    – Phate
    Commented Feb 23, 2020 at 18:33
  • Robots.txt is very important, it just shouldn't be used to NOINDEX pages, so yes, you can remove the command I mentioned.
    – Andre Guelmann
    Commented Feb 23, 2020 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.