According to matt cutts, even though we block the page using Robots.txt , it is better using noindex tag. ( Source : https://www.youtube.com/watch?v=KBdEwpRQRD0)

If it is blocked by Robots.txt, how Google bot read whether it has noindex tag or not? Because as I know if it blocked by Robots.txt, Google will not read any thing in that post.

1 Answer 1


Good catch! If you're blocking a page with robots.txt then crawlers will not able able to read the noindex meta tag. In these cases you should send the x-robots-tag HTTP header either via server-side code or .htaccess.

.htaccess example:

X-Robots-Tag: noindex

Sample PHP code:

header('X-Robots-Tag: noindex');

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.