5

It seems common wisdom that using X-Robots-Tag/meta robots with robots.txt to block a URL from being indexed "can cause issues", e.g.:

Using both X-Robots-Tag and meta robots on a URL is redundant because they are equivalent, and using both robots.txt and either of the others for a URL can cause issues because robots.txt blocks crawling, and crawling is required for a bot to even see either of the other ones since they are document-level directives.

Source: https://webmasters.stackexchange.com/a/130710

However, I cannot see what the "issue" is. Suppose I have a URL /foo.html for which crawling is blocked using robots.txt. Then the bot will not crawl the page, but it might still be indexed if foo.html is followed from another site. However, if we add a meta robots in foo.html, indexing will be blocked when the bot reaches the page from another site, independently of what happens to be in robots.txt.

Therefore, while it might be redundant to use both methods, I am not sure why doing so can cause foo.html to be indexed.

1
8

The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek at it!

Suppose I have a URL /foo.html for which crawling is blocked using robots.txt. Then the bot will not crawl the page, but it might still be indexed if foo.html is followed from another site.

Correct.

However, if we add a meta robots in foo.html, indexing will be blocked when the bot reaches the page from another site, independently of what happens to be in robots.txt.

There is a subtle misunderstanding here - the bot never actually reaches the page. What happens is the following:

  1. The bot sees the link to foo.html on an external site.

  2. The bot prepares to crawl foo.html, but first retrieves the robots.txt file from the domain on which foo.html is hosted, to make sure it is allowed to crawl foo.html.

  3. The bot notices that crawling foo.html is prohibited, so it does not request foo.html at all. It is unable to see any of the contents of foo.html, including the robots noindex meta tag.

  4. Unable to see whether the page has a noindex meta tag or not, the crawler makes its own decision for whether to index the page. It may decide to index the page, which it's allowed to do because it hasn't seen the noindex directive telling it not to index.

  5. You end up with the situation where both robots.txt and noindex were used, but the page still gets indexed.

2
  • Thank you Maximillian Laumeister. It also seems that steps 1-3 could also happen in principle when the bot sees a link to foo.html from a page on the current website, e.g., index.html, is that right?
    – jII
    Oct 13 '21 at 16:22
  • 1
    @jII Yep! When the bot sees a link to a page whether it's an internal link or an external link, it will want to crawl that page. But it will consult robots.txt first (it caches the robots.txt response for up to 24 hours). Oct 13 '21 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.