1

Working on a client's site, I noticed an error when trying to use a duplicate content checker tool. The tool stated that the home page was blocked by robots.txt.

In search console, I can see the robots.txt does indeed contain the lines:

User-agent: *
Disallow: /

But when I test it, it says "Allowed" for any page I check.

Also worth noting that all pages in sitemap are indexed, no errors/conflicts reported. The only thing I can think of is that the above lines in the robots file are then followed by a link to the sitemap.

Does this then override the Disallow command?

Improve this question
1
  • If you don't mind, please post the whole content of the robots.txt (e.g. with altered URLs). Some other content might interfere with the disallow and render it invalid.
    – schwarzbrot
    Commented Sep 26, 2018 at 8:04

1 Answer 1

Reset to default
1

A link to a sitemap file should not invalidate the Disallow command in the setup that you have described (specifically: if the directives occur in that order).

Things that I would look out for:

  • Does the "Disallow" line follow immediately after the "User-agent" line? (invalid commands in this place might lead to the "Disallow" being ignored)
  • Is there more than one "User-agent" block in the file? (might lead to Googlebot ignoring the "Disallow")
  • If there is more other stuff inside the robots.txt file: might it otherwise interfere with the disallow?

On one occasion I have come across a robots.txt file that contained invisible unicode characters. As a result of this paths where not as they looked in the browser. Solution: run file through a text editor that is able to convert between different encodings, to get rid of the special characters.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.