1

Working on a client's site, I noticed an error when trying to use a duplicate content checker tool. The tool stated that the home page was blocked by robots.txt.

In search console, I can see the robots.txt does indeed contain the lines:

User-agent: *
Disallow: /

But when I test it, it says "Allowed" for any page I check.

Also worth noting that all pages in sitemap are indexed, no errors/conflicts reported. The only thing I can think of is that the above lines in the robots file are then followed by a link to the sitemap.

Does this then override the Disallow command?

  • If you don't mind, please post the whole content of the robots.txt (e.g. with altered URLs). Some other content might interfere with the disallow and render it invalid. – schwarzbrot Sep 26 '18 at 8:04
1

A link to a sitemap file should not invalidate the Disallow command in the setup that you have described (specifically: if the directives occur in that order).

Things that I would look out for:

  • Does the "Disallow" line follow immediately after the "User-agent" line? (invalid commands in this place might lead to the "Disallow" being ignored)
  • Is there more than one "User-agent" block in the file? (might lead to Googlebot ignoring the "Disallow")
  • If there is more other stuff inside the robots.txt file: might it otherwise interfere with the disallow?

On one occasion I have come across a robots.txt file that contained invisible unicode characters. As a result of this paths where not as they looked in the browser. Solution: run file through a text editor that is able to convert between different encodings, to get rid of the special characters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.