For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below.

Our robots.txt file is not linked from anywhere on the site and contains just the following:

User-agent: *
Crawl-delay: 5

This only happens for some sites. Why is this happening and how do we stop it?

enter image description here

Screenshot 1: Google Search console

enter image description here

Screenshot 2: Google search results

  • According to your screenshot you use robots.tx instead of robots.txt. I don't know how google found that file, but as it is not the default name for the robots file, it might not automatically exclude it form search result? – Pit Oct 7 '19 at 5:40
  • The file name didn't fit on mobile. Nice try. 😂 – Knocks X Oct 7 '19 at 5:54
  • Well ignore my comment then … – Pit Oct 7 '19 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.