When I'm checking the Google results for our site's site:example.com, we get lots of index results like

  • xyz.example.com
  • xyz.example.com/abe
  • xyz.example.com/abcds

How do I disallow this type URL, so they don't show up in the search results?


Just include the following in the robots.txt and upload the file in the root folder of your subdomain you want to block from Search Engines. Verify using robot tester in the Search console tool.

User-agent: * 
Disallow: /

This is definitely the case when you should use meta tag "robots" and "noindex" the subdomain or what ever pages you want to hide from serps. After that, you could consider using url removal tool from google and getting rid of pages you don't want to see in serps, which will speed up the process a bit.

Do not go with robots.txt and blocking content of this type.


The correct answer to your question is to put a robots.txt file at the root folder of your subdomain or put robots meta tags on your subdomain's index page to block them from crawling and indexing.


It has very simple solution. You need to put the Robots.txt file in the root folder of your sudomain (i.e xyz.example.com).

After that you can use Robots meta tag to disallow SE from crawling your site (i.e xyz.example.com/abe, xyz.example.com/abcds).

  • The Robots meta tag is different from robots.txt. It doesn't disallow crawling, only indexing. – Maximillian Laumeister Jun 15 '20 at 17:40


User-agent: * 
Disallow: xyz.example.com 


Disallow: xyz.example.com/abc
Disallow: xyz.example.com/abcd
Disallow: xyz.example.com/abcde
  • 1
    That isn't correct. Robots.txt never has host names in it. – Stephen Ostermiller Jul 24 '19 at 0:21
  • I'm not talking about robots.txt for exemple.com u need to add this in robots.txt xyz.exemple. com directory – ben teber Jul 24 '19 at 20:26
  • 1
    Yes, you need to add the robots.txt in the directory root for the subdomain. You should edit your answer to say that. You should also edit out the "xyz.example.com" because those are not correct. – Stephen Ostermiller Jul 24 '19 at 22:19

When you have to execute your file, you allow it and you do not have to.

User-agent: * 
Disallow: xyz.example.com
Disallow: /wp-admin/
  • What is Disallow: xyz.example.com supposed to do? You can't have host names in a robots.txt file. If you are intending that to disallow a directory, that won't work either because directories start with a slash. – Stephen Ostermiller Jul 29 '19 at 14:45
  • This answer is very incorrect. – Maximillian Laumeister Jun 15 '20 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.