2

When I'm checking the Google results for our site's site:example.com, we get lots of index results like

  • xyz.example.com
  • xyz.example.com/abe
  • xyz.example.com/abcds

How do I disallow this type URL, so they don't show up in the search results?

1

This is definitely the case when you should use meta tag "robots" and "noindex" the subdomain or what ever pages you want to hide from serps. After that, you could consider using url removal tool from google and getting rid of pages you don't want to see in serps, which will speed up the process a bit.

Do not go with robots.txt and blocking content of this type.

1

The correct answer to your question is to put a robots.txt file at the root folder of your subdomain or put robots meta tags on your subdomain's index page to block them from crawling and indexing.

1

It has very simple solution. You need to put the Robots.txt file in the root folder of your sudomain (i.e xyz.example.com).

After that you can use Robots meta tag to disallow SE from crawling your site (i.e xyz.example.com/abe, xyz.example.com/abcds).

-2

Try:

User-agent: * 
Disallow: xyz.example.com 

Or

Disallow: xyz.example.com/abc
Disallow: xyz.example.com/abcd
Disallow: xyz.example.com/abcde
  • That isn't correct. Robots.txt never has host names in it. – Stephen Ostermiller Jul 24 at 0:21
  • I'm not talking about robots.txt for exemple.com u need to add this in robots.txt xyz.exemple. com directory – ben teber Jul 24 at 20:26
  • Yes, you need to add the robots.txt in the directory root for the subdomain. You should edit your answer to say that. You should also edit out the "xyz.example.com" because those are not correct. – Stephen Ostermiller Jul 24 at 22:19
-2

When you have to execute your file, you allow it and you do not have to.

User-agent: * 
Disallow: xyz.example.com
Disallow: /wp-admin/
  • What is Disallow: xyz.example.com supposed to do? You can't have host names in a robots.txt file. If you are intending that to disallow a directory, that won't work either because directories start with a slash. – Stephen Ostermiller Jul 29 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.