I want all my subdomains to not be indexed under my domain, that means when I search in Google for example.com to not appear in index test.example.com, but when I search test.example.com to be indexed.

Is it possible to add a Code in domain.com robots.txt to not index the subdomains when search on google site:example.com?

Now if I search on Google site:example.com Google shows results for all my sub-domains, but I want results only for example.com and not test.example.com.

Only when I search site:test.example.com I want the results for the sub-domain.

  • 1
    This is confusing. Can you clarify this some?? – closetnoc Feb 26 '16 at 22:18
  • Aren't you referring to the Google search results, rather than the actual "index" status of your subdomains? This would also make any reference to "robots.txt" irrelevant? – MrWhite Feb 26 '16 at 23:18

In Google, to do a site: search on the main domain (eg. example.com), but exclude the subdomain test from the search results you would use the search:

site:example.com -site:test.example.com

You would need to do this for each subdomain you wish to exclude from the results.

Is it possible to add a Code in domain.com robots.txt to not index the subdomains when search on google site:example.com?

However, you can't influence this behaviour in your server-side code, it is simply how Google's site: operator works. If your subdomains are indexed at all then they could be returned by a site:example.com search.

The only way to prevent your subdomain from appearing in the search results is to not have it indexed at all, which is not your intention.

  • Oh! Now I get it!! – closetnoc Feb 26 '16 at 23:37
  • 1
    I think he want to enforce it from the server (either on example.com or on test.example.com). A user who only knows about example.com and does a search for site:example.com should not even learn about the existence of test.example.com. At least that's how I read the question. – kasperd Feb 27 '16 at 7:45
  • 1
    @kasperd Yes, I think you're right - I've updated my answer. Thanks. – MrWhite Feb 27 '16 at 10:26

I usually handle this with an X-Robots-Tag HTTP header on any subdomains I create. Setting something like Header set X-Robots-Tag "noindex" in the Apache VHOST or .htaccess file, or add_header X-Robots-Tag noindex; in Nginx, will tell search engines not to index anything on that subdomain.

This will have to be done for each subdomain you create, but it's not so much of an extra effort.

  • 1
    Although they do appear to want the subdomain indexed, but only to appear in the SERPs for a site:subdomain.example.com search, but not included in a site:example.com search. – MrWhite Feb 27 '16 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy