Say I have a domain example.com

And it has a subdomain sub.example.com

And another subdomain internal.example.com

I have some links on example.com and sub.example.com for Google to crawl. But I don't want Google to fetch anything on internal.example.com

What should I do in this case?

  • 1
    Robots.txtx and any sitemap cannot refer to any other site. Sub-domains are sites unto themselves. This means that example.com and sub.example.com are completely different sites and would need their own robots.txt and sitemap. This is completely normal and search engines will find your files okay. Cheers!! – closetnoc Dec 23 '16 at 4:11
  • @closetnoc You can host your sitemaps on a subdomain, however. itunes.apples.com/robots.txt references sitemaps on sitemaps.itunes.apple.com. This is allowed. – Brendon Dec 23 '16 at 14:07

Here is link to Google's documentation about the validity of URLs robots.txt as per its location.

Do you have any links from the main domain or the other sub-domain to internal.domain.com?

If not then Google, in general, would not see your sub-domain on its own. I would still recommend to block it via the robots.txt file if they have never been exposed in past. If, by any chance URLs from internal.example.com have been exposed to Google then use Robots-Meta Tag to declare these pages non-indexable and no-follow.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.