3

Say I have a domain example.com

And it has a subdomain sub.example.com

And another subdomain internal.example.com

I have some links on example.com and sub.example.com for Google to crawl. But I don't want Google to fetch anything on internal.example.com

What should I do in this case?

Improve this question
2
  • 1
    Robots.txtx and any sitemap cannot refer to any other site. Sub-domains are sites unto themselves. This means that example.com and sub.example.com are completely different sites and would need their own robots.txt and sitemap. This is completely normal and search engines will find your files okay. Cheers!!
    – closetnoc
    Commented Dec 23, 2016 at 4:11
  • @closetnoc You can host your sitemaps on a subdomain, however. itunes.apples.com/robots.txt references sitemaps on sitemaps.itunes.apple.com. This is allowed.
    – Brendon
    Commented Dec 23, 2016 at 14:07

1 Answer 1

Reset to default
1

Here is link to Google's documentation about the validity of URLs robots.txt as per its location.

Do you have any links from the main domain or the other sub-domain to internal.domain.com?

If not then Google, in general, would not see your sub-domain on its own. I would still recommend to block it via the robots.txt file if they have never been exposed in past. If, by any chance URLs from internal.example.com have been exposed to Google then use Robots-Meta Tag to declare these pages non-indexable and no-follow.

Improve this answer
1
  • Can you add to this answer about what to do about sitemaps?
    – Stephen Ostermiller ♦
    Commented Dec 25, 2016 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.