I need to de-index some content from Google.

We have a main domain main.example.com which has listing of E-commerce products. Also, we give this for internal use to companies on their subdomain. Example:


Problem is, Google is detecting the content of our main site as duplicate.

How can this be prevented? I am ok to block crawling of microsites as they are for internal use only.


The easiest way to prevent indexing is to serve the X-Robots-Tag: noindex HTTP header from all of those subdomains.

You could also add the robots noindex meta tag to all pages, but that only works for HTML documents.

More info: https://developers.google.com/search/reference/robots_meta_tag

  • Thanks Maximillian. What if I add Disallow in robots.txt? Will that be serving any purpose. – Ankit Aggarwal May 17 at 3:32
  • @AnkitAggarwal It often works in practice, but it's not a 100% solution because Google is known to index pages in some cases even if they are disallowed from crawling them. Using the noindex tag or header is the way that works 100%. – Maximillian Laumeister May 17 at 4:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.