2

I need to de-index some content from Google.

We have a main domain main.example.com which has listing of E-commerce products. Also, we give this for internal use to companies on their subdomain. Example:

company1.example.com
company2.example.com
…
company100.example.com

Problem is, Google is detecting the content of our main site as duplicate.

How can this be prevented? I am ok to block crawling of microsites as they are for internal use only.

Improve this question

1 Answer 1

Reset to default
0

The easiest way to prevent indexing is to serve the X-Robots-Tag: noindex HTTP header from all of those subdomains.

You could also add the robots noindex meta tag to all pages, but that only works for HTML documents.

More info: https://developers.google.com/search/reference/robots_meta_tag

Improve this answer
2
  • Thanks Maximillian. What if I add Disallow in robots.txt? Will that be serving any purpose.
    – Ankit Aggarwal
    Commented May 17, 2019 at 3:32
  • @AnkitAggarwal It often works in practice, but it's not a 100% solution because Google is known to index pages in some cases even if they are disallowed from crawling them. Using the noindex tag or header is the way that works 100%.
    – Maximillian Laumeister ♦
    Commented May 17, 2019 at 4:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.