8

I am running static content through a CDN, cdn1-cdn5 I am using such subdomains for that.

I am loading just images, CSS and JS files this way, but apparently Google has indexed some pages on subdomains and they now appear in the Google index and they are duplicates of my "normal" pages.

The thing is that CDN is set the way to have files appear on subdomains without any extra uploading of stuff, meaning subdomains are mirror copies of content that is on main site, I can't upload files to subdomains, I can upload to main site and change www to cdn1 in address bar to show the same content through the CDN as is on my site.

I have 2 questions:

  1. how do I remove the subdomains from Google index in GWT if it only allows to write anything that goes after `http://domain.com/ ?
  2. how do I prevent from bots indexing the pages on subdomains when I can't upload special robots.txt files or upload a google's verification files to them to prove my ownership in GWT?

Maybe there is something else that I need to know related to this matter?

UPDATE: text in bold is updated

2
  • Are you only interested on robots.txt solutions? If so, the question will remain closed as duplicate, but if other solutions work for you, rephrase the question and flag it for moderation attention.
    – PatomaS
    Mar 4, 2014 at 12:52
  • When you remove/block the domain //cdn.xxxx.xx in WMT you also remove all the indexed pictures. Is that what you want?
    – user67659
    Aug 1, 2016 at 13:10

3 Answers 3

8

You can remove the sub-domains in webmaster tools, but first you need to add the sub domains as seperate sites and then submit a site removal. They should be gone within a day or so.

See these instructions for removing a site from google : https://support.google.com/webmasters/answer/1663427?hl=en

3
  • how do I prove my ownership of the subdomains in GWT if I cannot upload the files or edit the pages in there (see my updated question please)
    – CamSpy
    Mar 4, 2014 at 14:47
  • If you have access the to domain name info then you can verify by dns.
    – user29671
    Mar 4, 2014 at 14:58
  • Note that it doesn't tell you this, but you must include either https:// or http:// before your domain name, otherwise it will give you a (seemingly) unrelated error that the domain doesn't match the property, or something along those line. And if you're submitting a removal request for a subdomain (like dev.example.com) then I think this can only be done by first making a separate property for this domain -- it can't be done on the property for your top level domain name of example.com. Mar 21, 2022 at 14:37
1

Short answer.

Put <meta name="robots" content="noindex"> in the header of your HTML for all pages. Once the search engines have spidered these pages and you are sure of it, put

User-agent: *
Disallow: /

...in a robots.txt file in the root directory of each sub-domain.

This will take time of course. It can take 30-60 days typically for say Google to notice the changes and reflect it in the SERPs. It can take less or more time depending upon how Googles gauges freshness for your sub-domains.

7
  • I updated the question with more detailed info, telling that content on subdomains is mirror of the main site for CDN purposes, I cannot edit pages on subdomains or upload anything to them
    – CamSpy
    Mar 4, 2014 at 14:46
  • what do I add after Disallow: to have cdn1.xxx.com not allowed to be indexed in the robots.txt of the xxx.com domain?
    – CamSpy
    Mar 4, 2014 at 19:40
  • Actually, you would put the robots.txt in the sub-domain root directories. You cannot refer to sub-domains from the parent domain robots.txt file because search engines (and the web by default) sees these as separate sites.
    – closetnoc
    Mar 4, 2014 at 19:51
  • subdomains are virtual, they don't contain any special files on their own since they only have mirror versions of files from the main site. The file that I upload to main site is automatically available on all 5 CDN subdomains...
    – CamSpy
    Mar 4, 2014 at 22:38
  • Wow. You did say that before didn't you? You got me. I am stumped. You may have to deal with Google and Bing through their respective Webmaster Tools. One question: If someone accesses a file on the sub-domain, does it appear in the parent domain access log files? If so, perhaps a blanket 301 redirect from the sub-domains to the parent domain in the .htaccess on the parent domain would do.
    – closetnoc
    Mar 4, 2014 at 23:06
0

There are different way, here you have some, you can use only one or combine them

About the robots.txt, you can use it, but it's much better option to use a solution that is more robust and that all crawlers will have to follow, like the redirect.

Here you can see a short video from Matt Cutts talking about 301 redirects vs rel="canonical". A short extract from that page and video would be:

Okay, I sometimes get a question about whether Google will always use the url from rel=canonical as the preferred url. The answer is that we take rel=canonical urls as a strong hint, but in some cases we won’t use them:

  • For example, if we think you’re shooting yourself in the foot by accident (pointing a rel=canonical toward a non-existent/404 page), we’d reserve the right not to use the destination url you specify with rel=canonical.
  • Another example where we might not go with your rel=canonical preference: if we think your website has been hacked and the hacker added a malicious rel=canonical. I recently tweeted about that case. On the “bright” side, if a hacker can control your website enough to insert a rel=canonical tag, they usually do far more malicious things like insert malware, hidden or malicious links/text, etc.

On the video he mentions some more situations and reasons, like the fact that 301 has to be followed by everybody.

3
  • @pantomas I want pages removed from index and not indexed in the future, not use of canonicalization.
    – CamSpy
    Mar 4, 2014 at 14:45
  • @CamSpy That's what the 301 is for.
    – PatomaS
    Mar 4, 2014 at 23:26
  • It is advised to not use 301 redirects, but only 401 or 410 http status codes. I've experienced this recently on my own platform. My traffic is soaring! I believe it is because Google thinks you are "cloaking" pages, blackhat type SEO behavior is heavily penalized. Jul 5, 2019 at 7:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.