Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am running static content through a CDN, cdn1-cdn5 I am using such subdomains for that.

I am loading just images, CSS and JS files this way, but apparently Google has indexed some pages on subdomains and they now appear in the Google index and they are duplicates of my "normal" pages.

The thing is that CDN is set the way to have files appear on subdomains without any extra uploading of stuff, meaning subdomains are mirror copies of content that is on main site, I can't upload files to subdomains, I can upload to main site and change www to cdn1 in address bar to show the same content through the CDN as is on my site.

I have 2 questions:

  1. how do I remove the subdomains from Google index in GWT if it only allows to write anything that goes after `http://domain.com/ ?
  2. how do I prevent from bots indexing the pages on subdomains when I can't upload special robots.txt files or upload a google's verification files to them to prove my ownership in GWT?

Maybe there is something else that I need to know related to this matter?

UPDATE: text in bold is updated

share|improve this question
    
Are you only interested on robots.txt solutions? If so, the question will remain closed as duplicate, but if other solutions work for you, rephrase the question and flag it for moderation attention. –  PatomaS Mar 4 at 12:52

3 Answers 3

You can remove the sub-domains in webmaster tools, but first you need to add the sub domains as seperate sites and then submit a site removal. They should be gone within a day or so.

See these instructions for removing a site from google : https://support.google.com/webmasters/answer/1663427?hl=en

share|improve this answer
    
how do I prove my ownership of the subdomains in GWT if I cannot upload the files or edit the pages in there (see my updated question please) –  CamSpy Mar 4 at 14:47
    
If you have access the to domain name info then you can verify by dns. –  user29671 Mar 4 at 14:58

There are different way, here you have some, you can use only one or combine them

About the robots.txt, you can use it, but it's much better option to use a solution that is more robust and that all crawlers will have to follow, like the redirect.

Here you can see a short video from Matt Cutts talking about 301 redirects vs rel="canonical". A short extract from that page and video would be:

Okay, I sometimes get a question about whether Google will always use the url from rel=canonical as the preferred url. The answer is that we take rel=canonical urls as a strong hint, but in some cases we won’t use them:

  • For example, if we think you’re shooting yourself in the foot by accident (pointing a rel=canonical toward a non-existent/404 page), we’d reserve the right not to use the destination url you specify with rel=canonical.
  • Another example where we might not go with your rel=canonical preference: if we think your website has been hacked and the hacker added a malicious rel=canonical. I recently tweeted about that case. On the “bright” side, if a hacker can control your website enough to insert a rel=canonical tag, they usually do far more malicious things like insert malware, hidden or malicious links/text, etc.

On the video he mentions some more situations and reasons, like the fact that 301 has to be followed by everybody.

share|improve this answer
    
@pantomas I want pages removed from index and not indexed in the future, not use of canonicalization. –  CamSpy Mar 4 at 14:45
    
@CamSpy That's what the 301 is for. –  PatomaS Mar 4 at 23:26

Short answer.

Put <meta name="robots" content="noindex"> in the header of your HTML for all pages. Once the search engines have spidered these pages and you are sure of it, put

User-agent: *
Disallow: /

...in a robots.txt file in the root directory of each sub-domain.

This will take time of course. It can take 30-60 days typically for say Google to notice the changes and reflect it in the SERPs. It can take less or more time depending upon how Googles gauges freshness for your sub-domains.

share|improve this answer
    
I updated the question with more detailed info, telling that content on subdomains is mirror of the main site for CDN purposes, I cannot edit pages on subdomains or upload anything to them –  CamSpy Mar 4 at 14:46
    
what do I add after Disallow: to have cdn1.xxx.com not allowed to be indexed in the robots.txt of the xxx.com domain? –  CamSpy Mar 4 at 19:40
    
Actually, you would put the robots.txt in the sub-domain root directories. You cannot refer to sub-domains from the parent domain robots.txt file because search engines (and the web by default) sees these as separate sites. –  closetnoc Mar 4 at 19:51
    
subdomains are virtual, they don't contain any special files on their own since they only have mirror versions of files from the main site. The file that I upload to main site is automatically available on all 5 CDN subdomains... –  CamSpy Mar 4 at 22:38
    
Wow. You did say that before didn't you? You got me. I am stumped. You may have to deal with Google and Bing through their respective Webmaster Tools. One question: If someone accesses a file on the sub-domain, does it appear in the parent domain access log files? If so, perhaps a blanket 301 redirect from the sub-domains to the parent domain in the .htaccess on the parent domain would do. –  closetnoc Mar 4 at 23:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.