0

I use a cookieless sub domain static.example.com to serve all images, js, and css files. This static sub domain has as its root directory the same directory as the parent http://example.com. So static.example.com/index.php will result in calling the same file as http://example.com/index.php

Google has taken up indexing static.example.com and I need to stop this.

I can't modify robots.txt since it would apply to both domains. Webmaster tools allows "temporary" removal of a link, but this needs to be permanent.

I registered this static domain through google as a separate site from example.com, but this doesn't seem to buy me anything since I can't see a way to block crawling without using robots.txt - which would block it from the parent domain.

Any other ideas?

UPDATE

Found this conversation on Webmaster Help Forum

I am looking into whether it's possible to insert a META tag dynamically into the header of every page called on the static domain using php

CLARIFICATION

I am not putting any content on static.domain.com. There are NO webpages on it that are called - it only serves images. I have it redirecting to the same root directory as the main site for ease of coding image links (not every image is fed from the same image path).

Google got a hold of the static web root address - probably because .htaccess redirects to index.php if a file is missing and then crawled it. Now Google is showing the static version of the same pages with search results.

  • Why are you doing that? – Steve Dec 22 '16 at 8:01
  • @Steve - I updated the question with a clarification to answer your comment. – mseifert Dec 22 '16 at 8:11
  • understand what you are doing. I am just wondering why you are using a subdomain of the primary domain to deliver images. Why bother? – Steve Dec 22 '16 at 8:35
  • @Steve - To be cookieless - by calling a subdomain, every request for an image will not have a cookie sent. – mseifert Dec 22 '16 at 8:40
  • @steve when we use some third party service like analytics, ads etc in our main website then our static content like images, js also download same cookies in every request. Hence cookieless domain speed up webpage speed, – Goyllo Dec 22 '16 at 8:54
0

You should not redirect your user/Googlebot to your main subdomain when your static assets is not found. Instead you should return 404 error. But because you're redirecting them, hence your subdomain is index by Google, so

There is three solution for you.

  1. Redirect only your main static.domain.com page to domain.com, for example facebook redirect their static content cdn domain (fbcdn.net) to facebook.com, so Google will follow the redirection and will drop your subdomain link from search result.
  2. Use below robots.txt

User-Agent: Googlebot Allow: .js Allow: .css Allow: .png Allow: .jpg Disallow: /

  1. Use meta noindex tag on that webpage.

Make sure the noindex tag only placed on those webpages that you don't want to see on search result, I am telling that, because may be you're using some kind of parent template technique which applied to all child templates.

Another thing is that noindex tag don't stop from crawling, it just used to noindex specific webpage, so that page will not going to appear on Google search result, so if you're using any links on that webpage then Google will still follow all the links and pass it's value just like any normal webpage.

  • I am not putting any content on static.domain.com. There are NO webpages on it that are called - it only serves images. Google got a hold of the static web root address - probably because .htaccess redirects to index.php if a file is missing and then crawled it. Now Google is showing the static version of the same pages with search results. – mseifert Dec 22 '16 at 8:07
  • "I am not putting any content on static.domain.com...it only serves images" images = content – Steve Dec 22 '16 at 8:36
  • I update my answer, based on your comments :) – Goyllo Dec 22 '16 at 8:40
  • Since the robots.txt and .htaccess are shared between the domain an the subdomain (the real problem, I now see), it seems the best solution would be to makes changes in .htaccess to redirect differently based on the domain / subdomain. Currently, my redirect is simply: RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule ^(.*)$ index.php I will have to figure out how to specify the domain in the rewrite condition and rule. – mseifert Dec 22 '16 at 8:57
  • I am not expert in htaccess so can't add anything on that. Sorry :( But I will say don't use any kind of redirection when resource is not found, let server display the 404 error. – Goyllo Dec 22 '16 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.