5

I'm working on a website that's normally accessed trough a load-balance. But we also have direct URL to each server.

Lately, Google started to index the direct URLs to our servers, which is bad because we don't want our user to go directly to each server (if some server is taken down for maintenance).

We've correct canonical link tags pointing to the load-balance URL. All links in the sitemap.xml point to the load-balance URL. So we have no idea how Google got a hand on the direct server URLs.

To be extra clear: Our load-balance is foo.com. All the links on the site use this URL. So does the sitemap and canonical-tags.

But Google still indexes foo.server01.webhost.com and foo.server02.webhost.com

We cannot add 301-redirects from foo.server**.webhost.com to foo.com due to various reasons.

Do I have to verify that I own all the URLs in Google Search Console? Or am I missing out some other settings?

Improve this question
12
  • Is the user externally redirected to foo.server01.webhost.com - so the URL is exposed? Or is this rewritten internally (using a proxy)? "Do I have to verify that I own all the URLs in Google Search Console?" - it was my understanding that the target of a cross-domain canonical should also be a verified property in GSC, however, I can't find a reference for this at the moment. Note that the rel="canonical" is only advisory so there is no guarantee that Google will use it.
    – MrWhite
    Feb 22, 2016 at 9:26
  • No, the users are never exposed to foo.server01.webhost.com. The load balance is rewriting the response.
    – Kevin Candlert
    Feb 22, 2016 at 9:32
  • (I meant, target and "source" domains should be verified, not just the target.)
    – MrWhite
    Feb 22, 2016 at 9:42
  • I'll verify that I own the source/server URLs and see if that makes any difference. Thank you! :)
    – Kevin Candlert
    Feb 22, 2016 at 9:55
  • 1
    I found this article yoast.com/rel-canonical where Yoast state that SearchEngineJournal has articles with canonical-tags linking back to Yoast. See "Cross domain canonical". This makes me wonder if you really have to "own" both URLs in Google Search Console.
    – Kevin Candlert
    Feb 22, 2016 at 9:59

1 Answer 1

Reset to default
1

...Lately, Google started to index the direct URLs to our servers, which is bad...

Remove the link tag with rel="canonical" in it and place the following tag between <head> and </head> of the HTML:

<meta name="ROBOTS" content="NOINDEX">

This will tell all search engines not to index the page. This should work with all search engines as some might not even understand and/or process rel=canonical. Then after saving the changes, give the search engines a few hours to a few weeks to process your request.

There's more information here: http://www.metatags.org/meta_name_robots

Improve this answer
3
  • If I were to put <meta name="ROBOTS" content="NOINDEX"> into the headers of both servers nothing will be indexed as the load balance is doing a reverse proxy. So this will effectively say to who ever visit foo.com not to index the site. All I want is to have google to not index foo.server01.webhosting.com but instead, use foo.com instead. That is where Canonical meta-tag comes in, as google themselves recommend to use 301 or canonical. 301 is out of the question and I'm wondering is there're any more actions when using the canonical tag.
    – Kevin Candlert
    Feb 23, 2016 at 8:45
  • No, you only apply that tag to the pages with duplicate content, not every page. Let search engines index original content.
    – Mike -- No longer here
    Feb 24, 2016 at 0:03
  • Put the pages are the same. Same page but served over different domains. I'm thinking about if:ing in a NOINDEX meta tag if the Request Host header contains server**.webhosting.com. I just wished that there was a solution to this problem without doing code changes.
    – Kevin Candlert
    Feb 24, 2016 at 11:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.