5

Using site:example.com in Google is returning many results with the following format: https://www.w.example.com/services/edison/16mm-to-2k

Obviously this is not what I submitted and is not part of my sitemap. What are some solutions for dealing with this kind of problem?

This is particularly a problem since they indexed the HTTPS protocol and all the links are showing a warning before visiting the site as a result.

Getting wildcard SSL certificates for *.w.example.com and *.ww.example.com seems like a bad idea.

The site's DNS runs through AWS Route 53 and the site is running on an Ubuntu 12.04 EC2 with Apache.

Improve this question
1
  • It looks like your sitemap.xml has been moved from that location or deleted. Did you create this manually or are you using a plugin to create the sitemap.xml? If so which one?
    – Mohammad Ghorbni
    Feb 11, 2015 at 3:25

5 Answers 5

Reset to default
6

Most likely some part of your web site generated links like that, and that is how Google started to crawl the URLs.

You should check the links in your web pages to see where these incorrect URLs are, and you should fix them.

Also, you could change your Apache configuration so that requests for any other virtualhost than example.com or www.example.com would 301 redirect to the correct URL at www.example.com. This way Google will eventually index the correct versions.

Improve this answer
3
  • I'm positive I don't have any links that have www.w within mysite, that's the strange part.
    – egekhter
    Jun 22, 2014 at 20:14
  • I did the following: a) enforce SSL in my laravel app Route::filter('force.ssl', function() { if( ! Request::secure()) { return Redirect::secure(Request::getRequestUri()); } }); b) setup rewrite conditions in my apache virtualhost RewriteEngine On RewriteCond %{HTTP_HOST} !^www\.f [NC] RewriteCond %{HTTP_HOST} !^wp\.f [NC] RewriteRule ^ https://www.filmtransfer.com%{REQUEST_URI} [R=301,L,NE]
    – egekhter
    Jun 22, 2014 at 20:17
  • You could use wget to spider your web page, for example with link depth of 10 levels. Then you should see from wget output if there are any such URLs being crawled. Then you can look at the saved pages to see exactly where those links are.
    – Tero Kilkanen
    Jun 22, 2014 at 20:44
5

Sitemaps serve to include, not limit the content Google indexes. If you want to exclude some files, use a robots.txt file as mentioned, or setup redirects.

The reason this URL is included is likely that Google found a link pointing to it somewhere else. It could be on your site (which you can fix) or on a third-party site as incoming link. To figure that out, you can use the link syntax link:https://www.w.example.com/services/edison/16mm-to-2k that will tell you what page(s) is linking there.

Improve this answer
3
  • The link:-syntax doesn't work anymore as far as I can see.
    – Ivo van der Veeken
    Jun 23, 2014 at 7:57
  • @IvovanderVeeken you are right, this is weird as I used it recently and didn't care to check before answering. It looks like a bug as it is clearly still documented in the link I provided.
    – Calimo
    Jun 23, 2014 at 8:31
  • I've seen a few discussions about it and I think it hasn't worked (properly) for a while now. I've never used it myself though, so I can't say for sure.
    – Ivo van der Veeken
    Jun 24, 2014 at 9:23
3

do you have a google webmaster tools account? if you create a free account with them and verify that you are the actual site owner then google will allow you to request for removal of a folder or specific urls.

my personal experience is that search engines take the liberty of not following instructions but this step would at least remove your pages from their index.

before you create an account pls change your robots.txt to disallow access to specific areas. as soon as you verify google will check the robots.txt file and update itself.

https://www.google.com/webmasters/tools

Improve this answer
1
  • This approach helps with the symptoms, but it does not get rid of the original problem, which are the incorrect links in the original page.
    – Tero Kilkanen
    Jun 22, 2014 at 7:08
2

Google follows not only links made by other content writers, but it also heuristically interprets your javascript and even tries to "simplify" your URLs to strip them off wrappers, such as /index.php?page=news.php => /news.php! One way would be to ban those mangled URLs in your robots.txt, but that would (1) grow your robots.txt and make it messy, and (2) take away your rank for those links. You must either implement a 301 Moved Permanently or add a Canonical URL tag

 <link rel="canonical" href="http://moz.com/blog" />

pointing to the most basic address of the same content. Beware, most "Chinese" bots won't obey this, so you might consider a server-side conditional that would redirect everything else but Googlebot and user browsers and leave Gogolebot and users with the metadata.

Improve this answer
2

The question focuses a lot on what Google is doing but to me it appears that your fundamental problem not really Google specific at all.

Why do these names, which you clearly don't seem to want people to use, even exist in DNS?

If it is intentional that these names exist and resolve, why are you serving your actual site when people (and Googlebot) connect using these names? If you want to lead people to the site, it would be much better to do so by redirecting (permanent redirect / 301) them to the real site, using its canonical name, instead of leaving them navigating around your site using this incorrect name.

Improve this answer
2
  • It's only Google specific because that's how users find my site, with the wrong url indexed. Since I'm serving up all my pages https, you can induce that it's not intentional otherwise users get a warning because obviously I don't have a SSL certificate for ..mysite.com. The question was about resolving the issue, which has been done by 301 !^www.mysite to www.mysite .
    – egekhter
    Jun 23, 2014 at 23:06
  • There are valid reasons for multiple subdomains, such as load balancing, bandwidth limiting, cross-layer hints (webserver<>iptables) etc. Some DNS providers even give a wildcard record. The webmaster's responsibility here obviously is to redirect everything gracefully and hide the alternate addresses with history.replaceState.
    – Zdenek
    Jun 27, 2014 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.