Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a site with many pages (30MM+ indexed) and have been getting a extremely high number of URLs on your site warning message for a long time. I'm thinking of reducing the numbers of URLs Google sees especially between similar pages, specifically on pages that have a different language template but have the same content.

Current for every page I have this in my <head>:

  <link rel="alternate" hreflang="en x-default" href="http://mydomain.com/dir/page" />
  <link rel="alternate" hreflang="ja" href="http://ja.mydomain.com/dir/page" />
  <link rel="alternate" hreflang="ko" href="http://ko.mydomain.com/dir/page" />
  <link rel="alternate" hreflang="th" href="http://th.mydomain.com/dir/page" />
  <link rel="alternate" hreflang="es" href="http://es.mydomain.com/dir/page" />
  <link rel="alternate" hreflang="pt" href="http://pt.mydomain.com/dir/page" />

Will adding a rel=nofollow or a robots.txt disallow help Google focus on the more important pages he has to crawl instead of crawling a different template of the same page?

share|improve this question
1  
It's only a warning and nothing that you need to take action on. How many hreflang templates do you have in the <head> of each page? –  zigojacko Jun 18 at 8:37
    
Exactly like in my code snippet - 6. I know it doesn't necessarily mean I should take action, but it feels he doesn't really need to invest his time crawling these template pages, doesn't it? –  Noam Jun 18 at 9:49
    
They aren't going to be showing that warning because of 6 hreflang links in the <head> - that's nothing. How many other URL's are on the rest of your pages, what type of website is it? –  zigojacko Jun 18 at 9:53
    
@zigojacko for every page I have 6 language versions, so it's 6x not just 6. For example: twtrland.com/profile/zigojacko and ko.twtrland.com/profile/zigojacko ... (you can also get the feeling of what type of website it is) –  Noam Jun 18 at 10:00
    
Yes I get that but 6 links on 1 page multiplied by 30 million is not an issue in terms of links per page. It is only because you have such a large site that you see that warning. I wouldn't even expect Google WMT to be referring to the URL's in the `<head> of your pages with this warning either tbh. (Signing up to your website though as it looks useful) :) –  zigojacko Jun 18 at 10:05

1 Answer 1

Adding nofollow generally does not prevent Googlebot from discovering content. See: Does a "nofollow" attribute on a link prevent URL discovery by search engines?

If the nofollow link is the only link to that page, then Google won't index it. In your case, I highly doubt that your alternate language links are the only links into your other sites. Once Google finds a dofollow link to any page on the alternate language subdomain, they would be able to discover the rest with dofollow links there.

The "extremely high number of URLs on your site" warning from webmaster tools is nothing to worry about if your site is supposed to have that many pages. Google is more worried about cases such as:

  • A site with session ids in the URL that make the site appear to have a different URL every time Google crawls it
  • Sites that can auto-generate a page for any URL

Every site with millions of pages gets this message in Webmaster Tools. There is no problem with a site having that many pages if it has the content to back it up. I've worked with several such sites myself. None of them have had trouble with rankings because of it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.