1

Does anyone know if Googlebot will follow and index web pages that are only listed in the head of a HTML page with a link-hreflang element?

<link hreflang="fr" rel="alternate" href="http://example.io/fr/page/webpage" />
  • Why wouldn't it?? – closetnoc Dec 13 '15 at 23:51
  • I can't find any info on this. Since its not a hyper-link, I don't know if they follow it. – Kohjah Breese Dec 14 '15 at 0:00
  • 1
    It is Googles business to index your site content and serve your content within it's SERPs to the tune of 80 billion dollars a year. It would be foolish if Google did not. Google follows citations (mentions without a link) so I am extremely sure Google will follow hreflang links to your other language sites. – closetnoc Dec 14 '15 at 0:10
  • OK. I will post the answer here when I find out. – Kohjah Breese Dec 14 '15 at 0:21
1

As closetnoc states, Google's main business is to seek and crawl links in hopes that valuable pages are returned and if they are, they are then indexed.

Therefore, unless you explicitly point out to google not to index a certain page, it will crawl everything. Here's a few ways to stop google from indexing pages you don't want indexed.

Let's assume the page you don't want indexed in google is dontindexme.php in document root.

The nicest method to make a file non-indexable to google, yet a method that helps hackers learn more about what files are public on your site is to create robots.txt (a file also publicly accessible) in document root with the following contents:

User-agent: googlebot
Disallow: /dontindexme.php

Another method is to modify the apache configuration to add HTTP headers. Alternatively, you can modify the script to include the HTTP headers as well. The HTTP header you want outputted when the file requested is:

X-Robots-Tag: noindex

While this method doesn't prevent google from beginning the page fetch, it will instruct google not to advertise the page in its search results.

Now if you really want to stop google from crawling a particular URL, you can configure your script or apache so that the result returned to google isn't of status 200 OK. This means you can return a Not found page. But if you choose this route, give a 410 status to indicate the page is gone to google and that google will never index it again.

Another thing that google wouldn't crawl are links only accessible via the POST request method. This means accessing a URL as a result of filling out a form. This is because google does not want to get too interactive on your website (such as logging in or shopping).

Unless you take one of the above actions on a page that isn't meant to be indexed, just assume google will crawl and index the page.

0

Yes Google Crawler will index the hreflang pages and count them as unique page for your website.

0

I added the href-lang attributes to the site and seven days later none of the new language pages have been indexed. Google has crawled 100,000s of pages since then.

  • It can take Google a week or two to index pages sometimes after it crawls them. Is Googlebot actually hitting the URLs specified in the hreflang? – Stephen Ostermiller Dec 19 '15 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.