When Googlebot sees a link, will it click it or navigate to it?

Question

My site uses pushState and JSON data to display content. So, for example, this might appear on my page:

<a href="/some/page">some page</a>

The JavaScript then prevents the default action (following the link), and instead renders a view (using a different api, such as /getjson?some_page).

$('[href]').click(function(){
    history.pushState(...);
    handleURL(...);
});

Assume my server will respond to requests at /some/page with a pre-rendered version.

My questions are:

1. will Googlebot receive the prerendered version, or allow JavaScript to instead invoke pushState, etc.
2. if it doesn't make the direct request, will it wait for AJAX content to be loaded?
3. does Googlebot implement pushState, so it will show the proper URL in search results?

Answer 1

While Googlebot understands and parses some JavaScript, it doesn't run like your graphical web browser does. It makes a request to a URL (e.g. your home page) and the server sends back HTML.

If your site returns no true HTML (e.g. little more than just a script tag) then you may have problems with Google or other search engines crawling your pages.

However according to this video, Googlebot does understand pushState. So it appears that it will use the AJAX method rather than making separate requests for pages. That means it would show the correct URL in search results.

That's not to say you should forget about that - making a separate request should always return the proper page as it would be rendered. For one thing, Google understands "pages" so after it's crawled a page, even if it used pushState initially it would almost certainly make requests for separate pages in future, instead of always "navigating" from the home page.

Also, it's faster for users to receive HTML than receive a script, parse it, then fetch the HTML using JavaScript. And don't forget there are other search engines and browsers which may not handle pushState or JavaScript at all.

Answer 2

Google's Matt Cutts answered your question in a video:

A correctly implemented site that uses pushstate typically doesn't need any extra support for us to be able to crawl it.

This is because Googlebot will always follow the links that are in your page whether or not there is JavaScript that intercepts them. So it will get the pre-rendered versions of the pages and direct users to those URLs.

Googlebot may also try to parse some of your JavaScript and fetch URLs that it finds there. So you may see Googlebot also fetching some of the page fragments that you serve as push state.