5

I'm a developer for an AJAX webapp and, following Google's specification for crawlable webapps, we use #! to indicate that it's a AJAX application such that we can serve a static page to Google instead. This all works perfectly fine: Google fetches the ?_escaped_fragment_ URLs instead.

However, in the logs we found that even though we follow this specification Google also fetches the original AJAX pages, and in the process it generates script errors.

Is it expected behaviour that Google visits the AJAX URLs, even though it knows we have specially prepared pages for it? I can imagine that Google does this to train it's AJAX crawler, but I cannot find any information about it.

Additionally, does this have any influence on our ranking?

  • What is a script error? Even if Googlebot does run JavaScript, how would you know that it encounters an error? – Stephen Ostermiller Dec 4 '14 at 15:16
  • Can you give examples of the URLS that users see, that you expect Googlebot to crawl, and that you don't expect Googlebot to crawl? – Stephen Ostermiller Dec 4 '14 at 15:16
  • 1
    @StephenOstermiller - Script error is a general JS error. All errors that are triggered in the front-end are send to our servers so we can see them and fix them. Therefore I can see with our logging application (logstash) that the Googlebot encounters these errors. Example user url: magnet.me/public/#!/organization/magnet.me/timeline. Example Googlebot url: magnet.me/public/?_escaped_fragment=/organization/magnet.me/…. Note that Google does in fact crawl the later url, but it shouldn't crawl the former as well. – Tiddo Dec 4 '14 at 16:22
  • Any part of the URL following the # is never sent to the server (client side only), so I would assume that you are saying that Googlebot is actually requesting magnet.me/public/ from your server? – Stephen Ostermiller Dec 4 '14 at 16:24
  • 1
    I see it in my front-end logs. – Tiddo Dec 5 '14 at 11:08
3

To prevent misuse the inner workings of Googlebot are kept secret so there is no real "expected behaviour". While searching the net I stumbled upon the following:

... search engines understand URL parameters but often ignore fragments.

This implies URL fragments might get crawled.

On another page I found a second hint. I always thought Googlebot didn't do Javascript but apparently I'm wrong.

... we decided to try to understand pages by executing JavaScript.

So what this learns us is Googlebot might fetch the AJAX pages and as your front-end logs show it does. The reason might be it is checking if your page is trustworthy by comparing what it gets back from the AJAX-call versus the html-snapshot (via _escaped_fragment_)? That's just a guess.

Sources:

https://support.google.com/webmasters/answer/81766?hl=en&ref_topic=6003039 http://googlewebmastercentral.blogspot.be/2014/05/understanding-web-pages-better.html

  • That sounds like a very plausible explanation. I upvoted the answer, but I can't (yet) accept it since it's still a guess. – Tiddo Dec 13 '14 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.