I've read a few posts about Google AJAX snapshot creation and I'm still asking myself a few questions:

I have a full AJAX web site with really a lot of pages which differ from one to the other with a few content (like an annuary) : what if the html snapshot I serve to Google contains only the relevant part (who cares about header and footer?). I fear in doing that to be considered as a cloaker but in my opinion I do help Google indexing what the user will be looking for!

On a more technical way, I clearly do not understand why people would have to use specific softwares (htmlunit ...) to create a HTML snapshot. It's so simple to serve Googlebot the EXACT page he's requesting with something like this:

# REMOVE "?_escaped_fragment_=" FROM URI
$Uri = str_replace("?_escaped_fragment_=", "", $Uri);
$Html = file_get_contents($BaseURL."".$Uri);
echo $Html;

Google is most worried about cloaking when the crawler sees something that the user doesn't see. When the server pushes keywords to Googlebot that aren't on the page it leads to the page being indexed for the wrong content. Users get frustrated when they visit a page and it isn't about the keywords for which they searched.

The converse of that is much less of a problem. Google is not likely to penalize for withholding content from Googlebot when it is shown to users. There are many cases where that is happening already without any risk of penalty:

  • Text in images
  • Ads
  • Iframes blocked by robots.txt
  • Text loaded via AJAX that hasn't been made crawlable for Googlebot

I wouldn't worry about omitting the header and footer in the HTML snapshots. In fact, it might even make development easier not to because then the JavaScript on your site could use the same escaped fragment URLs to fetch the content that Googlebot uses.

On a side note, Google only recommends using htmlunit when pages are generated mostly by client side JavaScript. In cases where the JavaScript fetches hunks of HTML from the server to put into the main content of the page, Google recommends your approach. (Option 2 in the link, where htmlunit is option 3.)

| improve this answer | |
  • Thank you @Stephen for making my worries go away ! I didn't understand snapshot mechanism at the time I was posting my question : I ended by using php-phantomjs to create my snapshot as my content is mostly generated by javascript ! – hugsbrugs Oct 29 '14 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.