Is my ajax content crawlable with the Masonry Append method?

Question

Quick note: I couldn't make up my mind which section to post this question on - Webmasters or SO but considering this is more to do with SEO, I decided to post it here so forgive me if I have indeed made this mistake.

Right, so I've been using the Masonry plugin to give some of my element layouts some flexibility (all the reasons why someone would use this plugin) ... but I've now decided to go a little further and attempt the append method to load an external php file when someone clicks a LOAD MORE button (I guess this method is also a "lazy load" kind of function which is half the reason why I wanted to go with this approach).

After hours of testing / researching ways to do this (for some reason many solutions on SO didn't working for me and I also feel the Masonry documentation doesn't really explain well), I ended up coming right with the following code:

$('#towns').masonry({
columnWidth: '.grid-sizer',
itemSelector: '.town',
gutter: '.gutter-sizer',
percentPosition: true
});                     

$('.loadmore').click(function(){
$.get('inc/towns.php', function( data ) {       
    var $moreBlocks = $( data ).filter('.town');
    $('#towns').append( $moreBlocks );      
    $('#towns').masonry( 'appended', $moreBlocks);
});
$('.loadmore').addClass("active");

});

Now my SEO concern is, when I load the contents of towns.php and click on view source, those elements aren't in the source code. I read online that that's because the source code is showing the page when it was first loaded (not dynamically later after loading more content) and someone said online (sorry I can't find the links to these websites) if you right click to inspect element and can see the ajax loaded content, then the content is really 'there' ... thus crawl-able by SE's. However I'm not convinced. I've always said to myself, if it's not in source code, then it's not there to be crawled.

BTW - I assume that ajax loaded content and appended / prepended content is practically the same thing?

Now, just the other day (14th October 2015) Google's article explaining Making AJAX applications crawlable was deprecated. It's an interesting read (obviously) and I still think some of the points that's spoken in that article still applies. Google said:

"What the user sees, what the crawler sees"

I don't know if my tired eyes are causing brain fade but the phrase "What the user sees, what the crawler sees" doesn't seem entirely correct because further down in the article, they mention that if the content cannot be seen in the source code then that content basically cannot be crawled.

So appended content = CAN be seen by user.

However Appended content CANNOT be seen in source code therefore it isn't there to be crawl-able.

So it seems like their policy of "What the user sees, what the crawler sees" isn't quite true or it's just very vague (I could be wrong).

I think what google really meant when they said "What the user sees, what the crawler sees" only applies to someone who's surfing your website with javascript disabled (that kind of 'user' - not being able to see dynamically loaded content since JS is disabled). Even though the "Making AJAX applications crawl-able" has deprecated I think this still applies.

Further down in that article, they mention the term "HTML snapshot". Correct me if I am wrong, but in a nutshell are they basically saying that we should put a hidden link to the file-of-content-that's-ajax-controlled for SE's to follow and crawl? Then later on, they talk about "An agreement between crawler and server" which To be honest, I don't quite understand that part of it all but I assume it doesn't apply to my towns.php "link".

This is the link to the official explanation regarding the "Deprecating our AJAX crawling scheme"

and in that article is a link (sorry can't post more than 2 links with my rep) to another Google article regarding "Understanding web pages better"

After reading all that and With all that been said... What does my situation look like in terms of my masonry-appended content being crawl-able now that NEW algorithms are about? Do I need to take any extra steps to make it SEO friendly like adding a hidden url (or did I get that completely wrong)? I just feel like this topic is a little vague as whole (probably also because I am a noob at javascript). I'm guessing this could be of some help to many others using Masonry.

Appreciate your help in advance.

Answer 1

Google can't crawl if

• content loading is triggered by users action like mouse event, or i.e. onerror of an absent image,
• javascripts are closed for crawling with robots.txt,
• javascript loads content after very long timeout

In other cases, like in your, if Google has a read access to javascripts, and javascripts, it can crawl the content. Whether towns.php will be crawled, is another question. It depends whether towns.php and /inc/ are closed by robots.txt against crawling.

The best way to check is to publish an example, fetch it as Google in the search console, and then to look into Google's cache - the crawled/indexed content should be there.

Answer 2

The "Agreement" Google is mentioning is: You add this line in your html head:

<meta fragment="#">

When the crawler "sees" this tag, it will request the same page appending the ?_escaped_fragment_= parameter. So if the crawler gets to

http://example.com/post/1

and that site includes the meta fragment tag, it will alter the request like

http://example.com/post/1?_escaped_fragment_=

and use the content returned by this request to crawl the site. So you must provide the full page without user interaction based lazy-loading when this parameter exists. In your case, you could just render all of the content on the page (server-side, for example with PHP) when the parameter is present and only include the lazy-loading mechanism when the parameter is not present.