2

Quick note: I couldn't make up my mind which section to post this question on - Webmasters or SO but considering this is more to do with SEO, I decided to post it here so forgive me if I have indeed made this mistake.

Right, so I've been using the Masonry plugin to give some of my element layouts some flexibility (all the reasons why someone would use this plugin) ... but I've now decided to go a little further and attempt the append method to load an external php file when someone clicks a LOAD MORE button (I guess this method is also a "lazy load" kind of function which is half the reason why I wanted to go with this approach).

After hours of testing / researching ways to do this (for some reason many solutions on SO didn't working for me and I also feel the Masonry documentation doesn't really explain well), I ended up coming right with the following code:

$('#towns').masonry({
columnWidth: '.grid-sizer',
itemSelector: '.town',
gutter: '.gutter-sizer',
percentPosition: true
});                     

$('.loadmore').click(function(){
$.get('inc/towns.php', function( data ) {       
    var $moreBlocks = $( data ).filter('.town');
    $('#towns').append( $moreBlocks );      
    $('#towns').masonry( 'appended', $moreBlocks);
});
$('.loadmore').addClass("active");

});

Now my SEO concern is, when I load the contents of towns.php and click on view source, those elements aren't in the source code. I read online that that's because the source code is showing the page when it was first loaded (not dynamically later after loading more content) and someone said online (sorry I can't find the links to these websites) if you right click to inspect element and can see the ajax loaded content, then the content is really 'there' ... thus crawl-able by SE's. However I'm not convinced. I've always said to myself, if it's not in source code, then it's not there to be crawled.

BTW - I assume that ajax loaded content and appended / prepended content is practically the same thing?

Now, just the other day (14th October 2015) Google's article explaining Making AJAX applications crawlable was deprecated. It's an interesting read (obviously) and I still think some of the points that's spoken in that article still applies. Google said:

"What the user sees, what the crawler sees"

I don't know if my tired eyes are causing brain fade but the phrase "What the user sees, what the crawler sees" doesn't seem entirely correct because further down in the article, they mention that if the content cannot be seen in the source code then that content basically cannot be crawled.

So appended content = CAN be seen by user.

However Appended content CANNOT be seen in source code therefore it isn't there to be crawl-able.

So it seems like their policy of "What the user sees, what the crawler sees" isn't quite true or it's just very vague (I could be wrong).

I think what google really meant when they said "What the user sees, what the crawler sees" only applies to someone who's surfing your website with javascript disabled (that kind of 'user' - not being able to see dynamically loaded content since JS is disabled). Even though the "Making AJAX applications crawl-able" has deprecated I think this still applies.

Further down in that article, they mention the term "HTML snapshot". Correct me if I am wrong, but in a nutshell are they basically saying that we should put a hidden link to the file-of-content-that's-ajax-controlled for SE's to follow and crawl? Then later on, they talk about "An agreement between crawler and server" which To be honest, I don't quite understand that part of it all but I assume it doesn't apply to my towns.php "link".

This is the link to the official explanation regarding the "Deprecating our AJAX crawling scheme"

and in that article is a link (sorry can't post more than 2 links with my rep) to another Google article regarding "Understanding web pages better"

After reading all that and With all that been said... What does my situation look like in terms of my masonry-appended content being crawl-able now that NEW algorithms are about? Do I need to take any extra steps to make it SEO friendly like adding a hidden url (or did I get that completely wrong)? I just feel like this topic is a little vague as whole (probably also because I am a noob at javascript). I'm guessing this could be of some help to many others using Masonry.

Appreciate your help in advance.

1

Google can't crawl if

  • content loading is triggered by users action like mouse event, or i.e. onerror of an absent image,
  • javascripts are closed for crawling with robots.txt,
  • javascript loads content after very long timeout

In other cases, like in your, if Google has a read access to javascripts, and javascripts, it can crawl the content. Whether towns.php will be crawled, is another question. It depends whether towns.php and /inc/ are closed by robots.txt against crawling.

The best way to check is to publish an example, fetch it as Google in the search console, and then to look into Google's cache - the crawled/indexed content should be there.

  • Thanks buddy for your effort answering! You've made some logical points here which has definitely brought some light to the subject. As you can see in my code above, I'm doing an onClick function which I guess is a "mouse event"? I'm still developing the website, so I haven't dived into any robot.txt files as yet (I actually still need to research this more thorough) but I don't see any reason why I'd close javascripts from being crawled - but you've brought this to my attention. I don't think the timeout point will be an issue for me (I've got nothing that requires loading ... TBC ... – user3364730 Oct 31 '15 at 20:52
  • ... after a certain time / delay / window.setTimeout function. So I should be fine in that department. When you say "fetch it", do you mean my towns.php file? Interesting test case that though and I will definitely have to give it a go once I go live. My towns.php file is basically like a php include() with div elements and cos I'm still developing I haven't closed the /inc/ dir against crawling (Why would one do that anyway?). – user3364730 Oct 31 '15 at 20:58
  • @user3364730 . correct: if you load content after onClick, G will not crawl it. With fetch i mean both of towns.php and your javascript files, if they are files (not included into towns.php) – Evgeniy Nov 2 '15 at 8:02
0

The "Agreement" Google is mentioning is: You add this line in your html head:

<meta fragment="#">

When the crawler "sees" this tag, it will request the same page appending the ?_escaped_fragment_= parameter. So if the crawler gets to

http://example.com/post/1

and that site includes the meta fragment tag, it will alter the request like

http://example.com/post/1?_escaped_fragment_=

and use the content returned by this request to crawl the site. So you must provide the full page without user interaction based lazy-loading when this parameter exists. In your case, you could just render all of the content on the page (server-side, for example with PHP) when the parameter is present and only include the lazy-loading mechanism when the parameter is not present.

  • Somehow the HTML tag didn't show up in the answer before. – user57401 Nov 2 '15 at 13:37
  • escaped_fragment is not recommended anymore though, even if it still does work. They now recommend #! fragments in the URL instead. – Tony Gustafsson Nov 3 '15 at 7:34
  • You can either use a hashbang url (#!) or add the <meta fragment="#"> tag to make google re-request the page. Your hashbang url (http://example.com/#!/post/1) would be requested like http://example.com/?_escaped_fragment_=post/1. The meta tag is used for client-side routing with the HTML5 pushState API. – user57401 Nov 3 '15 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.