Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 2 down vote favorite

I guess there is a news that Google bots have the capability to understand our javascript code. It means this is possible to fully crawl a webpage which has lazy loading feature enabled. I am using Apache Nutch to crawl websites but I don't think it has the capability to fetch the URLs being injected in HTML page by javascript when the page is scrolled down. I see a lot of websites doing lazy loading for performance issue. So Can somebody please explain me how can i crawl the data which comes in HTML page on lazy load. (On scrolling the page down).

share|improve this question
Google can understand some JavaScript. – w3d Aug 31 '12 at 9:46

2 Answers

up vote 2 down vote

Googlebot understands links "hidden" with JS. But the LazyLoad just makes the browser render the content after the initial page load. The HTML is still there. So your bot should have no issue scanning it, since JS is client-side.

If you are having trouble with heavily-JS'd links, check them with parsechecker to see how the filters handle them, and adjust them accordingly.

share|improve this answer
1  
"The HTML is still there" - But I thought the point of the lazy load (as you scroll the page) is that the HTML isn't there? AJAX requests are fired off to fetch new content as you scroll. This is different to a simple show/hide script where the HTML is preloaded but initially hidden. – w3d Aug 31 '12 at 9:44
Agree with w3d. After using lazyloading, the initial number of Get requests reduces therefore the page loads faster so if the HTML is there then I don't think the performance is gonna be better. May be I have understood it wrong way. So please elaborate. Thanks anyway! – blunderboy Aug 31 '12 at 10:20
up vote 1 down vote

You can use some javascript parser at your server-side code(crawler) and parse the javascripts to fetch all the Ajax requests and then also crawl them. One of them is google-caja.

Try it. May be it will solve your purpose.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.