I guess there is a news that Google bots have the capability to understand our javascript code. It means this is possible to fully crawl a webpage which has lazy loading feature enabled. I am using Apache Nutch to crawl websites but I don't think it has the capability to fetch the URLs being injected in HTML page by javascript when the page is scrolled down. I see a lot of websites doing lazy loading for performance issue. So Can somebody please explain me how can i crawl the data which comes in HTML page on lazy load. (On scrolling the page down).

  • Google can understand some JavaScript.
    – MrWhite
    Aug 31 '12 at 9:46

Googlebot understands links "hidden" with JS. But the LazyLoad just makes the browser render the content after the initial page load. The HTML is still there. So your bot should have no issue scanning it, since JS is client-side.

If you are having trouble with heavily-JS'd links, check them with parsechecker to see how the filters handle them, and adjust them accordingly.

  • 1
    "The HTML is still there" - But I thought the point of the lazy load (as you scroll the page) is that the HTML isn't there? AJAX requests are fired off to fetch new content as you scroll. This is different to a simple show/hide script where the HTML is preloaded but initially hidden.
    – MrWhite
    Aug 31 '12 at 9:44
  • Agree with w3d. After using lazyloading, the initial number of Get requests reduces therefore the page loads faster so if the HTML is there then I don't think the performance is gonna be better. May be I have understood it wrong way. So please elaborate. Thanks anyway!
    – blunderboy
    Aug 31 '12 at 10:20

You can use some javascript parser at your server-side code(crawler) and parse the javascripts to fetch all the Ajax requests and then also crawl them. One of them is google-caja.

Try it. May be it will solve your purpose.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.