Let's say that googlebot is scraping https://example.com/page.

  • example.com has a robots.txt file that disallows /endpoint-for-lazy-loaded-content, but allows /page
  • /page lazy loads content using /endpoint-for-lazy-loaded-content (via fetch)

Does googlebot see the lazy loaded content?


Googlebot will see the lazy loaded content after it has been loaded onto the correct page. The page will be taken as a whole once any lazy loaded content or dynamic content has been loaded as Google will evaluate the page as the end user would see it. All that having the exclusion in your robots.txt file will do is prevent Google from listing the endpoint for the lazy loaded content on its own.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.