Is it possible that Google crawler doesn't fetch all the content on the page when it's size is 1.6 MB? How does Google treat big documents?

  • Yes, it's possible Google do not crawl full content if page is bigger. It usually depends on website and quality of article too. If Google consider your website important (gov or edu) then it will crawl full content, no matter how big your page is. But if Google found your content low quality then it will not crawl full. To crawl a page Google has to spend money so they avoid crawling low quality articles to reduce cost.
    – Robert hue
    Nov 27, 2015 at 7:33
  • Note that Bing Webmaster Tools reports pages over 125 KB risks not being fully cached. 1.6 megs of HTML just sounds impractical for a webpage. Nov 27, 2015 at 10:12
  • Google will fetch the entire HTML page and has since the very beginning. However, originally, Google would only index the first 4095/6 bytes of the page. This has been expanded and Google stated at the time that the entire document will be indexed. I am not sure this is entirely true because often programmers put limits in code that they think they will never hit. Only Google can tell us. Google did index very large pages after 2008. My advice is not to create such a large page. Big is one thing; quadruple-e-freakin--what-are-you-thinking-normous is another.
    – closetnoc
    Nov 27, 2015 at 16:22

2 Answers 2


There are in general no page size limits established by Google. Only Google News pages should be not bigger then 256 kB.

But the size of the page makes some limitations by design, which could prevent proper crawling. Such limitations, where the crawler just gets off, are i.e.:

  • exorbitant nesting of the DOM tree
  • too much links in above-the-fold area
  • load time too slow (i.e. because of too many assets included in the head)
  • assets, which Google means they were important, aren't crawlable (blocked by robots, unavailable by server)
  • too many 404 errors coming from page's assets (images etc.)
  • content, loaded dynamically AND which visibility is triggered by user's action

The page's size doesn't matter. What matters, is whether the crawler has any obstacle in its way.

  • For the record, Google will fetch the entire page and always has. However, how much is indexed has varied and may still have a limit. Who know. If there is a limit, it is set extremely high.
    – closetnoc
    Nov 27, 2015 at 16:24

When some one search for something than Google crawler, find out the best information about from Google database.First it crawl all data and then index with asked questions and then display the result.

  • It appears to me that you may be attempting to answer the question. English may not be your first language, which appears to be making it harder. What is the Google database that you mention? How do you find out the best information about something from it? What does any of your answer have to do with with large HTML pages? Nov 27, 2015 at 11:20

Not the answer you're looking for? Browse other questions tagged or ask your own question.