Skip to main content
Webmasters

Return to Answer

Need to point out that this was a 2015 experiment and repeating it in 2017 gives different results
Source Link
edit approved Mar 20, 2017 at 15:36
Community Bot
edit approved Mar 20, 2017 at 15:36
Community Bot
  • 1

In December 2015, Google only indexesindexed the first nine chapters of Pride and Prejudice on Project Gutenberg's website. Chapter 10 is right about 100k (1/10th of a megabyte).

Chapter 9

Chapter 10

Repeating this search in March 2017 showed the result from Chapter 10 however.

If you have very large amounts of text in a document, Google won't index all of it. In fact it appears that Google only indexesindexed approximately the first 100k in 2015, although this has since been increased at least for some sites.

I believe that Googlebot is willing to download more data than that. It just may not index the text in the document after a certain point.

It is also not clear from this experiment whether Google counts markup which is non-visible to the user towards that 100k. My guess is not. Many pages have far more than 100k of markup with text that Google would like to index near the bottom.

Google only indexes the first nine chapters of Pride and Prejudice on Project Gutenberg's website. Chapter 10 is right about 100k (1/10th of a megabyte).

Chapter 9

Chapter 10

If you have very large amounts of text in a document, Google won't index all of it. In fact it appears that Google only indexes approximately the first 100k.

I believe that Googlebot is willing to download more data than that. It just may not index the text in the document after a certain point.

It is also not clear from this experiment whether Google counts markup which is non-visible to the user towards that 100k. My guess is not. Many pages have far more than 100k of markup with text that Google would like to index near the bottom.

In December 2015, Google only indexed the first nine chapters of Pride and Prejudice on Project Gutenberg's website. Chapter 10 is right about 100k (1/10th of a megabyte).

Chapter 9

Chapter 10

Repeating this search in March 2017 showed the result from Chapter 10 however.

If you have very large amounts of text in a document, Google won't index all of it. In fact it appears that Google only indexed approximately the first 100k in 2015, although this has since been increased at least for some sites.

I believe that Googlebot is willing to download more data than that. It just may not index the text in the document after a certain point.

It is also not clear from this experiment whether Google counts markup which is non-visible to the user towards that 100k. My guess is not. Many pages have far more than 100k of markup with text that Google would like to index near the bottom.

Source Link
Stephen Ostermiller ♦
Stephen Ostermiller ♦
  • 99.4k
  • 18
  • 141
  • 364

Google only indexes the first nine chapters of Pride and Prejudice on Project Gutenberg's website. Chapter 10 is right about 100k (1/10th of a megabyte).

Chapter 9

Chapter 10

If you have very large amounts of text in a document, Google won't index all of it. In fact it appears that Google only indexes approximately the first 100k.

I believe that Googlebot is willing to download more data than that. It just may not index the text in the document after a certain point.

It is also not clear from this experiment whether Google counts markup which is non-visible to the user towards that 100k. My guess is not. Many pages have far more than 100k of markup with text that Google would like to index near the bottom.