I have a 10MB HTML file. Will Google crawl through the entire file, or does it only look at the first X MB?
The only post I found seems to have data from 2008. So I wouldn't trust that.
You'll have to test this. Go to Webmaster Tools and click Health > Fetch as Google You can trust that what it fetches will be exactly what the crawler fetches.
Short Answer: Google will index up to 2.5MB of an HTML file.
According to Google's Documentation:
All files larger than 30MB will be completely ignored.
They will index up to 2.5MB of an HTML file.
Non-HTML files will be converted to HTML. If the files exceed 4,000,000 bytes they will be completely ignored. Otherwise, the first 2MB will be cached.
As Suggested by Mr. Lavalamp, i went through the Google's Documentation and found the same even after searching the different sites as well. but Talking about Googlebots, they are far more smarter, i think:
This will be probably very less in size and Google can find most of the web page meta data in this document, after that Googlebot can think if more document need to crawl or not, but most of your Web Page resource which are relevant (More) is already Indexed by Google! So one should be careful to Fragment their long Document into pieces of Sections by using headings and subheadings.
Google only indexes the first nine chapters of Pride and Prejudice on Project Gutenberg's website. Chapter 10 is right about 100k (1/10th of a megabyte).
If you have very large amounts of text in a document, Google won't index all of it. In fact it appears that Google only indexes approximately the first 100k.
I believe that Googlebot is willing to download more data than that. It just may not index the text in the document after a certain point.
It is also not clear from this experiment whether Google counts markup which is non-visible to the user towards that 100k. My guess is not. Many pages have far more than 100k of markup with text that Google would like to index near the bottom.
Don't have an answer and not something that shows up on Google.
Create html files of varying sizes, ranging from 50KB up through to 10MB and link to all of them. Trigger a crawl of the site and see if any webmaster errors come up. That will give you the upper limit for the max size that Google will be hit before errors show up. Prior to the error showing, after some time, you'll see how much of each file gets indexed.