Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a 10MB HTML file. Will Google crawl through the entire file, or does it only look at the first X MB?

share|improve this question
    
I wonder if 10 MB is only html file or with images included. Because Google handle HTML and Images differently . –  user2054689 Apr 23 '13 at 8:25
add comment

4 Answers

The only post I found seems to have data from 2008. So I wouldn't trust that.

You'll have to test this. Go to Webmaster Tools and click Health > Fetch as Google You can trust that what it fetches will be exactly what the crawler fetches.

share|improve this answer
1  
Another way to test it is to go to Project Gutenberg, and search explicitly for something on the bottom of a text, eg google.com/… -- also, if your page is going to be bigger than a novel, that seems like it might end up being a bit confusing to users who end up there when searching for something on the bottom of the page... –  John Mueller Apr 25 '13 at 12:24
add comment

Short Answer: Google will index up to 2.5MB of an HTML file.

Long Answer:

According to Google's Documentation:

All files larger than 30MB will be completely ignored.

They will index up to 2.5MB of an HTML file.

Non-HTML files will be converted to HTML. If the files exceed 4,000,000 bytes (not quite 4MB) they will be completely ignored. Otherwise, the first 2MB will be cached.

share|improve this answer
2  
That documentation is for their search appliance. It doesn't necessarily tell us anything about Googlebot's behavior. –  Su' Apr 22 '13 at 17:08
    
Opps! Sorry about that. –  Mr. Lavalamp Apr 22 '13 at 17:10
add comment

Don't have an answer and not something that shows up on Google.

However, suggestion:

Create html files of varying sizes, ranging from 50KB up through to 10MB and link to all of them. Trigger a crawl of the site and see if any webmaster errors come up. That will give you the upper limit for the max size that Google will be hit before errors show up. Prior to the error showing, after some time, you'll see how much of each file gets indexed.

share|improve this answer
    
There's no reason why a large file would trigger any kind of error or message. –  nathangiesbrecht Apr 23 '13 at 2:41
    
An error being an explicit error(file too big, file not loadable) or an error of exclusion(file doesn't show up in the index of content). –  Wing Tang Wong Apr 23 '13 at 20:51
add comment

As Suggested by Mr. Lavalamp, i went through the Google's Documentation and found the same even after searching the different sites as well. but Talking about Googlebots, they are far more smarter, i think:

Googlebot will not reject a page just because its size is too large, rather it will first crawl Title, URL, Images, headings, Subheadings and Anchor text throughout the page.

This will be probably very less in size and Google can find most of the web page meta data in this document, after that Googlebot can think if more document need to crawl or not, but most of your Web Page resource which are relevant (More) is already Indexed by Google! So one should be careful to Fragment their long Document into pieces of Sections by using headings and subheadings.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.