Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the upper threshold for crawl speed?

share|improve this question
    
In general, 5 million quality pages don't just suddenly appear on the web. Have they been moved from somewhere? If they are low quality pages, Google will not be interested in crawling and indexing. –  Brendon Sep 9 '13 at 12:55
    
When you say that 10,000 pages per day are "indexed" how do you know? Is this the number that is crawled, or is this how much the number of indexed pages is increasing in webmaster tools each day? –  Stephen Ostermiller Sep 9 '13 at 14:01
    
It is shown in the crawl graph. –  stiv Sep 10 '13 at 8:54

4 Answers 4

The maximum speed of indexing is 10 times per second. This is the speed of StackOverflow indexing by Google (read this).

share|improve this answer
1  
That isn't an absolute upper bound. Just what is available to a site with stackoverflow's pagerank. I've worked with higher PR sites that have higher crawl rates available. –  Stephen Ostermiller Sep 9 '13 at 14:12

If they're crawling your pages and they're not being found in the search results then the crawl rate is not an issue. This sounds like your website is full of low quality content that Google does not want in its index. Is this original content? Is it quality content? Google not listing your pages indicates that it is not.

share|improve this answer

Google's crawl rate is a function of:

  • Pagerank -- the more reputation and inbound links your site has, the more it will be crawled. Within your site the most prominent pages (like the home page) will get crawled more often because they have higher pagerank.
  • How often your pages change -- pages that change frequently will get re-crawled more often that pages that don't.

In addition, Googlebot has several different crawl modes.

  • Re-crawl mode -- it will come back and visit pages that it has visited before.
  • Fresh crawl mode -- it will crawl lots of new pages in a new section of a site. The higher the pagerank of the site, the more pages get crawled.
  • Stale pages mode -- Googlebot finds a box of old links in the basement and plows through them just for "fun". These pages are often all pages that no longer exist and are redirected to other pages. They often have no pagerank and are crawled in URL-length order.

The upshot of this is that the best way to get your site crawled faster is to get inbound links and increase the pagerank.

share|improve this answer
up vote 0 down vote accepted

I've found out that it is possible to achieve 2 pages/second crawl speed by improving server responce time. Each page should responce as fast as possible. This may require garbage collector tuning, db tuning and code tuning. If average responce time is better then 50ms per second, then google would index at 2pages/sec, this is experimental fact.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.