Googlebot crawls a certain number of pages daily, depending on the size of the site. However, how long does it take for a particular page. In other words, how many seconds or minutes does it spend on the current scanned page before it goes to the next page it will scan. Is there a set time for this? Or can we think of it as spending 10 seconds while scanning one page and 50 seconds depending on the page's resources? Does it have an average?
See this related question: For how long does the google bot wait after document ready before taking a snapshot of the webpage?– dan ♦Jul 26, 2022 at 5:15
Google aggressively caches resources so it can load pages faster than a regular browser. It also honours robots.txt, so it has to load fewer resources.
Tests I made show Googlebot waits about 5.5 seconds to render a page before taking its snapshot.
I also believe it has a very tall viewport to help it pick up content on pages that dynamically load content like with infinite scroll.
So, do the cache times we give to the files have an effect, or is Google doing the scanning from the cache itself? Jul 27, 2022 at 9:40
Sometimes it takes several passes, especially for deleted pages. Google wants to make sure that the missing page has actually been deleted and is not just missing due to a server or site failure.
At the moment, Googlebot only scans the first 15 megabytes of the page and leaves it. So I think you should not think about how much time the bot passes a page, and you need to watch as he scans and how many pages are indexed after the bypass, these parameters is better to watch :)
The number of pages that Googlebot crawls daily depends on more than the size of the site. It depends on:
- The speed of your server. If you have a faster server Googlebot is willing to crawl more.
- The size of your pages. Google will crawl more smaller pages or fewer large pages.
- The reputation of each page on the site (in PageRank.) The higher the PageRank of a page, the more often it gets re-crawled.
- How often the content is updated. Frequently updated content will get re-crawled more often.
There are also multiple Googlebot crawl modes:
- Fresh crawling -- Googlebot finds new pages on a site and crawls lots of them (up to several thousand.)
- Return crawling -- Googlebot returns to check for updates on pages based on how popular the page is and how frequently it is updated. This is the vast majority of crawling that Googlebot does.
- Old URL crawling -- Googlebot batch checks URLs that have no reputation and haven't had content in some time (often years.) Googlebot typically checks hundreds in a row, and usually in length order starting with the shortest.
Googlebot is a huge distributed crawling system. It runs on hundreds of thousands of computers across multiple data centers. It is a mistake to think that Googlebot crawls your site page by page. Rather:
- Googlebot may crawl several pages from your site simultaneously.
- Googlebot may crawl your important pages many times before some pages on your site get crawled at all.
- Googlebot never stops crawling your site or starts over. Once your site is up, the crawling will be continuous. Each page gets revisited on its own schedule.
The amount of time that Google spends crawling a page is mostly going to be governed by the speed of your server and the size of your pages. Googlebot crawls each page as quickly as your site can serve it. If your server is fast enough to serve Googlebot everything it asks for, the crawl rate of your site will be limited by the size of your site and the PageRank of your pages.
Because Googlebot's crawling is so customized to your site, it doesn't make a lot of sense to talk about overall averages. The amount of crawling that Googlebot does will increase dramatically as you grow your site.