How often do search bots typically visit a website?

Question

A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed.

When a search engine returns for future crawls of a website, does it systematically go through all of the links in the Sitemap, regardless of whether there have been any changes since the last crawl? Conversely, does the relevant Bot visit the Sitemap, identifies the date under the 'Last Modified' and then only crawls the URL, if the date is after the last time the bot visited/Crawled the URL? The latter, making more sense to me.

I am curious about this as when I submitted the Sitemap, I noticed that in the first few weeks, there was no consistencies in each crawl. Some days, over 1,000 pages would be Crawled while other days, only 10-20 pages would be crawled.

The 'Crawl Rate' has now steadied out now but there are some pages still not having been indexed, despite having been created before other pages which are getting picked up quickly by Search Bots.

I have checked the .htaccess file as well as ensuring that the pages have not been noindexed.

Answer 1

Search bots typically never stop visiting a website. Googlebot will typically download some pages every day. It may even download nearly as many pages per day as real users do.

Search bots almost never crawl the entire site in one sitting. Rather they re-crawl pages each on their own schedule. They schedule page re-crawls based on how popular a page is and how often the search bot observes the page changing.

Google has said that they pretty much ignore the lastmod and changefreq fields in sitemaps because so many webmasters don't keep them up to date reliably. In fact, Google doesn't use your sitemap for very much at all. The main benefit of having a sitemap is seeing extra stats in Google Search Console. Sitemaps don't typically get Googlebot to crawl pages it wouldn't otherwise crawl and they don't help with rankings. See The Sitemap Paradox.

Most sites end up with some pages that don't get indexed. It isn't usually because Google hasn't gotten around to crawling. Rather, Google just chooses not to index some pages. See Why aren't search engines indexing my content? Pages that Google chooses not to index are typically one of:

• Duplicate
• Not much content
• Poor quality
• Not targeting topics for which users search
• Low reputation (PageRank)

Answer 2

What I have personally noticed across my sites is that when Google crawls one of my sites for many thousands of pages one day, but then it stops crawling it over the whole week, that it decided:

1. My site was interesting and maybe worth crawling
2. That it turned out it wasn't interesting enough to Google to come back everyday.
3. But it's interesting enough for Google to come back in a week or two to see if it got better.

When my sites start getting crawled by Google everyday is when I've found that Google starts to send them a lot of traffic. It seems that when my sites get crawled massively for one day or one week, but not at all the next day or week, it's because Google is interested but it doesn't want to send me much traffic yet. When I've improved the sites with infrequent crawl rates, Google has often then began crawling the site every day and sending it a lot more traffic.

As a result, I think that this is an indication that you really might want to figure out what you can do to take the site to the next level of improvement. Because Google is interested in you but may think you need to shine brighter.

This is just my thought and experience from similar crawl rate issues I've experienced and how I've interpreted them.