A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed.
When a search engine returns for future crawls of a website, does it systematically go through all of the links in the Sitemap, regardless of whether there have been any changes since the last crawl? Conversely, does the relevant Bot visit the Sitemap, identifies the date under the 'Last Modified' and then only crawls the URL, if the date is after the last time the bot visited/Crawled the URL? The latter, making more sense to me.
I am curious about this as when I submitted the Sitemap, I noticed that in the first few weeks, there was no consistencies in each crawl. Some days, over 1,000 pages would be Crawled while other days, only 10-20 pages would be crawled.
The 'Crawl Rate' has now steadied out now but there are some pages still not having been indexed, despite having been created before other pages which are getting picked up quickly by Search Bots.
I have checked the .htaccess
file as well as ensuring that the pages have not been noindex
ed.