1

A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed.

When a search engine returns for future crawls of a website, does it systematically go through all of the links in the Sitemap, regardless of whether there have been any changes since the last crawl? Conversely, does the relevant Bot visit the Sitemap, identifies the date under the 'Last Modified' and then only crawls the URL, if the date is after the last time the bot visited/Crawled the URL? The latter, making more sense to me.

I am curious about this as when I submitted the Sitemap, I noticed that in the first few weeks, there was no consistencies in each crawl. Some days, over 1,000 pages would be Crawled while other days, only 10-20 pages would be crawled.

The 'Crawl Rate' has now steadied out now but there are some pages still not having been indexed, despite having been created before other pages which are getting picked up quickly by Search Bots.

I have checked the .htaccess file as well as ensuring that the pages have not been noindexed.

Improve this question
2
  • 1
    You are working off of some serious misconceptions. Sitemaps do almost nothing unless you have content that is not linked or behind a paywall or login. Other than that, sitemaps do not have anything to do with a search engine crawling your site or even necessary.
    – closetnoc
    Mar 16, 2018 at 17:18
  • @closetnoc Thanks for your Comment. After using Google Webmaster's Tools, I have a greater breadth of understanding now. Just thought a Sitemap submission may have had some kind of impact, since within 24 hours of submission, the pages started to appear in search results. Thanks for the Comment, nonetheless. :-)
    – Craig
    Apr 3, 2018 at 21:24

2 Answers 2

Reset to default
4

Search bots typically never stop visiting a website. Googlebot will typically download some pages every day. It may even download nearly as many pages per day as real users do.

Search bots almost never crawl the entire site in one sitting. Rather they re-crawl pages each on their own schedule. They schedule page re-crawls based on how popular a page is and how often the search bot observes the page changing.

Google has said that they pretty much ignore the lastmod and changefreq fields in sitemaps because so many webmasters don't keep them up to date reliably. In fact, Google doesn't use your sitemap for very much at all. The main benefit of having a sitemap is seeing extra stats in Google Search Console. Sitemaps don't typically get Googlebot to crawl pages it wouldn't otherwise crawl and they don't help with rankings. See The Sitemap Paradox.

Most sites end up with some pages that don't get indexed. It isn't usually because Google hasn't gotten around to crawling. Rather, Google just chooses not to index some pages. See Why aren't search engines indexing my content? Pages that Google chooses not to index are typically one of:

  • Duplicate
  • Not much content
  • Poor quality
  • Not targeting topics for which users search
  • Low reputation (PageRank)
Improve this answer
7
  • I up voted quite a while ago, however, the line "Not targeting keywords for which users typically search" continues to bother me. Forget my opinion about the keyword chase for a moment. The line, in my opinion, would be better served as Not a popular search topic. Imagine I have a site about knitting and crocheting sausage cozies. Would targeting any keyword help? If a page sucks, then the page sucks - keywords be damned. There is nothing you can do to compensate for lousy or unpopular content. It seems to suggest that keywords fixes lousy content. Was I clear or confusing? Cheers mate!!
    – closetnoc
    Mar 16, 2018 at 22:49
  • BTW- You are technically correct. It just makes me uneasy in that it can be read that keywords is the answer when the content is likely no good to begin with. Otherwise, the terms used in good content would not be an issue. I do not recall good content that was lacking popular search terms. And I cannot remember any lousy content that could be fixed with a smattering of popular search terms. Clearer?
    – closetnoc
    Mar 16, 2018 at 22:54
  • I updated the wedding at your suggestion @closetnoc
    – Stephen Ostermiller ♦
    Mar 16, 2018 at 23:36
  • Cheers!! I hesitated to make the suggestion and thought about it for quite a while. I was not sure I was explaining myself well. I thought it was important not to give the wrong impression especially in light of how twitchy people can be about keywords like it this the only important aspect of content. BTW- Have you seen my sausage cozy site? Number one on Google for the shut in market!
    – closetnoc
    Mar 17, 2018 at 0:10
  • Some of us have been doing keywords so long it is hard to change. I'm not nearly as far down the anti-keyword path as you are, but even I recognize that Google has gone off keywords to a large extent. As a user of Google search it frustrates me to no end when Google returns results that don't actually match my query. It isn't the synonyms that I mind so much as Google blatantly ignoring words in my query and returning pages that are for more general than my very specific search. I've actually switched to using "verbatim" mode all the time. I must be one of the few it bothers though.
    – Stephen Ostermiller ♦
    Mar 17, 2018 at 1:49
0

What I have personally noticed across my sites is that when Google crawls one of my sites for many thousands of pages one day, but then it stops crawling it over the whole week, that it decided:

  1. My site was interesting and maybe worth crawling
  2. That it turned out it wasn't interesting enough to Google to come back everyday.
  3. But it's interesting enough for Google to come back in a week or two to see if it got better.

When my sites start getting crawled by Google everyday is when I've found that Google starts to send them a lot of traffic. It seems that when my sites get crawled massively for one day or one week, but not at all the next day or week, it's because Google is interested but it doesn't want to send me much traffic yet. When I've improved the sites with infrequent crawl rates, Google has often then began crawling the site every day and sending it a lot more traffic.

As a result, I think that this is an indication that you really might want to figure out what you can do to take the site to the next level of improvement. Because Google is interested in you but may think you need to shine brighter.

This is just my thought and experience from similar crawl rate issues I've experienced and how I've interpreted them.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.