7

I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot).

But I have 1 article that have 300 page views in only 12 hours. I am curious, so I track the user agent and IP address and found that my page crawled by Googlebot. Too often.

Page view track

My question is, what makes Google crawl this page too often, since other page is not? Do I need to worry about this?

  • 2
    Keep in mind, 300 over 12 hours is ~25 per hour, or approximately one every other minute. In terms of server load, that's practically nothing. If your page is huge, and that usage continues over an extended period of time, it could be a marginal problem for your bandwidth, but as others said it seems to be based on the URL's popularity, which means it should still remain a small fraction of the real traffic you get. – Doktor J Oct 14 at 19:25
  • There's a chance it might not actually be the real Googlebot. Here's how to find out support.google.com/webmasters/answer/80553?hl=en – Tony McCreath Oct 14 at 20:34
9

Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others.

URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.

Also, John Muller confirmed that on Twitter.

enter image description here

In addition, how Google's crawling works could also affect access to your website.

Some pages are known because Google has already crawled them before. Other pages are discovered when Google follows a link from a known page to a new page. Still other pages are discovered when a website owner submits a list of pages (a sitemap) for Google to crawl. If you're using a managed web host, such as Wix or Blogger, they might tell Google to crawl any updated or new pages that you make.

3

There is no need to worry about this. In fact, it's a good thing, this way google would be able to index your updated content and would increase the chances of your page ranking higher.

If you, however, do not want to get your page indexed by google(for any reason whatsoever), there are multiple ways to do that.

You can add a noindex tag on the head of your page. Or if you want to try and reduce the crawl frequency, you can decrease the crawl rate

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.