0

According to Google Search Console for a website I am working on, Googlebot crawls ~5000 pages per day (min 2500, max 8500).

However, when looking at the Apache log files, GoogleBot only shows up ~10 times per day ...

For example: 

66.249.64.88    [22/Jan/2020:15:09:01   +0100]  [22/Jan/2020:15:09:01 +0100]    GET / HTTP/1.1  200 1358    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

It is GoogleBot since doing a reverse dns does point to Google servers:

$ host 66.249.64.88
88.64.249.66.in-addr.arpa domain name pointer crawl-66-249-64-88.googlebot.com

But I am wondering : If GoogleBot appears only 10 times in Apache log files while it crawls 5000 pages per day, where are the remaining 4990 crawls going?

How can I know which resource GoogleBot crawls when it does not appear in the log files ?

Thanks!

Improve this question

1 Answer 1

Reset to default
1

The value Google is showing (~5000) it's how much they "set aside" to crawl your website, based on what they think it'll be needed - it doesn't mean they will use it or even that there's a need for this much.

If your website isn't large and/or doesn't get to much organic traffic, having 10 visits per day isn't unheard-of.

Look into which pages are being visited instead; if important pages are not getting crawled then, yes, you have a problem.

Improve this answer
2
  • Well yes I think there is a problem: 90% of GoogleBot entries show a 404 when trying to access the robots.txt. However, the robots.txt is directly accessible from root and it does return a 200, I checked with curl --head ... And apart from the homepage, Googlebot accesses very few other pages / resources in 2 weeks, like 2 or 3 product pages ...
    – Antoine Brunel
    Feb 2, 2020 at 12:36
  • Yeah, it does look like you are having some issues - depending on the size of your website, it can be some sort of "crawler trap": searchenginejournal.com/…
    – Andre Guelmann
    Feb 2, 2020 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.