Does Googlebot execute Google Tag Manager?

Question

I wanted to understand how Googlebot (and other crawlers) crawls my site. Specifically whether it passes a document.referrer and if it maintains localStorage keys, so I implemented a script via Google Tag Manager that detects these crawlers and logs data to Logstash.

This is the condition I'm using to detect crawler user agents (returns true for crawlers):

function() {
if(navigator.userAgent.indexOf('robot de Google') < 0 &&
   navigator.userAgent.indexOf('Googlebot') < 0 &&
   navigator.userAgent.indexOf('bingbot') < 0 &&
   navigator.userAgent.indexOf('msnbot') < 0 &&
   navigator.userAgent.indexOf('BingPreview') < 0 &&
   navigator.userAgent.indexOf('Yahoo! Slurp') < 0) {
      return false;
   } else {
      return true;
   }
}

And this is the tag that sends sends a request to Logstash via an image pixel on the GTM Pageview event:

<script type="text/javascript">
(function (d) {

var pagePath = encodeURIComponent(document.location.pathname);
var pageReferrer = encodeURIComponent(document.referrer) || "null";
var userAgent = encodeURIComponent(navigator.userAgent);

var viewCount = Number(localStorage.getItem("preview_view_count")) + 1 || 1;
localStorage.setItem("preview_view_count", viewCount);

var js;
js = d.createElement('img');
js.style = 'display:none;';
js.alt = 'tracking img';
js.src = 'http://MY_LOGSTASH_ENDPOINT_DOMAIN/pixel.gif?EVENT=LogCrawl&USER_AGENT=' + userAgent + '&PAGE_PATH=' + pagePath + '&PAGE_REFERRER=' + pageReferrer + '&VIEW_COUNT=' + viewCount;
d.body.appendChild(js);
})(window.document);
</script>

Now when I look at Logstash, I only see 40 hits in the last 4 days from Googlebot, but Search Console reports ~50,000 pages crawled per day.

Has anyone tried to log Googlebot with GTM before? I'm trying to figure out if there is something wrong with my script, or if Googlebot just doesn't execute Javascript most of the time.

Any ideas are highly appreciated. Thanks.

Answer 1

The Search Console reports the overall number of files (!) that are downloaded (source). This includes JS / CSS / image files etc. and leads to quite huge numbers, depending on how much caching is involved.

So you might get more accurate numbers for crawled HTML pages by analyzing your server logs (if you do that: don't forget to verify hits from Googlebot, there's a lot of pretenders out there).

Regarding your tag: localStorage should be supported by Googlebot, but is unlikely to work across multiple page impressions. So if you took those 40 hits straight from VIEW_COUNT, you might instead want to check back to your log and add those hits up by yourself.

(very interesting!)