3

I wanted to understand how Googlebot (and other crawlers) crawls my site. Specifically whether it passes a document.referrer and if it maintains localStorage keys, so I implemented a script via Google Tag Manager that detects these crawlers and logs data to Logstash.

This is the condition I'm using to detect crawler user agents (returns true for crawlers):

function() {
if(navigator.userAgent.indexOf('robot de Google') < 0 &&
   navigator.userAgent.indexOf('Googlebot') < 0 &&
   navigator.userAgent.indexOf('bingbot') < 0 &&
   navigator.userAgent.indexOf('msnbot') < 0 &&
   navigator.userAgent.indexOf('BingPreview') < 0 &&
   navigator.userAgent.indexOf('Yahoo! Slurp') < 0) {
      return false;
   } else {
      return true;
   }
}

And this is the tag that sends sends a request to Logstash via an image pixel on the GTM Pageview event:

<script type="text/javascript">
(function (d) {

var pagePath = encodeURIComponent(document.location.pathname);
var pageReferrer = encodeURIComponent(document.referrer) || "null";
var userAgent = encodeURIComponent(navigator.userAgent);

var viewCount = Number(localStorage.getItem("preview_view_count")) + 1 || 1;
localStorage.setItem("preview_view_count", viewCount);

var js;
js = d.createElement('img');
js.style = 'display:none;';
js.alt = 'tracking img';
js.src = 'http://MY_LOGSTASH_ENDPOINT_DOMAIN/pixel.gif?EVENT=LogCrawl&USER_AGENT=' + userAgent + '&PAGE_PATH=' + pagePath + '&PAGE_REFERRER=' + pageReferrer + '&VIEW_COUNT=' + viewCount;
d.body.appendChild(js);
})(window.document);
</script>

Now when I look at Logstash, I only see 40 hits in the last 4 days from Googlebot, but Search Console reports ~50,000 pages crawled per day.

Has anyone tried to log Googlebot with GTM before? I'm trying to figure out if there is something wrong with my script, or if Googlebot just doesn't execute Javascript most of the time.

Any ideas are highly appreciated. Thanks.

1

The Search Console reports the overall number of files (!) that are downloaded (source). This includes JS / CSS / image files etc. and leads to quite huge numbers, depending on how much caching is involved.

So you might get more accurate numbers for crawled HTML pages by analyzing your server logs (if you do that: don't forget to verify hits from Googlebot, there's a lot of pretenders out there).

Regarding your tag: localStorage should be supported by Googlebot, but is unlikely to work across multiple page impressions. So if you took those 40 hits straight from VIEW_COUNT, you might instead want to check back to your log and add those hits up by yourself.

(very interesting!)

  • thanks! Great to know that Search Console reports all files and not just HTML. The localStorage view count metric and my logs have allowed me to determine two things: Googlebot does not maintain state from page to page (i.e. no document.referrer, etc), and it does not seem to execute JS on most hits (or, if it does, it must be doing it at a later time "offline" after having crawled the page, since otherwise I should be seeing a lot more hits on my client side tracker). – Ralph Apr 6 '18 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.