7

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js) to access and use that JS variable. After publishing these changes live to the web, we began getting JS errors generated by Googlebot that are sent to us by our window.onerror function. The errors are like:

Uncaught ReferenceError: GLOBAL_VAR is not defined

Line #: 1

http://example.com/Scripts/search/index.js

When I visit the /Search/Index page on our site, I clearly see the new JS variable is there and defined.

It's almost as if when Googlebot is crawling our site, it isn't detecting that the HTML page changed. So that means it is using old, cached HTML that doesn't include the setting of the GLOBAL_VAR variable, which causes the JS error.

We published the HTML/JS changes on 2/6, and we are still getting JS errors as of 2/10. I would have thought Googlebot would recognize the HTML changes by now. We've never experienced this problem in the past.

Why would Googlebot not update their cache if the HTML of a page changes? Most importantly, how can we get Googlebot to detect the HTML changes and update their cache, so we stop getting these JS errors?


2019 UPDATE - One thing I'll add here is that we use Microsoft Azure Cloud Services that have a production and a staging environment. We publish to staging, swap staging to production, test the new production, then delete staging. The scenario described above appears to only occur when swapping/deleting environments, so it must have something to do with that. It's like Googlebot is crawling our site when we swap staging/production, and it starts getting some data from staging and some from production, which distorts their cached resources from our site. It is still happening and still filling up our error logs.

  • How are you getting JS errors in your server logs? Only client side software usually encounters JavaScript errors. – Stephen Ostermiller Feb 10 '16 at 16:26
  • 1
    We have a call to window.onerror to send each JS error to our server via Ajax. – John Washam Feb 10 '16 at 16:36
  • I tried that once, but I got so many errors from ads, that they drowned out any errors from my own site. – Stephen Ostermiller Feb 10 '16 at 16:40
  • Just to be clear, it appears that Google is using the latest .js file, but is rendering old versions of the pages? It is possible that Googlbot doesn't render pages as it downloads them. It may download them and save them to be rendered later. How long ago did you make this change? – Stephen Ostermiller Feb 10 '16 at 16:43
  • 1
    It sounds like Googlebot has a render queue that is separate from the download queue and can have several days worth of pages in it. I'm not sure what the solution for the problem is. – Stephen Ostermiller Feb 10 '16 at 16:54
0

The easiest way to solve this is to version your JavaScript files. Then only new pages will fetch the new JavaScript.

  • Our JavaScript files are versioned on production (they append a query string parameter to signify them as distinct from previous versions of the same file). Again, the problem is not with the JS file. Googlebot has the newest JS file (as evidenced by us seeing the correct query string param in the JS error), but Googlebot does not have the newest HTML that is referencing the new JS file. We know this because Googlebot is using the new JS file that uses a global variable that is set on the new HTML page. – John Washam Feb 10 '16 at 20:27
  • I considered this, but most JavaScript versioning solves the opposite problem. Users are using cached version of the JavaScript files and the versioning forces them to fetch the new one. Versioning would only work if you kept a copy of the old JavaScript file around to serve when you get the old version parameters. Usually you version by serving the newest version no matter what the parameters are. That will solve this problem. – Stephen Ostermiller Feb 12 '16 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.