1

Whilst analysing the server log files using Screaming Frog Log File Analysis I have noticed that Googlebot is accessing dynamic CSS and JS files that return a HTTP status of 404.

The site is on Wordpress and uses the Automizer plugin.

They purge the cache every time they change the files so I think the non existing CSS and JS files are ones that Googlebot has cached.

Is there anyway to prevent Googlebot from recrawling these files once they 404? Would it be best to automatically create a 301 redirect when the CSS/JS files change?

  • 1
    Once Googlebot crawls URLs, it will likely continue to crawl them forever. If you can redirect, it is always a good idea to do so. – Stephen Ostermiller Dec 14 '17 at 15:31
  • 3
    That's because your setup is creating revision numbers on the JS/CSS files which is generally a bad idea on a live site since you want visitors having a local cached copy and then having control when to have them re-fetch using expires. Having to rewrite old css to new css everytime you purge the cache is a waste of your time, fix the underlying issue... your plugin/setup or accept that 404 is the 'RIGHT' server response since that file no longer exists, Google will not punish you for it... 404 is not a BROKEN server. – Simon Hayter Dec 14 '17 at 16:47
  • If you need to produce dynamic css and js use a versioned/random query string which you ignore. but which means the user will never reuse a cached copy – Rob Sedgwick Dec 15 '17 at 17:14
  • I've asked the developer about feasibility on redirecting the CSS/Js files and they believe a 404 is the correct response. Multiple resources are concatenated, compressed and served with a far future expiration date. The resources have a unique name so that when the cache is purged, the user is served a fresh file rather than their browser respecting the previous requests expiration date. What is the SEO impact vs the performance gain? – GrapeSoda Jan 10 '18 at 13:17
1

It sounds like the site is still linking to old previous versions or they are linked somewhere like the sitemap or the HTML of the page. Setup a 301 redirect to resolve the symptom of the problem.

Do you need these files cached? if not, disallow them from the robots.txt. Or overtime a new CSS and JS is created, disallow the old ones.

It sounds like the plugin is not working properly as those files should return a 404, but they should not be accessible in terms of links.

Also why does the cache need to be cleared if the new versions are setup with proper version control? Something worth looking into.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.