3

We have been experiencing constant crawl anomalies caused by googlebot continued crawling of 'rnd.js?asdfasdfasfs3423' (hash is random on each pageload). It has been 3 months since we removed rnd.js from all of our pages, yet googleboot insists on fetching this rnd.js, and then it shows a crawl anomaly in the webmaster tools (time and date matches up almost to the second with our nginx logs).

What can be done to force googlebot to fetch the parent html file, and only request resources from what it gathers at that exact time, and not rely on a 'stale' version of an old html. Furthermore, some of the urls that google appends as referral don't even exist anymore since we have all of our http pages 301 to https, so there is no way google has just crawled an http version and found rnd.js on it.

All rnd.js requests are returning 410 for over 3 months now, but our crawl anomalies have not gone away. All rnd.js requests that show up in our nginx logs, are mirrored in google webmaster tools, meaning that these rnd.js are responsible for the crawl anomalies we are encountering.

Any suggestions or information is welcome.

|improve this question
  • Quick addition, crawl anomalies are for parent html files and not the rnd.js file itself. Parent html files are the ones that used to include rnd.js. – Jason McNoles Aug 28 at 19:29
  • Can you confirm this behavior with corresponding log file? – Evgeniy Aug 28 at 21:42
  • 66.249.64.198 - https www.i2m.example.com - - [28/Aug/2019:01:23:30 -0400] "GET /rnd.js?f839dfe4e5857b15faaa0678e94c0535 HTTP/1.1" 410 538 "http//www.i2m.example.com/someurl" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +google.com/bot.html)" "-" >> 0.000 - – Jason McNoles Aug 28 at 22:25
  • Regarding crawl anomaly, it will show up in webmaster tools within a few days. Crawl anomaly will be for the parent url which is www.example.com/someurl according to the nginx logs we've just posted. We used this rnd.js to attach HSTS header so that all future connections are on https instead of http, that is my refurl sent by googlebot is in http, but request is in https. HSTS header then caused a 307 redirect. We are no longer using this method, and simply forward all http to https with a 301. But this resource seems to be important to google if it wants it. – Jason McNoles Aug 28 at 22:27
  • We are also not doing 301 on this resource to www.example version of our domain, just so we can 410 it as soon as possible, as any extra redirects within this request chain will further cause crawl anomalies. – Jason McNoles Aug 28 at 22:31
1

Once Googlebot discovers a working URL on your site and crawls it, it will NEVER stop crawling it. It doesn't matter what status code you put on that URL. Googlebot will periodically come back and check on those URLs in the hopes that the content that used to be there returns. I have URLs that have been redirecting for 15 years. They don't have any links to them anymore. Googlebot still occasionally comes to check on them.

Using a "410 Gone" status is the most correct status to use. However, Google says that it treats it almost exactly the same as a "404 Not Found". That is, it may get removed from the search index slightly more quickly than a 404, but Googlebot will continue to come back and check on the URLs just as frequently as 404 Not Found".

Having crawl anomalies listed in Google Search Console doesn't hurt your site. Google reports them to you so that can evaluate whether or not they are actually problems. Google's John Mueller has this to say about 404 errors:

404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html

The same applies to your "410" errors. They are listed so that you can fix them if they are actually problems, but you can ignore them if the URLs are supposed to return that status.

When you say that the parent pages haven't been linking to those resources for three months, what you are experiencing is the render queue crawl delay. Google uses two modes of crawling:

  1. HTML crawling with indexing based on the source code
  2. Render crawling where resources are also fetched, and what is rendered is indexed.

The rendering mode is much more expensive for Googlebot. It has a queue of pages to render that is months long. Googlebot is trying to render pages for which it fetch the HTML months ago. When it can't render the pages due to the JavaScript now being gone, it complains and may de-index the parent pages.

In practical terms for a webmaster that means that you should keep the resources needed to render a page stable for months, even after you change the page so that it doesn't need those resources. I think the only two ways to fix that problem are:

  1. Put the JavaScript back so that Googlebot can render months old pages
  2. Use the Google Search Console Inspect URL tool to fetch and render those pages (one at a time) so that it picks up the latest version.
|improve this answer
  • The 410 responses are for resources and NOT urls. It is the rnd.js file thats responding with a 410 and the crawl anomaly registers only when rnd.js is requested by googlebot. Crawl anomalies do hurt rankings, primarily since until they are gone the parent HTML url can't be indexed, and any internal/external link ranking to and from parent url will also be deminished. – Jason McNoles Aug 29 at 13:21
  • We do have 410 for some actual html URLS that are gone, for example, page 52 of a certain category which no longer has any content will return a 410, and there is no issue there. What's puzzling is these specific JS resources that used to be included on all of our pages have been gone for 3 months, but googlebot insists on fetching them, and if it can't it will show a PARENT URL crawl anomaly, obviously de-indexing the page completely. – Jason McNoles Aug 29 at 13:24
  • rnd.js?asdfasdfasfs3423 is a URL. Resources are served from URLs. Adding to the answer to address the 3 month delay. – Stephen Ostermiller♦ Aug 29 at 14:11
  • If it is a URL in its own right, why is the parent URL de-indexed based on this url resource? It is the core of the whole issue. A resource that is no longer on any pages still requested by google, and the parent url is punished for it. How exactly does this NOT harm my rankings? When 95%+ of my urls are de-indexed because of this, and the ones that are still alive dropped down dozens of positions? Rnd.js does not show up as a page/url within webmaster tools, only its parent url the .html file shows up. – Jason McNoles Aug 29 at 14:27
  • I see googlebot crawling a fresh version of the parent urls all the time, why does it still insist on fetching the stale/nonexistent resource urls from that page? – Jason McNoles Aug 29 at 14:31
0

I see firstly in my life, that Googlebot visits 410 URLs twice. Nice finding - good to know!

But i think to know the cause of this behavior - this should be the always new hash, which appears as parameter. Googlebot "thinks" it would be always new url and crawls it. If you set parametrizing off - it stops crawling immediately.

|improve this answer
  • Duplicate 410 rnd.js visits are because of the subdomain issue we had. We did not 301 all random subdomains such as i5.example.com or www.ina_1234.example.com to www.example.com. From our logs we did not notice duplicate googlebot visits for the same subdomain and the same rnd.js?randomhash. Sometimes google picked up multiple rnd.js?randomhash files from the same url, and it does attempt to fetch them all. – Jason McNoles Aug 29 at 10:07
  • Regarding parameterizing, we have set to 'no effect' our md5 parameter on mp4 files, and googlebot still attempts to fetch stale md5 parameters for our mp4 files, and again it also causes a crawl anomaly even know this file is no longer available. The essence of the issue lies in the fact that google has its own 'resource to crawl cache' and doesn't compare it to the latest version of the page. Crawls those stale js/css files and throws crawl anomalies for the parent html. – Jason McNoles Aug 29 at 10:29
  • Btw, I could not find the option within google webmaster tools that would set all parameterizing to off, and ignore all and any parameter by default. – Jason McNoles Aug 29 at 11:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.