2

To my knowledge, Googlebot is currently fully capable of rendering complex SPA applications, and this is recommended as a rule of thumb.

The website is currently routed from server side and served as static pages. It has no problems with SEO and presists in Google index.

Upcoming new website has been converted to quasi-SPA powered by Angular where all SEO significant parts are still served as static pages. Client-side routing does PJAX, so all pages from SPA routes correspond to the ones served by server-side router.

The idea is that the website will gracefully degrade when Angular application is not working (isn't loaded or browser scripts are switched off). All informative content is still there, but interactive widgets (cart, real-time graphs, etc) that don't need to be searchable are being cut off.

Our intention is to not disturb Google's status quo in any way. For instance, I've encountered problems with <title> in SPAs indexed by Google (the page itself appeared ok in user's browser).

Another problem is that data-heavy widgets may take several seconds to initialize, and I would like to give Googlebot no chances to stop with time-out or mark the website as 'slow' and lower its rank.

So I consider that the website is gracefully degraded, and I would like Google to see it like some of our visitors - a simplified yet informative website with same text content. In my opinion, it is certainly not being cloaked.

TL;DR: we want to augment existing static website with dynamic features like widgets and PJAX, but also want to disallow most JS scripts in robots.txt to keep same appearance in the eyes of Googlebot in order to not disturb SEO.

Are disallowed JS scripts in robots.txt acceptable to achieve the goal? What are possible implications from Google's side? Can it consider that as cloaking and/or award a penalty? Can Googlebot ignore robots directives and peep at fully functional website somehow to get an idea on what's going on?

Improve this question
3
  • You really don't need to block JS, it's totally fine if your angular application is stop working for some time. For such a site I will allowed all js to render, and will check my major pages on cache result. Cached pages on search result will tell you how they render your app with time to time.
    – Goyllo
    Feb 17, 2018 at 11:41
  • It sounds like the server side rendered version might be a better user experience for everybody in this case. If it is going to be so slow to load with JS, you might want to use the faster server side version all the time.
    – Stephen Ostermiller ♦
    Mar 19, 2018 at 10:32
  • @StephenOstermiller Thanks for the comment, but I don't think that this will work. As I mentioned, there are data-heavy widgets. They aren't SEO significant, and there's no reason to have them server-rendered, because they will have to be re-rendered on client side (they are interactive). They cannot have their initial state (de)serialized, so there will be additional AJAX requests for these widgets that can take several seconds. I don't want Google to render widgets and do these requests, because it won't do any good, and I don't want it to mark the site as 'slow' (it generally isn't)
    – Estus Flask
    Mar 19, 2018 at 10:51

1 Answer 1

Reset to default
1

I think that Google has made it relatively clear that it wants to be able to access all of a page's content. And because javascript can lead to very vunerable scripts such as malware, viruses and browser hijacking, if you disable Googlebot's access to your .js file I imagine that it would be very leery of sending traffic to your page.

Imagine this scenario. A website has a .js script that redirects the user to a malware site, and/or causes a virus.exe application to download on the user's client side. The website then blocks Googlebot from accessing this .js file. Google then sends traffic to the page and their visitors are infected with malware. This seems like a very large security flaw that Google has almost certainly been made aware of.

As a result, my best guess is that disabling Googlebot's access to your .js files is a very risky position to put yourself in, and could potentially derank your pages altogether.

Improve this answer
1
  • Google consider cloaking only if user landed on different site, and to check that, they come with different user agent and IP address, and if there is no any redirection and actual information is still present then It don't cause any problem at all. Lot's of people block js including me. It just matter of weather you're hiding actual information or not. Google does not assume if site javascript is blocked then it might contain malware, virus etc, they don't assume such a thing, they just come and check it themself.
    – Goyllo
    Feb 17, 2018 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.