13

I have couple of websites recently created (~8 months) and Google is marking them as "Duplicate without user-selected canonical" or "Page with redirect" errors

How I implemented:

  1. Browser requests webserver for webpage
  2. Webserver returns back the HTML, CSS and JS
  3. Browser requests "Appserver" for any further user data to navigate web-application.

In above implementation, There is only once the request is sent to webserver and it light-weighted and low-maintenance.

With some researching on the topic and trying to understand on why Google is not indexing my web-pages. It seems that my webpages are all implemented as client-side rendering and Searching engines do not perform JS rendering while crawling.

If this is common issue in software architecture, How to mitigate and what are the option on the table that are used in most of the cases?

Why do Search Engines like Google not render JS as they crawl? If there any way to enable them to crawl with JS rendering?

3
  • 2
    "Why do Search Engines like Google not render JS as they crawl?" Besides if they do it or not, do also see thing as: JavaScript in a webpage is like a computer program run on the client... but under total control of the server. Even if not malicious, it can inadvertently explode RAM or CPU consumption. Said differently for any client (browser or search engine) to execute remote content (being JavaScript mostly) requires not just blind faith in humanity but also tight controls and safeguards if you don't want to explode at the first wacky website... Commented Aug 28, 2022 at 23:46
  • @PatrickMevzek "Blind faith in humanity" should be a football field and a half from any browser or search engine's JS runtime anyway. There are plenty of easy ways to sandbox some possibly malicious JS code (browsers already do that, and lightweight VMs could easily add another layer of security), and if resource consumption is a worry then cgroups or a VM would take care of that. I don't think whether or not search engines do JS rendering has much at all to do with security concerns from "wacky websites". Commented Aug 30, 2022 at 21:01
  • "There are plenty of easy ways to sandbox " Yeah of course, "easy" right. Of course. Which is why this is an issue since 20 years with multiple and different technologies and protections. Of course that is ironic. No, this is not trivial. Thinking it is trivial, is the exact first step to be sure to be hit by problems. So my advice: don't have such a naive view of things. Commented Aug 30, 2022 at 21:41

1 Answer 1

25

Google and Bing now render JavaScript when crawling websites. However, there is a lot that can go wrong in that process. The first step you should take in debugging the problem is to sign up for Google Search Console, verify your site, and then use the inspection tool. It can show a rendered screenshot of your site so that you can see if Googlebot is actually seeing your content or not.

Here is a list of the common things that can go wrong:

Only Google and Bing are advanced enough to index JavaScript websites

Other search engines such as Yandex, and Baidu are still not indexing client-side rendered websites as far as I know. Since Google has a 90%+ share of the search market, this may not be a deal breaker for you.

If you need your JS site to show up in the other search engines, you can use server-side-rendering (SSR) which is described further in a section below.

Google takes months to index new websites

Google seems to be taking months to index any new website these days, regardless of whether it it requires rendering. I'd expect an 8 month old site to have at least some of its content indexed, but keep in mind that it could just be a matter of waiting longer.

Websites that require rendering take longer

Googlebot has separate queues for regular crawling and rendering. It does a first pass to grab the server supplied HTML then it comes back later to do the rendering. Google made some announcements that typical delay between first crawl and rendering is now down to seconds. Despite that, websites that require rendering often seem to lag in indexing by days or weeks compared to pages that don't need to be rendered. See Rendering Queue: Google Needs 9X More Time To Crawl JS Than HTML | Onely

Assign each piece of content its own URL

When you are using a single-page-application (SPA) framework, it is tempting to just use a single URL for your entire website. Doing so will kill your SEO. Google needs to be able to direct users to specific content deep within your site instead of sending all visitors to your home page. That means that you need to assign each piece of content on your site its own URL. Google will only crawl and index content that has its own URL. If you have a true one-page site, Google will only ever index the content that is visible when the home page loads.

Note that you can still use SPA frameworks, you just have to use pushState to change URLs for users without causing the full page to re-load from the server.

Ensure that your web app loads and shows the correct content for any starting URL

Your web app needs to load for every URL on your site. The typical way of implementing this is to put a front controller rule into .htaccess that causes index.html to be served, regardless of what URL is requested.

Your site needs to show the correct content for that deep URL without navigating there from the home page. Googlebot will crawl your site by starting at every URL. If it doesn't get the content for that URL by visiting it first, your site won't get indexed. Additionally, users from Google need to see that deep content for the URL when the click from the search results to it.

You need to make sure that the content is visible within a few seconds of the page loading. Googlebot only allows the page to render for a limited time.

All content needs to load for a URL without any user interaction. Googlebot doesn't simulate any user interaction such as clicking, scrolling, or typing.

Render <a href=...> anchors for navigation

Googlebot only finds deep URLs in your site by scanning the document object model (DOM) for links. Googlebot doesn't click on anything, so you need to use links in your rendered HTML to tell Googlebot about all the pages.

When users click on these links, your JavaScript can intercept the clicks and load the desired content without reloading the whole page.

Pay attention to 404 errors

If a URL on your site is requested that shouldn't have any content, you shouldn't serve default content for it, you should show a "404 Not Found" error. With an SPA this is harder to do than with server-side content. Common ways of simulating a 404 that Googlebot understands are to have JavaScript change the URL to an actual 404 page, or just render an error message that says "404 Not Found."

Consider using server-side rendering

Most client-side JavaScript rendered frameworks have some way of rendering the initial page load server-side, usually by running Node.js on the server. When you implement this, search engine bots end up getting a normal HTML and CSS page which makes crawling and indexing much easier.

Users will get their first page view pre-rendered, but then use client-side rendering for their subsequent page views.

It could be a problem with your content, your link structure, or your reputation

If you have all the technical stuff related to client-side rendering figured out, there are lots of more basic reasons that Google chooses not to index content. See Why aren't search engines indexing my content?

3
  • Wow, Thanks, That's a detailed explanation. I checked my article pages (not homepage) in inspector tool by Google search console and looks good. I then submitted to index. Hoping it will get indexed soon as per crawler queue size. One other thing that is bugging me is that I add this article page in sitemap.xml but only homepage got indexed so far. Seems that a delay from Google to index my sub-pages which have articles.
    – hasanac
    Commented Aug 29, 2022 at 5:36
  • 2
    XML sitemaps do little other than tell Googlebot to come crawl extra pages. Google rarely decides to index a page just because it is in the sitemap. Google likes to find links to pages so that the pages have some reputation before it decides they are worth indexing. See The Sitemap Paradox Commented Aug 29, 2022 at 12:27
  • We also recently had another report from somebody that couldn't get Googlebot to crawl anything other than the home page: Why doesn't Google follow the links on my site while crawling? which I don't think is related to client-side rendering. Commented Aug 30, 2022 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.