In a client-side-rendered app, some pages are successfully crawled by Googlebot while others fail to fully render. How can I get all pages to render?

Question

The /home and /home/search pages render fine, but /home/product/ID_XYZ pages appear as just a loading icon when doing a "Live Test" in Google Search Console.

There has been some flux recently in the URL of product pages. I thought this may be related and found the sitemap was broken. I fixed this but rendering issues persist. In GSC the new sitemap URLs are "Excluded" with errors: "Crawled - currently not indexed" and "Duplicate, submitted URL not selected as canonical" (but the duplicate URL is completely unrelated, like /about basically).

When testing in Google Lighthouse I see the following latency metrics:

/home

FCP: 0.7s - Speed Index 2.5s - LCP 3.6s - Time to Interactive 5.1s - Total blocking time 1.02 s - Cumulative Layout Shift 0.254

/home/product/ID_XYS

FCP: 0.7s - Speed Index 4.5s - LCP 5.2s - Time to Interactive 6.1s - Total blocking time 1.32 s - Cumulative Layout Shift 0.008

Obviously this would be fixed with SSR or prerendering, but I am trying to find faster/easier solutions first. Would this be purely a latency issue? I cannot find data on how long Google waits for a page to render, and I hoped fixing the sitemap would give it another signal that the link is valid and it should wait longer, but that didn't work. Is there anything I can do besides strictly targeting latency improvements? What would be a realistic target to hit?

Answer 1

It's unlikely the latency issue. I mean, I haven't seen how fast it works and whether lighthouse was able to fully render the page.

Given that, yes, a prerenderer would be the simplest solution here.

But you still can do a bit of debugging in here (in no particular order):

1. Record client-side JS errors and warnings and then check what errors Google has when you're asking it to preview the page through GSC (Google Search Console). Debug from there.
2. Make sure none of your scripts are blocked based on user-agent. Change your user-agent to Google bot's and make sure the site responds adequately to that. I saw modern designs making mistakes by not rendering stuff properly for unrecognized/unlisted useragents.
3. Don't use Lighthouse, use your devtools profiling abilities and debug your load times there, while doing so, use VPN and try different locations to make sure it's not a geo issue.
4. Finally, although this may be one of the first things to do, check your access and error logs of your web server, see what server-side errors may happen to the googlebot useragents.

Answer 2

I have found the issue.

In Google Search Console you can examine a URL and do a "Live Test". When live testing the broken URLs and the working URLs I noticed a difference in the "More Info" tab. The broken URLs had JS Console Errors and 2 more Page Resources that could not be loaded than in the working page (which still had 3 page resources unavailable). One of these blocked resources was a script!

When I opened the Page Resources that couldn't be loaded, the URLs were something like static-cdn.com/FOLDER_NAME/a10sdnk23asdkaskn32dnsd and were failing with a "Blocked by Robots.txt" error.

Turns out my company's internal CDN required explicit whitelisting of FOLDER_NAME in the CDN's robots.txt. Since it wasn't whitelisted, the CDN's robots blocked the Googlebot request to load the script file that finished loading the product page. It was just a coincidence that the working pages had FOLDER_NAME2 that was already whitelisted, and the broken files had FOLDER_NAME.

I also double backed and checked the "Excluded" links on the "Coverage" page in Google Search Console. Our number 1 reason for exclusions was "Blocked by robots.txt". This was perplexing because the "Test robots.txt blocking" tool said none of these links had issues with the robots.txt and they should all be allowed. Turned out it was the CDN's robots.txt that was blocking it, not the actual website's.