The /home and /home/search pages render fine, but /home/product/ID_XYZ pages appear as just a loading icon when doing a "Live Test" in Google Search Console.

There has been some flux recently in the URL of product pages. I thought this may be related and found the sitemap was broken. I fixed this but rendering issues persist. In GSC the new sitemap URLs are "Excluded" with errors: "Crawled - currently not indexed" and "Duplicate, submitted URL not selected as canonical" (but the duplicate URL is completely unrelated, like /about basically).

When testing in Google Lighthouse I see the following latency metrics:


FCP: 0.7s - Speed Index 2.5s - LCP 3.6s - Time to Interactive 5.1s - Total blocking time 1.02 s - Cumulative Layout Shift 0.254


FCP: 0.7s - Speed Index 4.5s - LCP 5.2s - Time to Interactive 6.1s - Total blocking time 1.32 s - Cumulative Layout Shift 0.008

Obviously this would be fixed with SSR or prerendering, but I am trying to find faster/easier solutions first. Would this be purely a latency issue? I cannot find data on how long Google waits for a page to render, and I hoped fixing the sitemap would give it another signal that the link is valid and it should wait longer, but that didn't work. Is there anything I can do besides strictly targeting latency improvements? What would be a realistic target to hit?

  • 1
    Sitemaps have very limited effect on SEO. They can get Googlebot to crawl URLs and give you extra stats in GSC. But I wouldn't expect Google to give URLs contained in a sitemap special treatment like longer render times. Feb 22, 2022 at 20:00

2 Answers 2


It's unlikely the latency issue. I mean, I haven't seen how fast it works and whether lighthouse was able to fully render the page.

Given that, yes, a prerenderer would be the simplest solution here.

But you still can do a bit of debugging in here (in no particular order):

  1. Record client-side JS errors and warnings and then check what errors Google has when you're asking it to preview the page through GSC (Google Search Console). Debug from there.
  2. Make sure none of your scripts are blocked based on user-agent. Change your user-agent to Google bot's and make sure the site responds adequately to that. I saw modern designs making mistakes by not rendering stuff properly for unrecognized/unlisted useragents.
  3. Don't use Lighthouse, use your devtools profiling abilities and debug your load times there, while doing so, use VPN and try different locations to make sure it's not a geo issue.
  4. Finally, although this may be one of the first things to do, check your access and error logs of your web server, see what server-side errors may happen to the googlebot useragents.
  • Both just have 404 errors in the JS console. Network tab shows HOMEPAGE: 4.1MB transferred, 16MB resources, Finish 1.7min, DOM content load 1.01s, Load 1.79s PRODUCT: 5.5MB transferred, 23.6MB resources, Finish 3.4min, DOM load 1.16s, Load 2.4s. DOM load looks very close for both and acceptable. Putting Googlebot as the user agent doesnt seem to change anything on the site, and I don't see error logs on the server.
    – James L.
    Feb 23, 2022 at 23:04
  • Just 404? For what? what endpoint returned the 404? Go debug this 404, go to the web-server logs, find the 404 and debug why YOUR backend returns a 404. As I said, doesn't look like a latency issue.
    – BNazaruk
    Feb 23, 2022 at 23:19

I have found the issue.

In Google Search Console you can examine a URL and do a "Live Test". When live testing the broken URLs and the working URLs I noticed a difference in the "More Info" tab. The broken URLs had JS Console Errors and 2 more Page Resources that could not be loaded than in the working page (which still had 3 page resources unavailable). One of these blocked resources was a script!

When I opened the Page Resources that couldn't be loaded, the URLs were something like static-cdn.com/FOLDER_NAME/a10sdnk23asdkaskn32dnsd and were failing with a "Blocked by Robots.txt" error.

Turns out my company's internal CDN required explicit whitelisting of FOLDER_NAME in the CDN's robots.txt. Since it wasn't whitelisted, the CDN's robots blocked the Googlebot request to load the script file that finished loading the product page. It was just a coincidence that the working pages had FOLDER_NAME2 that was already whitelisted, and the broken files had FOLDER_NAME.

I also double backed and checked the "Excluded" links on the "Coverage" page in Google Search Console. Our number 1 reason for exclusions was "Blocked by robots.txt". This was perplexing because the "Test robots.txt blocking" tool said none of these links had issues with the robots.txt and they should all be allowed. Turned out it was the CDN's robots.txt that was blocking it, not the actual website's.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.