4

The website has around 300 URLs.

There are no Mobile Usability Issues reported in the google search console.

There are no Security Issues reported in the google search console.

There are no Ad experience issues reported in the google search console.

The Website uses HTTPS and that shows up green in the google search console as well.


There are 50 URLs (out of ~300) discovered by google search console with problems in LCP. All the URL are in pending status (not failed).

Testing each of the 50 reported URLs 1 by 1 using https://developers.google.com/speed/pagespeed/insights/ we discover that all of them have

  • all scores above 90% (so GOOD/FAST category)
  • all FCP, LCP, CLS in the green zone

Some of the links have some issues with Time to Interactive, but only a smaller subset.

The 50 reported URLs are grouped by google search console in 2 groups of "similar URLS".


Also, after testing using Pagespeed Insights all 300 URLs 1 by 1 multiple times, there are at least 150 URLs (around 50%), even with the most pesimistic estimates, that have EVERY metric Green (Lab Data Only, since Field Data not available):

  • FCP < 1800 ms (GREEN)
  • LCP < 2500 ms (GREEN)
  • TBT < 100 ms (GREEN)
  • CLS < 0.1 (GREEN)
  • Interactive < 3800 ms (GREEN)
  • Speed Index < 3400 ms (GREEN)
  • Category FAST

The Google Search Console Report has been run multiple times. At least one full 28-day cycle. The results are updating at the moment as well.

The site started since launch with 0% Good URLs and in less than a month went up to 10% good URLs then back to 0% Good URLs.

I understand that the feedback is not immediate, and that the data is updated a different timeframes from different sources, but 1 month should be enough time to start registering those 150 good URLs.

The Origin Summary Data doesn't look too good, but even that shows 50% green.



Question 1: What else can I do to find out why google search console is not discovering any of the 150 good URLs from the website?

It can't be just a matter of time since I've been seeing frequent updates in the past month going from 0% to 10% and back to 0%, right? The most recent update was yesterday.


Update here is an example of an URL I think should be detected as GOOD URL by page experience:

How is this link not detected as a good URL? There are a lot of links with green score. Actually very few have a score under 90 on the entire domain.At least SOME of them should be seen as good urls, not 0%.

Right now, for a good number of days there are 0 Good URLs on Desktop and 0 Good URLs on Mobile.

enter image description here

And this is what core vitals is saying about this domain:

Core vitals summary

7
  • 1
    When you test yourself with PageSpeed Insights, you are doing a "lab" test. Google bases its evaluations on "field data" rather than lab data. That is data collected from real users as they browse your site. What is Google Search Console reporting for field data for your site? Jul 14, 2021 at 14:17
  • It says not enough data for all urls except the root. Jul 15, 2021 at 8:53
  • 1
    I have 100,000+ pages on my site and Google has "enough data" for only 70 of them. Google says that they are planning to base page experience on "similar pages" when there isn't enough data. It isn't clear to me which pages they will choose or how they will combine data from all the pages. Jul 15, 2021 at 8:58
  • @StephenOstermiller so basically, you are saying that it falls back to origin summary data? For this domain I think I don't have field data for any of the pages. If there is no field data for any of the pages, how do they calculate the origin data? An what on earth can be done to register these good URLs? Jul 21, 2021 at 11:06
  • 1
    Google has said they're not going to use lab data at all. When they don't have enough field data for a page they'll average it over your whole site. Jul 21, 2021 at 15:51

3 Answers 3

2

The CRuX (Chrome User Experience) data used by the search console is aggregated over a rolling previous 28 days. This means the last day reported is based on the average of the 28 days leading up to it. This means any changes will take 28 days to take full effect.

It is also real user data. Users may have slow machines on slow connections which are not near the server.

While you are probably testing on your own fast machine with a good connection.

This means what you see in your tests is typically not representative of the real user data used by Google.

The origin data is closer. It is real user data aggregated for your whole site. This means some pages could pull down the final scores. If your lucky you may also see real user data at the page level.

6
  • Thank you for the answer. The good URLs are updating daily. I know some users may have slower machines than the average. Yes, the Field Data differs from Lab Data. But the difference cannot be THAT big. There are a good number of URLs which get very very good scores on all mertrics. Even if you cut by half the performance it should still pass as good URLs for at lease some of the 300 URLs. Jul 14, 2021 at 8:35
  • I just don't understand why the number of good URLs are 0% after 2 months almost. Jul 14, 2021 at 8:36
  • 3
    To clarify what 0% of good URLs means, it doesn't mean that 100% of visitors to those pages had a bad experience and all page loads were bad. It means that none of the pages had "good" experiences (e.g. meet the Core Web Vitals metric thresholds) for at least 75% of visitors. In other words, even if 74% of visitors to your pages experience super-fast loads, since that's less than 75% those pages would still not be considered "good URLs". Does that make sense? Jul 14, 2021 at 20:44
  • Thank you @PhilipWalton, yes I am aware of the 75 percentile field in the CrUX field data. It makes sense. But that is not what I am baffled about. I am manually testing all URLs and generating averages. Pagespeed lab data results vary quite significantly up to 50% diferences in values. But when I run 100 tests with pagespeed, and make the averages and the values are all significantly better than the minimum threshold for green, I don't understand how at least some of the URL's are not seen as Good URL's in the page experience. Even a user with bad connection should still get a good speed. Jul 21, 2021 at 10:41
  • The only explanation I could have for now is that there is something bad across the entire domain that CrUX is detecting but lighthouse does not. Otherwise it's quite ridiculous to have 100+ URLs all with very green results in pagespeed lab data, and none of the are categorised as good URL in page experience. Jul 21, 2021 at 10:44
2

I have exactly the same problem/ condition described by Vlad and Tony´s answer doesnt convince me, since page speed insights is supposed to emulate a slow 4G in a mid tier device. We will never achieve green in search console because we can´t controle where people are accessing our websites. In Portugal, the mobile network has a really bad coverage. So the conclusion is that I will never get a green in search console.

2
  • Having a lot of users with poor connections is a concern I've heard raised. Another way of thinking about it is that speed is even more important if your visitors have bad connections. And this this will affect your competitors with the same audience in the same way. It may be hard for you to get the benefit of a good rating, but it will also make it hard for your competition. But to be honest, I had a single page that attracted lots of users will poor connections. The page was of little value to me so I dropped it to boost my score! Jul 15, 2021 at 9:20
  • @TonyMcCreath that makes sense, but my problem is that none of the 300+ pages are registered as GOOD URLs. I want to figure out how to start registering them, what else I can do. Jul 21, 2021 at 11:09
2

OK, So I think I finally managed to piece it together. (please confirm or infirm what I am saying below)

  1. The lab data is not counted at all towards the page experience or core web vitals data.

  2. The lab data on pagespeed, which is generated using lighthouse is only a "debugging" tool that can give you a rough indication where you whave some problesm, but the numbers should not be taken as is, since the real-world data might actually be quite different. The general idea is that if you make a change that shows an improvement in the lab data will probably show an improvement in real-world data as well, but the values are relative, not absolute.

  3. Google Search console uses ONLY real-world data to generate the scores in page experience and detect issues.

  4. Real-world data is gathered using CrUX tool, that captures real analitics from the browser from users that have usage statistics options turned on in the browser settings.

  5. The tool/mechanism/algorithm that captures real-world data is different from the one that captures the live data. CrUX does not use lighthouse engine.

  6. The priority of data is the following:

    • If the URL has a lot of real-world traffic with users that have usage statistics on (not sure just how much), then CrUX will generate Field Data REPORT.
    • If the URL does not have enough real-world data to generate a field data report, then it will fallback to the origin summary report.
  7. The Field data report is generated by

    • calculating an average for each metric for all the real-world views (FCP, LCP, CLS, FID)
    • calculating the % of the views that are green, yellow or red on that specific URL for each specific metric.
    • 75% green, 15% yellow and 10% red means that from 100 views on that page, 75 passed the score for that specific metric (eg, FCP < 1500), 15 were averagely slow and 10 were very slow.
  8. If the URL has Field data report, it will pass and become a GOOD URL, IF the total score > 90% for at least 75% of the real-world views.

  9. Also a GOOD URL must also have no mobility issues, no security issues and no ad experience issues in the first place. (0, 0, 0)

  10. This means that a good URL can still have some low scores for some of the metrics for some of the views, as long as the total score is 90+ for 75%+ of the views on that URL.

  11. What happens, when you don't have FIELD DATA REPORT generated? This does not mean that you don't have any field data at all, most fo the times, but you probvably don't have enough data to make a reliable aggregated score/conclusion/averages.

  12. If you don't have the field data report generated, the URL will inherit the score from a "similar" URL that has enough field data.

  13. If the URL does not have any similar URL that has field data report generated, then it will fallback on the origin summary.

  14. The origin summary is probably calculated by averaging all the views from all the pages, all mushed together. I gues that fi you have 1 bad URL with 99 bad views, and 1 good URL with 1 good view, you will get 1% green and 99% red in origin summary data.

  15. If you have 1000 pages with 2 views each then you have a total of 2000 views per domain which will allow you to have enough data to generate a reliable origin summary. But 2 views on a page are probably not enough to give a reliable score for a specific page, so google says it's probabyl better to use the domain summary origin dtaa for all the pages that do not have traffic.

Now the solution:

Why I think I have 100% Bad URLs and 0% GOOD URLs:

  • When the website was launched the traffic registred real-world views on some bad pages that generated a bad origin summary.
  • Even though, since then all pages have been improved drastically, and there were technically good URLs then as well, none of them are being registered by webcore vitals, because they don't have enough individual traffic on the specific URL and applies the origin summary to ALL URLs, which makes the all fail.
  • Technically, what I can do is
    • I can improve the overall experience on all pages, especially the 150 bad URLs, which will improvle the Origin Summary to pass the thershold, and when that happens technically the GOOD URLs will jump from 0% to 100%, since there is no field data report for any URL. But making the improvements is not enough. My problem was that all the imprvements are made, and the origin summary should improve, but it isn't, and that is because, in order to improve the origin summary you need to generate more traffic. When the NEW views will affect the averages enough to pass the 75% validation thereshold, then the GOOD URLs will be registred. So you need to fix the pages and the you NEED traffic on the domain.

    • I can try to geenrate traffic on the URLs which I know are fixed in order to generate the field data report for that URL. If one URL gets enough views and 75% of them ar epositive, it should appear in the console as 1 good URL. And I can do that 1 by 1 for each good URL, attract more traffic there at least to generate a proper report.

    • The same applies for all those failed validation reports. Basically there are URLs which fail validation, but I know they are GOOD URLs and SHOULD pass validation. But when I run the report, it says it takes 28 days, but after 28 days, the validation fails, even if it shold pass by all means. That is probably because while the validation on the issues was inprogress, there was not enough new traffic registered on those failed/with issues URLs, and the report fails again. So technically if you have 1 URL that is registered as bad. If you fix it and make it the best URL in the world, but you have 0 real-world views on it, then that issue will never be fixed, no matter how many times you run the report, as long as you don't have traffic on the URL.

Also, when you run pagespeed for the same link multiple times you get very different results in the lab data. Why the results can vary in lightshouse lab data:

  • Server response time (depending on traffic, load balance etc.), the actual server that serves the page can respond faster or slower. The same applies for the CDN that serves the images and/or static files, or an external party that serves a resource (a script such as for youtube, or an image such as gravatar).

  • Depending on how lighthouse engine actually interprets the data, sometimes a slight randomness in the code, or variablity in timings may generate different values for things such as LCP, FCP, TTI etc. Sometimes the differences may be drastic if that small tipping point make lighthouse slide between 2 radically different interpretations of LCP for example.

  • I guess lighouse is run server-side on a VPS or smth similar and the characteristics of the VPS, the load, the hardware, the software installed etc. might actually generate variations in the browser redndering times during the analysis.

  • The server-response time is also impacted by the actual

The same causes for variations apply to the real-world data as well with some differences.

  • Realworld data does not use lighthouse, but is influencted by the client hardware and software on the client machine, as well as the type of browser used, plugins installed etc. This is much more drastic than lighthouse since, lighhouse uses a fixed configuration every time, but the devices, OSs and browsers of clients vary way more.

  • Also real-world data is only gathered from users that allow the permission to gather usage statistics.

  • Realworld data is impacted by client Internet speed as well, probably more drastically than lab data. Countries will bad internet connections would probably mess up your page experience scores.

So yeah, I tried my best to sum up all that I think I understand right now, some the the things are facts, other things are educated guesses. I would appreciate any of you can 100% confirm or deny any of the info above.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.