3

Web App Google Search Console Index: Crossover

I have a web app with a lot of dynamically generated pages. Each page has similar html layout with different text/image content, as well as a unique url. Recently the Google Search Console Indexing chart looks like this:

Google Indexing Report

Here is a chart under "Full report"

[Google Search Console Full Report](https://i.stack.imgur.com/YPXhp.png)

The Problem:

  • My "discovered" pages steadily increases while the "indexed" drops off with "not indexed" increasing.
  • I also have more "discover - able" pages but Google seems to have slowed down its crawling discovery rate.

I have tried :

  • reuploading an updated site-map.

  • I have also asked Google to re-index some of the Parent pages to these pages.

Notes

  • 12,478 instances of "Crawled - currently not indexed". After "URL inspecting" some of the instances, they actually are indexed properly.
  • I cannot ask google to index each page individually because of the sheer volume.
  • No major changes have been made to the web structure between 1/31/24 and 2/12/24

1 Answer 1

4

Google's search console is not a good SERP tracker, which is why other trackers exist. It also looks like they have a math problem, 12.4K vs 12.2K. It is unusual that Google will crawl 1000s of pages from a new site. If the site is new then it may have had a prior owner who had a large site, but don't expect that to carry over to a new site with different content.

I would look at technical issues, can you see Googlebot in the log files? Do the web log files show error messages being returned to Googlebot? Does it look like Googlebot has found an infinite loop in the dynamic content and is reading essentially the same page but with a different query string? Using up all of the budget for crawling - ranking.

Useful at this point to consider that crawling, ranking, and servering are unique and different functions.

A loop in crawling will keep sending different URLs but the same content to the ranking system and the ranking system will keep starting over never getting the chance to completely rank the content on the page.

Resulting in nothing getting into the index to be served.

You should have click-through rates being reported for the first 21 days the content was being served. Are those rates above 1%, 142 clicks per day? 2%? 5%? .1% ???

If you are not seeing click-throughs the ranking and serving may be below the Helpful Content Qualifications. Part of the ranking is user interactions but 21 days is a very short period to determine the interactions with content ... It can take months for interactions to show a benefit in search result positions, and months for a new site to get established.

It would not be unexpected for a new site to take months to rank with 1000s of pages. It would be a surprise or even a hack if it did not.

If you have query data, are the pages cannibalizing each other for the same keywords? Google search is not going to allow a flood of pages from the same site for the same query terms. Even the New York Times needs to not cannibalize it own search terms. Nor will they burn GPU time to select the very best page for the term for a site with 100s of pages trying from the same site.

Google recognizes that people prefer new pages over pages years old. So for the New York Times, when a new page cannibalizes an old page the old page falls away and no longer receives traffic from Google. Likely falling into the "Crawled - currently not indexed," bucket at some point when people are no longer looking for the page with long tail searches or are no longer talking about the page on other sites.

Echoing Google's statement they don't need to index a page for every model of cell phone a store has ... the important page is they have cell phones.

The pages that failed to get traffic may be cannibalizing the same keyword terms ... the Parent pages are the ones that need to be indexed. The anchor text on them contains the important keywords to the subpages allowing the parent page to come up in long tail searches.

What I would consider doing is to use .htaccess to add the no-index tag to the non-parent pages for a period of time to allow the parents to get indexed. The no-index meta tag is not a "forever," tag. I've used it on pages that needed to be updated and removed it after updating and google indexes the pages again almost immediately after I give Google permission to index.

Dynamically served pages do not provide published date details to Google. If cannibalization is taking place with keywords; Google short of burning GPU time does not have an easy way to determine which pages are the most relevant.

If the content is machine-created, Google is boastful and proud to say there are trillions of pages they know about but only 400 billion are served. For 2023 they are boasting for having removed over 170m fake reviews with their new algorithm. How many of those "fake" reviews were legit who used a fake review as a template? Poor souls? They consider their customer the people who search the internet, not websites; They will use GPU time for their customer's benefit, not websites.

Was the crossover Feb the 8th? That was the date of a minor update I'm afraid to say. But check the log files for a technical issue, don't assume a bigger problem. If the date was the 6th or 10th, it was not an algo adjustment.

enter image description here

1
  • Hi Wayne, thank you for your in depth reply. The URL query is sure to always be unique as it is created using Google reverse geocode tool. For example: a URL would end with ".../123-main-street,-city,-state,-ZIP". As of this time, there are no meta tag keywords being generated, just content unique to the URL. Average CTR is 1.4% taking a steep dive around the same time the crossover occurs, Feb 6. Average Position is 9.5, Total impressions at 14.4K with 197 Total clicks. Again all of my links will have unique information generated to the specific property URL. there are about 10 parent pages. Feb 20 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.