I am reaching out for some guidance regarding the Google Search Console (GSC) indexing of my website. I launched this site about a month ago, and I'm encountering some challenges with Google's indexing process.


Nature of Website: My website connects to a database created by a scraper that collects public company information, making it easier for users to search through these listings.

Technical Specs:

Scraper: Built in Python

Database: SQL

CMS: WordPress

Total Listings: Approximately 570,000 company listings/pages

Sitemap Structure: Sitemap contaning 570 sitemaps, each containing 1,000 pages

Current Indexing Status: Google Indexing Rate Started with 13 pages every 3 days, gradually increased to 100 pages every 3 days GSC Sitemap Recognition: Only 11,000 pages are recognized out of 570,000

Challenges and Queries:

Indexing Rate: The indexing started slowly and has only marginally improved. Given the size of my website, is this a normal progression, or could there be technical issues hindering the process?

Time Factor: Should I simply wait for Google to increase indexing rate or are there strategies I could implement to expedite this process?

GSC Sitemap Limitation: Why does GSC recognize only 11,000 out of the 570,000 pages listed in my sitemaps? Could this indicate a potential issue with how my sitemaps are structured or submitted?

The site is 100% legitimate and not a spam site. It's designed to provide a valuable resource by enhancing the accessibility of public company information.

I would greatly appreciate any insights, suggestions, or experiences you can share, especially if you have dealt with similar challenges in indexing large websites.

Thank you in advance for your time and assistance!

  • 3
  • 1
    If it's a new website, then you have to be patient as well, because GSE doesn't trust it as much. Nov 16, 2023 at 22:17
  • "enhancing the accessibility of public company information" in other words, your site has nothing but content that is copied from elsewhere but you are trying to make it easier to find and view. That often isn't enough for Google. If Google is able to index this content elsewhere already, your page with that content probably won't get indexed. Nov 17, 2023 at 14:35


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.