What can prevent Google from Indexing Pages?
- Low quality (low interaction, thin or duplicate)
- Incorrect header responses
- Blocked resources
It's about the quality of articles
While an article made up of 250-500 unique words may not be treated as Google as 'thin' doesn't necessary mean the content is high quality. It used to be that size matters but now SEO has shifted and is more about quality than ever. I very much doubt you have 5 million quality articles!
What is a quality page or article?
A quality page is not one that is unique but one that people expect to find and want to see, a page that receives little, or no interaction is a page that will likely be poorly ranked. Sites that have an insane amount of content with little natural interaction makes Google believe that the site is not important and will allocate less time for crawling per a visit.
It's about quality not quantity
Nowadays it is considered better SEO to publish less often and with higher quality content, the type of content people want to see, as interaction is key for rankings, authority and crawl time allocate.
Crawl time allocation
If you bothered to read the previous paragraphs you will notice that I've mentioned crawl time allocation, this is key... and most likely the cause of your index issues.
You have an insane amount of pages, Google has limited resources and it will only ever crawl a site for a certain amount of time before it stops and goes to another site. The allocate time that each site is allocated varies depending on how important Google believes your site is, so the chances. You also need to factor in the 'return' rate, of which Google's bots decide to revisit your site between visits.
What effects Google's Crawl Time and how often Googlebot will return to the site
Domain and site authority is a huge factor for Google in which it decides how often it will return to your site and the duration it does, to put this into respective a site like Stack Overflow will have Googlebot pretty much visiting every few minutes a day, maybe even sooner, but a site with just as much content with low interaction and low authority will be at best a few times a day.
Ensuring Googlebot can crawl as much as possible every visit
- Website Speed
- Not only does Google reward SEO value for sites that are fast you can also have more pages crawled between visits. Use website speed tests from multiple locations, and ensure that your website responds fast for both Google and your main target region. I recommend WebpageTest, aim for below 1.5seconds on first visit as a good guide.
- Server Uptime Availability
- If your DNS or server that fails to respond for just a few seconds each day can mean that you missed your return crawl, so its important to monitor your website and ensure that its performing well, Pingdom, and other providers can provide this service for you.
- Ensure that you have a good robots.txt, most sites don't need to worry about robots.txt since they only have a low volume of URLS, but sites with high volume and low authority, then the time spent on your site is critical and having the Googlebot, crawl, not index things like login pages, and pages with noindex will only prevent Google from spending more on pages it can index. Use robots in conjunction with noindex header responses.
- Clean up 404's with 301 or 410's
- Google loves to crawl, and re-crawl pages that return a 404 status, for most webmasters this is not an issue. However since you have a high volume of pages, it falls back on time is critical. Ensure that your 404's redirect to pages that are on topic, or they return the 410 Gone status. Google will learn by this and should stop attempting to crawl those pages, giving you more index time on pages not crawled.
- Remove duplicate pages and avoid canonicals
- Nowadays most SEO savvy webmasters will use canonical links to avoid duplication, this for most webmasters is AWESOME! it let Google know what is duplicate, and what is not, but the major problem with canonicals is that they too are a page, and that page needs to be crawled, if you have a page that can be accessed via www, non-www, tag pages and all other type of pages then your simply losing crawl time that could be spent on discovering new pages. So something to bare in mind.
- Compile pages
- If you have a high volume of pages then its extremely likely that you have similar pages, or pages that can be merged. Google loves long pages! so do users, if you have pages 1 of 5, merge them, if you have relevant pages to one another, merge them.
- Crawl errors
- Actively monitor your crawl errors, these use crawl time up and you need to keep on track.
- Tracking Google
- Record when Google visits your site and for how often, keep a track on it and see if you can improve it, doing the above will certainly help.