0

I was wonrdering why on google search console, 200,000+ of my pages are not indexed (it's not because of robots.txt or volontary excluding them):

+------------------------------------+---------+
| Exclusion reason                   | Number  |
+------------------------------------+---------+
| Discovered - currently not indexed | 119,000 |
+------------------------------------+---------+
| Crawled - currently not indexed    | 101,000 |
+------------------------------------+---------+

enter image description here

This google article says :

Crawled - currently not indexed: The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.

Discovered - currently not indexed: The page was found by Google, but not crawled yet. Typically, Google tried to crawl the URL but the site was overloaded; therefore Google had to reschedule the crawl. This is why the last crawl date is empty on the report.

Some examples of URL's:

Discovered - currently not indexed:

Crawled - currently not indexed:

--

Those page I gave are all user generated and exist in multiple language (the user content is not translated, but the website interface is)

  1. Is it because of my canonical markup ? All website exist in 2 languages, and each page has a canonical tag to itself (ex: /fr/example points to /fr/example, and /en/example points to /en/example.

  2. Could it be because of some technical reason ? (bad use of canonical ? bas use of hreflang ?

  • Do you think you might be able to tell me where you got the data from in your image screenshot? It's really interesting and I'd like to be able to view that for my sites as well. Is that in search console? Because I haven't seen a tab with a chart likes yours – Michael d Feb 26 '18 at 13:43
  • It is the beta search console from google search.google.com/search-console – Tristan Feb 26 '18 at 14:00
1

I was running a site last year that had about 30 million pages indexed with Google. I found that Google would crawl many pages but choose not to index a large percentage of them. I felt that many of my best pages that were crawled weren't indexed while pages with far weaker content were. Ultimately I concluded that Google chooses which pages to index and which pages not to and I had very little control over the selection process. Even proper internal linking with many links pointing to a specific page didn't mean it would get indexed. Pages with very few, in some cases even zero internal links were often indexed.

Google may discover links on your site but choose not to crawl those pages. And it may crawl pages and choose not to index them even if they are high quality pages.

I wish there was a simple fix for this issue to help you. But in the end I think you'll just have to cross your fingers and hope the pages you want indexed end up there.

If you can increase the links pointing to that page from external domains this will send a huge signal to Google that it needs to index them because those pages will have larger page authority than other pages on your site.

If you think there could be errors in the crawling process of these pages you should always try fetch and render with Googlebot. Enter some of the URLs into the following link and click fetch and render, then click on the result after Google has crawled it. Google will show you a screenshot of the page it has crawled which will let you know if it sees all of your content or if there are any issues. If it sees all of your content and still chooses not to index it then it's simply an algorithmic thing on Google's end and there is only so much you can do to improve your index status.

https://www.google.com/webmasters/tools/googlebot-fetch?hl=en&siteUrl=

This is a screenshot of search console for one of my sites. Google has chosen to exclude many of my URLs as well. enter image description here

Not the answer you're looking for? Browse other questions tagged or ask your own question.