Google recently cut my indexed pages in half, which cuts revenue in half, which is pretty concerning.

I have a large database driven website (millions of pages) which dynamically creates content with PHP based on MySQL content for that item. I was hoping to improve the user experience, ranking, and number of indexed pages.

So I created a large amount of unique written content for product categories and sub-categories, which includes images, charts, etc. This is printed at the bottom of each product's page that is in that category. The products name is inserted into the paragraph, so it's slightly unique but still, obvious template content.

It seems... as a result Google, about cut my indexed pages in half and moved them into the "Crawled - currently not indexed" category".

enter image description here

enter image description here

This is obviously surprising and frustrating to be penalized for the a lot of hard work, which I believed would benefit the user. There wasn't any other structuring changes made around this time. So unless it's random, it appears Google doesn't like this template content.

There are a few reasons listed for the pages being doomed to "Crawled - currently not indexed" status, the only one I can see that fits, is the "thin content" complaint. Now, of course, the pages don't have any less unique content. But it dawned on me that Google could be looking at the page content as a ratio/percent of unique content.

As the total content increases, the percent of unique content per page decreases. Perhaps this has soured the algorithmic view on my pages?

If these pages were made of complete duplicate content, I could see the problem, but does simply adding duplicate content from my own site create a problem?

I couldn't find any documentation on webmaster tools or good advice from SEO guru sites.

I would hate to go back and remove all the content since I think it adds real value to the user and took a lot of work.

So I'm not sure what to do...

If anyone has experience, research, or back-testing on an issue like this, I would deeply appreciate it. Thank you!

2 Answers 2


Interestingly, I have a similar situation blya with my auto portal and one article site, also on cars and there are millions of pages.

  1. Find a list of all pages with content less than 500 characters and close these pages with meta robots noindex, nofollow. This way, you will have a better user experience, but the number of pages that Google will spend its resources on will decrease, and the ratio of bad pages to good pages will also improve. This will make your website look more high-quality in the eyes of search engines.
  2. Check Last-Modified and 304 Not Modified settings for errors. This is necessary to optimize the crawling budget of search engines, and as a result, the indexing of new pages will happen much faster.
  3. You mentioned "products," and if I understood correctly, start generating reviews to increase the amount of content on the page and give it greater value in terms of uniqueness. There is a strong suspicion that you have cannibalization on the site, where 2 or more pages answer to 1 keyword.
  4. Check the optimization of the very last page of the product to ensure that the density of the lowest frequency keyword is the highest. This is in the case where you have 10 pages for the iPhone 14, only different modifications, but, for example, all are optimized specifically for the iPhone 14, not for color or memory volume.

I hope I've made myself clear.)

  • Thank you for your input, I appreciate it. I'll figure you out how to apply it to my site. But one suggestion you had, I'm not very familiar with it. When you say "Check Last-Modified and 304 Not Modified settings for errors." Is that is search console, the site map, etc... where or how do I check those? Thank you! Mar 12, 2023 at 2:15
  • On the last-modified (dot) com site, you insert each type of page on your site into a line and look at the dates and the correctness of their display.
    – SEO-in TOP
    Mar 12, 2023 at 5:01
  • thank you for the input Mar 13, 2023 at 20:13
  • No questions - contact me if you have any questions_)
    – SEO-in TOP
    Mar 14, 2023 at 4:43

If you have duplicate content, then Google will not index all pages. It will use its own algorithm to decide which pages are should be indexed. Changing a few words and inserting a name does not make the content unique.

And you are correct in that the larger the volume of duplicate content, the less unique that page is.

My suggestion is to have the duplicate content on a separate page initiated by a button. Or even have it pop up using ajax (or similar).

  • Thank you for your input, I appreciate it. So I have 2 questions: 1) If I load that content with Ajax, doesn't google now parse JavaScript and will still see that? 2) I'm still a little curious about why adding (same site) duplicate content would warrant de-indexing a page. For example, many sites need to add disclaimers, or general company info at in the footer. Are they all hurting their rankings by doing so? I'm not saying I'm disagreeing with you. But I am curious for the basis of that view point, for example, can you link any guidance from by Google, back testing, studies, etc Mar 12, 2023 at 2:34
  • Headers, Footnotes and Menus are small in comparison to the content. And they have appropriate tags such as <header>, <nav>,<footer>. Yes, Googlebot executes javascript, but you have control of the parameters it uses. You can set the parameter in the Google Webmaster Tools or use the canonical to specify the parameter. Using this, you can get the ajax call to return a shorter version for the bot. Mar 12, 2023 at 10:29
  • thank you for your input Mar 13, 2023 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.