How to find duplicate content within a website, not just titles and meta descriptions?

Question

I am working on a website that has thousand of pages. One of my main concern is that the previous webmasters used the same text blocks on various pages. Thus, even though most of the pages are different, the because of that useless text the content will me 50% duplicate to all the other pages in that category.

I will be noindex-ing useless or thin pages, create unique titles and meta descriptions which will help. But, as I have said I think that several blocks of the same text was used on multiple pages. Is there a way to check the website for this? Perhaps a tool or a crawler that will notify me in case it finds duplicate content on the website's pages? I cannot manually visit every page because I am talking about thousand of pages.

Please note that I am talking about CONTENT and not just titles and meta descriptions. I am talking about the same block of text/description/story that has been put on 100 different pages.

Any idea what to do in such a case?

Answer 1

You don't have to worry much about content that is duplicated within your own website. When Google detects duplicated content within a website it simply chooses one of the duplicated pages to index and ignores the rest. It doesn't index them, but it also doesn't penalize the site in any way.

If there is also unique content on those pages, Google will usually index the unique content and ignore the duplicate content.

The only time that duplicate content within your own site is a problem is when there is so much of it that Googlebot has trouble crawling the whole site.

See What is duplicate content and how can I avoid being penalized for it on my site?