4

When a page has content on it that is also on other pages on the site but also some unique content, how does Google handle it? Does Google:

  • Index the entire page (including the duplicate content)
  • Index just the unique text on the page
  • Index none of the page (not even the unique content)

For clarification, I'm only talking about content duplicated within a website, not content copied from other sites.

I ask because I have answered several questions here assuming that Google will index unique content even when it has duplicate content near it in the same page. However, I realized I don't have any evidence that this is actually true.

This is a duplicate content scenario that is not addressed by our catch all question on duplicate content: What is duplicate content and how can I avoid being penalized for it on my site?

  • G would have to index something to know to ignore it which is part of the point. There may be three ways to look at this: one, widely repeated (templated) content; two not so widely repeated content (occasional); three, not repeated at all. Templated content is largely ignored by marking it. Duplicate content that is only repeated a few times across the site would have full benefit though may see a variable downgrade if repeated too much. There is not likely one place where anyone has written about this if that is what you are looking for. – closetnoc Jan 19 '16 at 21:39
  • It would be interesting to know how Google deals with boilerplate on every page too. I know that some boilerplate doesn't prevent Google from indexing all your pages, but I don't know if searching for a snippet from the boilerplate along with unique content will bring up any results. – Stephen Ostermiller Jan 19 '16 at 21:41
  • I'm mostly asking about content in a few places on the site. Such as products listed under different filters. Will using duplicate content on a page with some amount of unique content prevent that unique content from getting indexed? – Stephen Ostermiller Jan 19 '16 at 21:42
  • Boilerplate created content has taken a hit lately and how that has manifested is too early to tell exactly. However, for a few paragraphs here and there, it is weighted the same anywhere it appears. I do believe there is a downward value that can be applied for sections that are seen as being repeated too much and hence how and why boilerplate seems to be suffering with the new updates. I believe that there is new boilerplate detection alog(s) that can slice and dice individual sections/sentences looking for for variable based automation causing havoc for some. – closetnoc Jan 19 '16 at 21:52
  • Let's say you have a site with quite a few pages that are completely or nearly completely duplicated. Let's say the site is not so big that metrics are rather clear. You will see all pages indexed in Googles Search Console, however, doing a site: search will show less pages. Duplicate pages appear to be handled in the SERP ranking/filter phase as part of a search query from where I stand. This is evidenced with the recent template content algo(s) where some sites are being hit these days. – closetnoc Jan 19 '16 at 22:55
2

Just about every website you visit will at least have a certain percentage of duplicate content. A perfect example of this is a logo specific to the website that appears on all content pages to indicate that the content is part of the website itself. This kind of thing is something google will index in its entirety (provided of course the rest of the content is original and unique to the site and not copied verbatim from another site). Heck, if google did not index this, then thousands of legit online companies will make mass complaints about their site not being indexed.

If on the other hand you have pages that mostly consist of duplicate content where the difference between each page is less than a few words of text, then google could very well treat that as duplicate content and will decide which one of the duplicate pages to index if any.

What I would suggest is to try to make the duplicate level between two pages to less than 60% (ideally), or at least under 80% at bare minimum.

Using tools such as the one found here: http://www.webconfs.com/similar-page-checker.php can give you an idea how similar two pages are. Never aim for 100% with this tool.

  • 1
    When you say "less than 60% (ideally), or at least under 80% at bare minimum" how did you decide that? Are you aware of SEO experiments, or any documentation from Google that support that, or is that just a hunch? – Stephen Ostermiller Jan 20 '16 at 10:48
2

I assume, Google decides about indexing through measuring of duplication (or similarity) rate of certain page on URL basis and indexes all pages, containing less then 100% (or 90%, or X% - only Google knows exactly the number) duplicate (if nothing, like noindex, prevents it).

Finding duplicated content isn't a trivial task and is error-prone because of page chrome. That is why i think Google would index pretty all pages and kick out only doubtlessly duplicated pages.

An interesting thing is, that pages having some internal duplicated content (requirement again: less then 100%) can cannibalize rankings of their internal competitors.

2

Okay. I will try and explain what I know the best that I can quickly. Perhaps just explaining some of this will make things clear.

In the early days of Google, a term index would be, in effect, a relational or leaf table that tied terms in a term index (forward and reversed) to a document using a docID and wordID with other metrics. Part of semantic tradition is to track the position of a term (word) in relation to points within the document. Google, when researched, only maintained a single position metric based upon the start of the document (0) in bytes. This did not include HTML markup of course, but in the early days included HTML header, footer, sidebar content etc.

In this way, Google would be able to look for patterns of terms in relationship with each other. This means that while a document did not have to completely duplicated, it was fairly easy to determine that a document was duplicate within a certain set of metric guidelines be it percentage, ratio, or whatever.

The problem with this method is that rearranging a document or using a spinner could easily defeat this.

Given that semantics is more involved than term relationships from a single point and the use of ontologies that relate similar terms, plural terms, etc., duplicate content was more easily found, though still not fully complete if taken in a relatively linear comparative model.

Enter the DOM.

Using the HTML DOM model, sections of repeated content can more easily be compared to extract templated sections such as headers, footers, sidebars, etc. This is a given these days since this has been in place for a long time with excellent results. Content is now the page content that people would recognize. These templated content sections are indexed of course (basing this on a Google flaw that evidenced this fact even in 2015), but largely ignored for search matches.

Okay, we understand this. But what about actual content?

The HTML DOM model is still used. For each content DOM element, largely header tags, paragraphs, tables, etc., each is semantically weighed using a variety of semantic algorithms some singular and some in combination to create a matrix which you can think of a spreadsheet/table of sorts. This lists each term with the algorithm weights. Since semantics is not a direct comparison of terms, meaning that car, automobile, vehicle, etc., are all the same, along with plural versions of these terms, etc., any algorithm can easily find content that has been spun, reorganized, etc. The key is that a matrix can cover varying sizes of content by overlapping several matrices into a matrix of matrix.

A matrix will represent content segments (as defined in semantics). This, for HTML, would be a header tag, the paragraphs found following the header ending at the next header taken as both singular paragraphs and as a group. A content segment can also be a singular sentence but we will get into this in a bit. Using term position from the beginning of a header, the beginning of a paragraph, the beginning of a group of paragraphs between header tags, etc., the original patterns of term relationships can still be used. But more importantly, within the matrices, patterns also can be seen quite easily. It does not take a rocket scientist to recognize them. The semantic scores give duplication away.

Knowing that a content segment is also as small as a singular sentence, there is something new going on. Content segments are also being looked at in new ways to recognize content that is being created using variables from a programming language. This is still rather easy to discover, though as of right now, I am still figuring this out. It is still semantics based, but how that varies may only mean a more granular semantic analysis. Be that as it may, header tags, paragraphs, and sentences are being analyzed beginning in 2015 for automated content creation that may otherwise escape other duplicate content analysis. The result of this analysis is penalizing sites as we speak.

Okay. Back to what duplication is effected.

The first thing to remember is that once Google fetches a page, the entire HTML code is stored for reference. This is used to build the cache of a page, but is really used to allow Google to go back and reapply new or updated analysis to content without re-fetching the page.

Obviously, HTML templated content is completely ignored when a search query is made though there are some extremely minor exceptions that seem to have escaped Google until recently. You will find that it is extremely rare that Google will match a search query to a header, footer, sidebar, etc. Good.

Google has stated that replicated portions of content are indexed and weighted normally assuming that spam is not an issue. This is because for most sites, it is nearly impossible to not replicate portions of one page on another for a site of a certain size or greater. As well, this would cover quoted sections of content as a citation. Still good.

Google, as stated, is looking at smaller content segments for variable based content creation. This is where it gets tricky and not all of this is figured out yet. If you were to look at some automated sites, some are being hit while others are not. Clearly, these sites are programmatically generated and extremely similar, but what is the difference? Looking at Whois sites as an example, it is still fuzzy. I believe that other factors that we all know come into play such as the velocity of page creation, link velocity, site and page authority as defined by linking patterns, social engagement, etc., continue to play a role but in a different way. So for a site with a good reputation and solid metrics will be forgiven if content is driven by filling in variables where others will be more strongly seen as spam if the metrics are poor. This means that the bar for content quality and value is more measured by users than the content itself thus raising the bar of acceptability. One savior from this effect is unique content. Is the site adding value that is significant over others? How this is measured is still unclear, however, it seems that for now, uniqueness of a portion of content within a field of comparable sites is a metric though likely less than the others listed above.

Clear as mud?? Did I do a good job here?

  • "Google has stated that replicated portions of content are indexed and weighted normally assuming that spam is not an issue. " -- do you have a reference? – Stephen Ostermiller Jan 20 '16 at 10:50
  • This is the first time I've heard that Google might analyze the markup structure to detect duplicate content. I've always heard that they use shingling algorithms to detect duplicate and near duplicate content: nlp.stanford.edu/IR-book/html/htmledition/… – Stephen Ostermiller Jan 20 '16 at 10:53
  • @StephenOstermiller folks at MOZ do chrome recognizing and content purifying: github.com/seomoz/dragnet I presume, Google can do and does the same, because it seems to be the main problem on comparison of website content - to clearly divide chrome (template etc.) from textual, meaningful content. – Evgeniy Jan 20 '16 at 11:56
  • Google has stated that replicated portions of content... I guess I could have been more clear on this... referring to a paragraph or two. I did see a statement somewhere, but mostly you can search for a quoted statement (typical in SEO sites) and find it everywhere. If you look harder, you can see more verbose quotations. That is what I was referring to. – closetnoc Jan 20 '16 at 13:54
  • Your stanford.edu link more closely refers to the first method I talked about. Analysis of the term index using the position of the term allowed the content to be rebuilt to an extent that this method could be used. I had not read that bit in a while. As well, I was not referring to HTML tags but actual content within the HTML tags as content segments as defined in semantics. This analysis allows detection of spun content rather easily and is used in some search applications these days to varying degrees. I guess I should keep track of this stuff better. – closetnoc Jan 20 '16 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.