1

This page addresses the question of duplicate content, but does not describe what I'm facing. After first encountering the Google "canonical algorithm bug" 2 months ago, I've made only a little progress. I was able to get Google to stop selecting old pages as canonical for important new pages. Unfortunately, now the issue is Google can't tell the new pages apart. Here are 2 examples (replace example with signalogic):

https://www.example.com/index.pl?page=directcore

https://www.example.com/index.pl?page=codec_samples

Inspecting codec_samples in GSC shows directcore as "Google-selected canonical":

enter image description here

Of course this blocks the codec_samples page, which used to bring in 50+ clicks a day. Now I cannot get the page indexed (have verified that with a site search).

According to webconfs "similar-page-checker", these pages are 5% similar. Obviously they look different. Neither links to the other. What else is Google paying attention to ? What can I do to get Google to see the "codec_samples" page as unique ?

Note about GSC URL parameters settings: I have it set to index "Every URL" and the subcategory to "Specifies".

  • 1
    This looks like the same question you already asked: Google selects one unrelated page as Canonical for many other pages, blocking our pages from indexing How is this different? – Stephen Ostermiller Jun 13 at 12:28
  • The other question was one important page vs. misc old pages. That problem had to do with the way Google is handling URL parameters and some things they are doing vs. e-commerce, and is now resolved, thanks to a friend of a friend who works in Google. This case is unique, fully qualified content vs. content, on the same website. I have several cases of these, but this is a good example because the codec_samples page is a resource popular with universities, government agencies, etc for which there would seem to be no fundamental reason for Google to block. – Jeff Brower Jun 13 at 16:51
1

After a few months time and effort, I have managed to work around Google's canonical page selection problems. I am posting a step-by-step procedure here. It's not obvious, not published by Google, and not a trivial effort, but it does seem to be working, at least for our site.

Please note that our site runs a page generation script. Our URLs contain a Perl script that generates the visible page, for instance https://www.example.com/index.pl?page=directcore, where text following "page=" is a URL parameter 1. The basic concepts here probably apply to PHP page gen scripts also, and it may be they apply for fixed URLs also -- or it may not be. Since Google's algorithms are complex and closed it's always hard to reach a conclusion without 1000s of hours of testing.

Steps

1) First and foremost, the script parameter that controls page selection (for our site, this is the text following "page=") should be long and descriptive, and contain actual words in the dictionary, nouns, or names. Abbreviations and short-hands, and even short names that are actual words, seem to be interpreted by Google's anti e-commerce algorithms as bot-generated, and thus likely to be associated with duplicate content.

2) Second, related to 1), in your Google Search Console (GSC), set "URL Parameters" for your script parameter as follows:

Q: "Does this parameter change page content seen by the user?"
A: "Yes, changes, reorders, or narrows page content"

Q: "How does this parameter affect page content?"
A: "Specifies"

Q: "Which URLs with this parameter should Googlebot crawl?"
A: "Every URL"

Below is a screenshot showing an example. enter image description here

3) Also related to 1), do not leave any short page names as active, or hanging around that Google can find, for example if you have old pages or are moving from http to https. Clean up any shorthand script parameters.

4) When you remove a page, make sure Google sees it as a "hard 404". It's not enough to return a soft 404. For page generation scripts, this might be an issue, depending on the programming of the script.

5) Any time Google mis-identifies your correct (target) canonical page, immediately attack the problem as follows:

  • Remove the name Google chose wrongly. If you need to keep that page create a new name for it (long and descriptive, as noted above), and immediately re-index the new name. If you are using sitemaps, don't forget to remove the old name and add the new one. Basically, you want to leave no trail of the old name
  • Attempt to re-index the old name, and verify that GSC rejects the request, seeing it now as a hard 404
  • Re-index the target page
  • If the old name was widely linked externally, use .htaccess rewrites to preserve it, but only do this once GSC shows the old name is no longer indexed. This may take a few days

What you may find is that fairly soon GSC will still show the target page as not indexed, but now the canonical URL will show as "N/A". That's good. Any time you force Google's anti e-commerce algorithms to recalculate, that's progress. As soon as you see N/A, re-index both pages again. Google might then latch on to another wrong page; if so keep repeating this cycle. It may take some time.

6) Constantly monitor around 20 or so of your key pages in GSC; try to choose these pages as a general, broad representation of your site (i.e some of A, some of B, etc). At any time, Google might again mis-identify a canonical due to periodic crawls, and you're back in the meat grinder again. Patience is required.

7) Things I found that did not help: submitting/removing sitemaps, temporarily removing URLs ("Remove URLs" in GSC), no-index tags, and changing page content.

Analysis

I can only guess at what Google is actually doing. Based on the evidence I've collected, it seems in general they are suspicious of page generation scripts, and employ hidden algorithms and decisions against such scripts, as they can be weaponized by e-commerce. But as we know, page generation scripts are used for legitimate reasons, including:

  • They avoid extensive "dynamic content" to generate the page, which usually amounts to extensive reliance on Java script. Unfortunately Java script is increasingly seen as a security risk and an increasing number of users de-activate it
  • They are widely used by small businesses and organizations that cannot afford dozens of Java script programmers and IT personnel to maintain highly complex web sites with 100s of pages of code (i.e. code vs. content)

This may also be true for fixed URLs, if they contain abbreviations or shorthands, for example for a site with many such pages Google's algorithms might decide such URLs are bot-generated for duplicate content and "rankings spam" purposes. But that is just a guess, I did not find evidence of that as our site is not organized in that way.

The latter point in 7) above is worth emphasizing: actual content is not what Google is using when it makes canonical mistakes. You can change text and images all day long and Google will still incorrectly select canonical pages, if you have otherwise triggered their anti e-commerce thresholds.

1 We use .htaccess rewrites to allow shortened URLs, for example if you enter https://example.com/directcore, our web server rewrites to https://www.example.com/index.pl?page=directcore. However, we never submit the shortened URLs to Google for indexing, as they would be rejected as "re-directs"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.