After a few months time and effort, I have managed to work around Google's canonical page selection problems. I am posting a step-by-step procedure here. It's not obvious, not published by Google, and not a trivial effort, but it does seem to be working, at least for our site.
Please note that our site runs a page generation script. Our URLs contain a Perl script that generates the visible page, for instance https://www.example.com/index.pl?page=directcore, where text following "page=" is a URL parameter 1. The basic concepts here probably apply to PHP page gen scripts also, and it may be they apply for fixed URLs also -- or it may not be. Since Google's algorithms are complex and closed it's always hard to reach a conclusion without 1000s of hours of testing.
Steps
1) First and foremost, the script parameter that controls page selection (for our site, this is the text following "page=") should be long and descriptive, and contain actual words in the dictionary, nouns, or names. Abbreviations and short-hands, and even short names that are actual words, seem to be interpreted by Google's anti e-commerce algorithms as bot-generated, and thus likely to be associated with duplicate content.
2) Second, related to 1), in your Google Search Console (GSC), set "URL Parameters" for your script parameter as follows:
Q: "Does this parameter change page content seen by the user?"
A: "Yes, changes, reorders, or narrows page content"
Q: "How does this parameter affect page content?"
A: "Specifies"
Q: "Which URLs with this parameter should Googlebot crawl?"
A: "Every URL"
Below is a screenshot showing an example.
3) Also related to 1), do not leave any short page names as active, or hanging around that Google can find, for example if you have old pages or are moving from http to https. Clean up any shorthand script parameters.
4) When you remove a page, make sure Google sees it as a "hard 404". It's not enough to return a soft 404. For page generation scripts, this might be an issue, depending on the programming of the script.
5) Any time Google mis-identifies your correct (target) canonical page, immediately attack the problem as follows:
- Remove the name Google chose wrongly. If you need to keep that page
create a new name for it (long and descriptive, as noted above), and
immediately re-index the new name. If you are using sitemaps, don't forget to remove the old name and add the new one. Basically, you want to leave no trail of the old name
- Attempt to re-index the old name, and verify that GSC rejects the request, seeing it now as a hard 404
- Re-index the target page
- If the old name was widely linked externally, use .htaccess rewrites to preserve it, but only do this once GSC shows the old name is no longer indexed. This may take a few days
What you may find is that fairly soon GSC will still show the target page as not indexed, but now the canonical URL will show as "N/A". That's good. Any time you force Google's anti e-commerce algorithms to recalculate, that's progress. As soon as you see N/A, re-index both pages again. Google might then latch on to another wrong page; if so keep repeating this cycle. It may take some time.
6) Constantly monitor around 20 or so of your key pages in GSC; try to choose these pages as a general, broad representation of your site (i.e some of A, some of B, etc). At any time, Google might again mis-identify a canonical due to periodic crawls, and you're back in the meat grinder again. Patience is required.
7) Things I found that did not help: submitting/removing sitemaps, temporarily removing URLs ("Remove URLs" in GSC), no-index tags, and changing page content.
Analysis
I can only guess at what Google is actually doing. Based on the evidence I've collected, it seems in general they are suspicious of page generation scripts, and employ hidden algorithms and decisions against such scripts, as they can be weaponized by e-commerce. But as we know, page generation scripts are used for legitimate reasons, including:
- They avoid extensive "dynamic content" to generate the page, which usually amounts to extensive reliance on Java script. Unfortunately Java script is increasingly seen as a security risk and an increasing number of users de-activate it
- They are widely used by small businesses and organizations that cannot afford dozens of Java script programmers and IT personnel to maintain highly complex web sites with 100s of pages of code (i.e. code vs. content)
This may also be true for fixed URLs, if they contain abbreviations or shorthands, for example for a site with many such pages Google's algorithms might decide such URLs are bot-generated for duplicate content and "rankings spam" purposes. But that is just a guess, I did not find evidence of that as our site is not organized in that way.
The latter point in 7) above is worth emphasizing: actual content is not what Google is using when it makes canonical mistakes. You can change text and images all day long and Google will still incorrectly select canonical pages, if you have otherwise triggered their anti e-commerce thresholds.
1 We use .htaccess rewrites to allow shortened URLs, for example if you enter https://example.com/directcore, our web server rewrites to https://www.example.com/index.pl?page=directcore. However, we never submit the shortened URLs to Google for indexing, as they would be rejected as "re-directs"
