fixing Search Console index problems after submitting no subdomain or extension in canonical URLs and sitemap

Question

I prefer clean URLs... so my internal links lack the .html or .php extension, and I'm omitting the www. subdomain. The links are resolved by .htaccess rules, and the browser shows example.com/my-page. This makes me smile.

For consistency, I entered the clean URLs as canonical links and in the sitemap (as https://example.com/my-page) that I submitted to Google. My hope was to see these clean URLs used everywhere consistently.

Unfortunately, I now see a mess in Search Console. Some indexed pages still have .html. Some still have www. Some pages were not indexed.

I'm attempting to clean up the mess by resubmitting individual URLs, but I have dozens to fix. It seems each link has to be submitted separately. It takes about 30 seconds to submit each one, and I hit my submission cap each day. This will take forever.

There must be a better way to clean up the mess. Any suggestions to make it easier?

Also, did I make the wrong choice with the canonicals and sitemap? If so, what should I have done?

Here's what I have in .htaccess:

Options +FollowSymLinks
RewriteEngine on

RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www\. [NC]
RewriteCond %{HTTP_HOST} ^(www\.)?(.+)$
RewriteRule (.*) https://%2%{REQUEST_URI} [NE,L,R=301]

# if x.php is a file, add .php to x
RewriteCond %{REQUEST_FILENAME}\.php -f
RewriteRule !.*\.php$ %{REQUEST_URI}.php [NC,QSA,L]

# if x.html is a file, add .html to x
RewriteCond %{REQUEST_FILENAME}\.html -f
RewriteRule !.*\\.html$ %{REQUEST_URI}.html [NC,QSA,L]

# if xindex.html is a file, add index.html to x
RewriteCond %{REQUEST_FILENAME}\index.html -f
RewriteRule !.*index\.html$ %{REQUEST_URI}index.html [NC,QSA,L]

Answer 1

Here is the solution I came up with after researching for many hours...

Options +FollowSymLinks -MultiViews
RewriteEngine On
RewriteBase /

RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www\. [NC]
RewriteCond %{HTTP_HOST} ^(www\.)?(.+)$
RewriteRule (.*) https://%2%{REQUEST_URI} [R=301,NE,QSA]

# 301 external redirect all .php extensioned requests to extensionless
RewriteCond %{THE_REQUEST} ^[A-Z]{3,}\s([^.]+)\.php [NC]
RewriteRule ^ %1 [R=301,NC,NE,QSA,L]

# 301 external redirect all .html extensioned requests to extensionless
RewriteCond %{THE_REQUEST} ^[A-Z]{3,}\s([^.]+)\.html [NC]
RewriteRule ^ %1 [R=301,NC,NE,QSA,L]

# if x.php is a file, internally add .php to x
RewriteCond %{REQUEST_FILENAME}\.php -f
RewriteRule !.*\.php$ %{REQUEST_URI}.php [NC,QSA,L]

# if x.html is a file, internally add .html to x
RewriteCond %{REQUEST_FILENAME}\.html -f
RewriteRule !.*\.html$ %{REQUEST_URI}.html [NC,QSA,L]

# if xindex.html is a file, internally add index.html to x
RewriteCond %{REQUEST_FILENAME}\index.html -f
RewriteRule !.*index.html$ %{REQUEST_URI}index.html [NC,QSA,L]

Answer 2

Best solution: 301 redirect

Whenever a user or bot opens https://www.example.com/my-page or https://example.com/my-page.html or https://example.com/my-page.php or https://www.example.com/my-page.html or https://www. example.com/my-page.php they should be redirected to https://example.com/my-page

OKish solution: canonical tags Make sure that whatever version of https://example.com/my-page is opened (with/without www, with/without .html/.php) there is always a self-referencing canonical tag, pointing to https://example.com/my-page

Hygiene

• Make sure all internal links are pointing to the version you want.
• Make sure the sitemaps only contains links to the version you want.
• As far as possible, make sure backlinks point to the version you want.