Multiple forms of index page being indexed by search engines

Question

I noticed Search Engines were flagging my index or home page with duplicate content/title/description, etc. Basically, it was indexing:

http://www.example.com
https://www.example.com
http://www.example.com/
https://www.example.com/

I think I solved one problem with using canonicalization like this:

<link rel="canonical" href="https://www.example.com/" />

But it still shows me there is a duplicate between these two URLs:

http://www.example.com/
https://www.example.com/

Strangely, I specifically canonicaled that I wanted

https://www.example.com/

How can I stop the Search Engines from indexing all of these variations so they see them all as one domain? I could add them to the robots.txt file but I fear it would stop indexing the site altogether.

Answer 1

The correct way to favor https over http was fixed by modifying the htaccess file with the following lines

<IfModule mod_rewrite.c>
    RewriteEngine On

    RewriteCond %{SERVER_PORT} !^443$
    RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301,NE]
</IfModule>

I downloaded Screaming Frog to crawl the site to see exactly what the search engines are seeing. I realized that both versions were crawled but the http version showed that it was 301ed. Which it's exactly what search engines need to see. However, I realized that my pages were being duplicated within each version without a 301 Status (only the home path was). After some researching, I found out that a directive to a 404 Error page fixes that. So I modified the htaccess once again and this is the final code

<IfModule mod_rewrite.c>
    ErrorDocument 404 /404.php
    RewriteEngine On

    # It favors the www version without a canonical tag
    RewriteCond %{HTTP_HOST} !^www\.example\.com$ [NC]
    RewriteRule ^ https://www.example.com%{REQUEST_URI} [L,R=301,NE]

    # It favors the https version of URI
    RewriteCond %{SERVER_PORT} !^443$
    RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301,NE]

    # To externally redirect /dir/file.php to /dir/file
    RewriteCond %{THE_REQUEST} \s/+(.+?)\.php[\s?] [NC]
    RewriteRule ^ /%1 [R=301,NE,L]

    RewriteCond %{REQUEST_FILENAME}.php -f
    RewriteRule ^(.+?)/?$ $1.php [L]
</IfModule>

See the original question in my StackOverflow here (https://stackoverflow.com/questions/51504638/removing-the-php-file-extension-from-the-url-request)