1

I noticed Search Engines were flagging my index or home page with duplicate content/title/description, etc. Basically, it was indexing:

http://www.example.com
https://www.example.com
http://www.example.com/
https://www.example.com/

I think I solved one problem with using canonicalization like this:

<link rel="canonical" href="https://www.example.com/" />

But it still shows me there is a duplicate between these two URLs:

http://www.example.com/
https://www.example.com/

Strangely, I specifically canonicaled that I wanted

https://www.example.com/

How can I stop the Search Engines from indexing all of these variations so they see them all as one domain? I could add them to the robots.txt file but I fear it would stop indexing the site altogether.

  • It meant to have https in it. But I fixed it using htaccess. I haven't had a chance to update my question with an answer and proof of crawl – LOTUSMS Jul 24 '18 at 21:53
  • I edited your post to fix the example and both make it conforming to RFC2606. You can click on Edit to edit your own question and add any relevant details as necessary. If you found a solution yourself, you should post it as a solution (do not put it in the question) and then just accept your own answer if it fits so that people see that the question have been answered. Leave the question open only if you would like other people to give other answers. – Patrick Mevzek Jul 24 '18 at 22:11
  • There is no difference between with a slash and without for your home page. It is not technically possible to fetch your home page without the slash. Even if they don't show it, all browsers add it automatically. Search engine crawlers do so too. No search engine should be telling you they have duplicate content/title/description between the home page with and without a slash. A deeper directory is a different story. In that case the slash matters. – Stephen Ostermiller Jul 24 '18 at 22:22
  • Search engines (especially Google) are very good these days about identifying duplicate content and only indexing one variant. Even if Googlebot is crawling both http and https, I've only ever seen it choose one of the two to index when they have the same content. Where are you getting your data from? – Stephen Ostermiller Jul 24 '18 at 22:23
0

The correct way to favor https over http was fixed by modifying the htaccess file with the following lines

<IfModule mod_rewrite.c>
    RewriteEngine On

    RewriteCond %{SERVER_PORT} !^443$
    RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301,NE]
</IfModule>

I downloaded Screaming Frog to crawl the site to see exactly what the search engines are seeing. I realized that both versions were crawled but the http version showed that it was 301ed. Which it's exactly what search engines need to see. However, I realized that my pages were being duplicated within each version without a 301 Status (only the home path was). After some researching, I found out that a directive to a 404 Error page fixes that. So I modified the htaccess once again and this is the final code

<IfModule mod_rewrite.c>
    ErrorDocument 404 /404.php
    RewriteEngine On

    # It favors the www version without a canonical tag
    RewriteCond %{HTTP_HOST} !^www\.example\.com$ [NC]
    RewriteRule ^ https://www.example.com%{REQUEST_URI} [L,R=301,NE]

    # It favors the https version of URI
    RewriteCond %{SERVER_PORT} !^443$
    RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301,NE]

    # To externally redirect /dir/file.php to /dir/file
    RewriteCond %{THE_REQUEST} \s/+(.+?)\.php[\s?] [NC]
    RewriteRule ^ /%1 [R=301,NE,L]

    RewriteCond %{REQUEST_FILENAME}.php -f
    RewriteRule ^(.+?)/?$ $1.php [L]
</IfModule>

See the original question in my StackOverflow here (https://stackoverflow.com/questions/51504638/removing-the-php-file-extension-from-the-url-request)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.