1

Ahrefs is returning a series of duplicate content URLs for my own website i've asked them about and which they've said needs to be addressed on the CMS side, and which they cannot help with.

The issues seem to stem from their Ahrefs Platform identifying these two URL structures:

https://example.com

and

https://www.example.com

This has been driving me crazy for ages now (on a different domain) - my question is, is there a way, by using something like the .htaccess file (or possibly on the webhost side), to address this issue?

The webhost (ionos) appears to have added the following line/s to the .htaccess file in the last month, which as I understand it addresses directing visitors from http: to https (unless I'm wrong):

RewriteEngine On

RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.example.com/$1 [R,L]

I'm also uncertain as to whether this is having an impact on the duplicate content question?

I'll apologise in advance if this is covered elsewhere - I have trawled for the answer on StackExchange, but I cannot come to grips with (a) exactly what the problem is, and (b) how specifically to address it.

Improve this question
1
  • What is your canonical hostname? How are you linking internally?
    – MrWhite
    Commented Apr 5, 2021 at 10:26

1 Answer 1

Reset to default
1

What is "duplicate content"?

"Duplicate content" is simply where you have two or more different URLs referencing essentially the same resource and are not canonicalising this in some way (eg. with a rel="canonical" element or external redirect). https://example.com/foo/bar and https://www.example.com/foo/bar are two different URLs and are consequently treated differently when setting cookies, client-side caches and search engines. Whilst these two URLs might return the same content on your site (and indeed most sites), technically they might not, so do need to be treated differently.

In terms of SEO, search engines might end up (unnecessarily) dividing crawl budget between the two URL versions. They might choose to favour the non-canonical (ie. non-preferred) URL over the other when returning results in the SERPs. You may get backlinks to both URLs which will potentially divide so called "link juice" between the URLs which could result in neither page ranking as well as if you only have one canonical URL for one resource.

Having said that, search engines can often resolve www vs non-www duplicate content issues themselves, providing there is a strong enough signal. But they can still get it wrong.

Resolving "duplicate content"

You can potentially get a duplicate content issue if you are inconsistently linking (or referencing) both example.com and www.example.com throughout your site.

  • If you are using absolute URLs in your internal links then you need to consistently link to www or non-www. OR, use root-relative URLs starting with a slash and omit the scheme and hostname entirely.
  • rel="canonical" link element in the head section of your HTML needs to reference the canonical hostname.
  • XML sitemap needs to reference the canonical hostname.
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.example.com/$1 [R,L]

From this HTTP to HTTPS redirect, I assume your canonical hostname uses the www subdomain? If www is not your canonical hostname then that's obviously an error and should be fixed.

(If you have tested that this is working as expected then this should be a 301 - permanent - redirect, as opposed to a 302 - temporary - redirect, which is what this is currently defaulting to. ie. Change R to R=301 in the RewriteRule flags argument.)

Once you have confirmed/fixed the steps mentioned above then you can implement a non-www to www 301 external "redirect" in your .htaccess file to resolve any URLs that might have been indexed on the non-canonical hostname.

UPDATE: The redirect ensures that only the canonical URL (the www variant in this case) is viewed by users and crawled by search engine bots.

For example, after the HTTP to HTTPS redirect above:

# Redirect example.com to www.example.com
RewriteCond %{HTTP_HOST} ^example\.com [NC]
RewriteRule (.*) https://www.example.com/$1 [R=301,L]

TIP: Always test first with 302 (temporary) redirects to avoid potential caching issues, since 301s are cached persistently by the browser.

Improve this answer
9
  • Really appreciate the feedback! i've added in my individual responses just below this. Regarding referencing URL using either the root domain or the www. prefix, i'm using root relative linking? So for example, to get to my profile page the link is href="/profile.html" from anywhere in the website, so that it finds the root. i'll admit that i still do not understand the implications around the search engine optimisation for this - would this then mean that this could be interfered with when it comes to search engines identify the canonical pages?
    – spl
    Commented Apr 5, 2021 at 13:58
  • Regarding the sitemap, my XML sitemap only references 'https:' URLs.
    – spl
    Commented Apr 5, 2021 at 13:58
  • On the question of whether my canonical hostname uses the www subdomain - i don't know, to be honest? Can i check this (and, is there a location where i should check it?) Is this something to inspect on the webhost system, or within the actual site content?
    – spl
    Commented Apr 5, 2021 at 14:00
  • On the 301 redirection - here is what I've added within the htaccess file: RewriteCond %{SERVER_PORT} 80 RewriteRule ^(.*)$ https://www.example.com/$1 [R,L] # and added this to then get from non-www to www RewriteCond %{HTTP_HOST} ^example.com [nc] RewriteRule ^(.*)$ https://www.example.com/$1 [r=301,nc] Based on what you've outlined above, it looks as though my solution is (a) not necessarily correct if not canonical in the first place, and (b). also not correctly written up. I don't quite understand what the L is for in place of the NC?
    – spl
    Commented Apr 5, 2021 at 14:03
  • I've just checked, and I believe the canonical hostname is https://www.example.com.
    – spl
    Commented Apr 5, 2021 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.