4

Google Webmaster Tools/Search Console is giving me errors regarding duplicate title tags and meta descriptions.

The website in question is a static HTML website. All documents do have a .html extension. In order to remove the .html from all documents I am using the code below in my .htaccess file:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^\.]+)$ $1.html [NC,L]

So for example http://example.com/about.html becomes http://example.com/about Now Google thinks that there are two separate about pages - even though it's only one. Can someone explain to me how to resolve this?

2

If your .html URLs were already indexed at the time you changed your URLs (and removed the .html extension) then the only way to preserve your SEO and avoid duplicate content from the get go is to implement 301 redirects from the .html URL to your desired URL.

(This assumes you have changed all the URLs in your application to your desired "extensionless" URLs.)

Something like the following at the top of your .htaccess file:

RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule (.+)\.html$ /$1 [R=301,L]

The check against REDIRECT_STATUS is to avoid a redirect loop by ensuring the rewritten request (to .html) is not redirected (when the internal rewrite is triggered, REDIRECT_STATUS is set to 200).


In order to remove the .html from all documents I am using the code below in my .htaccess file

Aside: I guess this is probably just how you are describing it, but that isn't actually what that snippet of code does. You "remove the .html" from the URL by physically changing the URLs in your application (not with .htaccess). You then use .htaccess to internally rewrite the URL back to the actual filesystem path (with the .html extension) - and it's this that your snippet of code does. It re-appends the .html extension, it doesn't remove it.

  • The website is very new and preserving any SEO doesn't play a role at this point. Does this mean that a 301 redirect is still the preferred solution? (Including the canonical links without the .html extension.) – Alex May 15 '17 at 7:21
  • Yes, a 301 redirect would be the preferred solution in order to eliminate the duplicate content warnings. However, a canonical tag should have been enough if the URLs were not already indexed and all the internal links are correct. If the "website is very new" and the .html URLs were not already indexed but still you are getting a duplicate content warning in GSC then it suggests a problem with your link structure (sitemap, etc.) - otherwise, Google should not have discovered the .html links in the first place. – MrWhite May 15 '17 at 7:53
  • 1
    Indeed the outdated sitemap.xml file could be the problem! All the links in there do have .html extensions... so Google most likely have indexed it. I have updated the sitemap. Let's see if it changes something – Alex May 15 '17 at 8:33
4

Let's assign www.example.com/about - Is your main URL and that URL you want to index in Google.

And www.example.com/about.html - Is your duplicate URL and that you don't want to index it on Google.

So There are two perfect solution. You can use any one or both.

1 ) Use 301 redirection from example.com/about.html to example.com/about . So Google will index only the final or redirected version of URL.

2) Use Canonical link tag on head section.

Your pages are duplicate hence your canonical link tag will be same on all these pages.

www.example.com/about/
www.example.com/about
www.example.com/about.html
www.example.com/about/index.html

So when you place below canonical link tag then all above pages will inheirt same canonical link tag, just like the webpage title/description is same for all URL's

<link rel="canonical" href="https://www.example.com/about" />

So here Google will index only that canonical link tag, other pages will consider as duplicate and Google avoid to index it.

  • All pages actually do have correct canonical link tags already. For example: example.com/about.html <link rel="canonical" href="example.com/about" /> Remember, I am removing the .HTML extension in the .htaccess file – Alex May 14 '17 at 13:12
  • Both pages are accessible via browser, right? Both display same result but there is no any redirection, so do 301 redirection And I think your canonical link tag is not same for duplicate pages, Is example.com/about and example.com/about.html contain same canonical link tag? – Goyllo May 14 '17 at 13:51
  • @Alex This answer is correct. You are doing a rewrite so that any search engine thinks the about.html is resolvable because the rewrite is transparent and a 202 result code is given. Your canonical tag gives a clue that /about exists which also resolves with a 202. However, with a redirect, the about.html is replaced with /about and you get what you want. – closetnoc May 14 '17 at 16:16
  • @Goyllo Both pages are accessible in the browser and display the exact same result because it's actually one an the same page. There are not two versions because example.com/about is the same as example.com/about.html and there is no redirection. This also means that I am only having one canonical link in this page which is <link rel="canonical" href="example.com/about"; /> – Alex May 15 '17 at 7:17
  • @Goyllo If I understand your comments and answers correctly, I need to add 301 redirects to all individual pages and duplicate content is not an issue regarding SEO? – Alex May 15 '17 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.