5

I'm using WordPress and I added some rewrite functionality so that certain posts that normally have a link like:

http://example.com/category/post-name

Now look like:

http://example.com/special/path/post-name

But if you were to visit either link, the same content is served. Working on fixing this, but I'm curious if I should set up redirects or canonicals.

On my website, there are absolutely no references to the initial link, only the second link. My guess is that as long as there aren't links pointing to the old link, Google doesn't know about it and won't magically find it either, making 301s or canonicals more of a "just in case".

Is my thinking correct?

  • 1
    Yes. You are correct, however, I always say an ounce of prevention goes a long way especially when the ounce is really a gram. If it is not too much trouble, use a canonical tag just to be sure. If it is too much trouble, forget I said anything. ;-) Cheers!! – closetnoc Dec 14 '16 at 1:46
  • Has the "normal" URL format ever been public? Or is the "special" URL the only one that has ever been used on the site? – MrWhite Dec 14 '16 at 8:42
5

You would think you thinking is correct, but it actually it isn't.

I have worked on many, many sites and some URL that do not have any physical links to them (or none we were aware of) always managed to get indexed in Google.

Who knows where Google finds the links, but invariably it does. So this is something you should definitely fix.

IF you can 301 redirect the duplicate pages to a single URL that would be the best fix, or if you need the duplicate URL to be live for what ever reason, set a canonical tag on the duplicate URL referencing a single URL.

<link rel="canonical" href="http://example.com/special/path/post-name" />

If for some reason you cannot set a canonical tag, you can set the robots meta tag to noindex them.

In the header section of the page:

<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">

Or in the HTTP header

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)

And as a very last resort, if you could not implement any of the above, you can block them in your robots.txt file, using something like:

Disallow: /category/
  • 1
    More on a general note, there is always a risk, especially with larger websites that there are still lingering links to these pages. Or the pages get included by an automated sitemap generator. I think it would be good practice to take extra measures like setting canonical tags. – Lucas Dec 14 '16 at 7:37
  • 1
    But what actually is the problem in this instance, why is it bad for SEO? That would seem to be the main point of the question; not the resolution. But this is not addressed in the above answer? Yes, Google might discover and index the content at this erroneous/duplicate URL - but that in itself isn't necessarily a problem. So, what actually is the "problem" in terms of SEO? – MrWhite Dec 14 '16 at 8:40
  • @w3dk That seems to be the title of the question, but not what is really asked in the question. I think this answer covers it well! – Tim Malone Dec 15 '16 at 18:39
5

Internal duplication is rarely bad for SEO. These days Googlebot is very good at detecting duplication and handling it appropriately.

Yes, Googlebot is likely to find and crawl the duplicate URLs eventually. However, in the case that Googlebot finds two URLs on the same site with the same content, it simply picks one to index. The one that it chooses is going to be either the one it found first, or the one with higher Pagerank. In all cases that is likely to be the one you have linked.

Google won't hand out any penalties for internal duplication. The worst thing that can happen is that Google will occasionally index a page on a URL that you would not prefer. It is also possible that Googlebot will use a lot of bandwidth and crawl budget crawling duplicate sections of your site that won't get indexed.

Other answers correctly tell you how to fix the problem, but I wanted to give a realistic expectation about how "bad" it could be.

See also: What is duplicate content and how can I avoid being penalized for it on my site?

  • I'd have to respectfully disagree here. I have seen big improvements on many websites after simply fixing internal duplicate content/pages issues. – Max Dec 15 '16 at 10:42
  • I did too -- ten years ago. Back then redirect to www, removing index.html, had measurable impact. When implemented more recently, I haven't seen rankings change because of it. Canonicalization is easy enough to do that I still recommend it, but I haven't seen cases of ranking gains recently except in cases where there are infinite duplicates that cause crawling problems because of session ids. – Stephen Ostermiller Dec 15 '16 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.