An SEO expert was testing my site, and noticed that my URLs contained the special character
:. He said that would create duplicate content, because google would interpret any url containing
: as two separate URLs: one with
: and one with
%3A. Is he right?
I suspect this is lies/misinterpretation from your "SEO expert" (such roles do not exist IMO). Essentially
Otherwise you could argue that any non-alphanumeric character could cause duplicate content as they all have an URL encoded entity for someing (eg
Both resolve to the same place, except that
Your "SEO expert" might be a lying bastard, but this probably isn't the reason. He's absolutely right about this. This is a little known edge case in URL construction.
RFC 3986 is the official definition of the URL format and rules on how to encode and decode URL. Any URL parser should be following this as closely as possible to avoid errors and be interoperable with the rest of the Internet. That includes search engines which, if they apply the rules incorrectly, won't actually be able to crawl or index certain resources at all (e.g. they will get a 404 because they screwed up your URL, or your application will misinterpret the URL or query string).
The RFC gives rules on how to do the percent encoding and decoding, which we're all familiar with, but it also explains when to do the encoding and decoding, and to which characters.
Note that search engines normalize URLs (so that they can be compared) but they do not dereference them. Web servers dereference URLs to locate documents and to pass decoded data to your web application. When a URL is normalized, only a subset of percent encoded characters are decoded; when it is dereferenced, all of them are decoded.
In particular it specifies how to compare two URLs for equivalence (all of section 6) and which characters must be percent decoded before doing so (section 6.2.1 and 6.2.2). Here we find out that the only characters to be decoded before comparing URLs for equivalence are the so-called unreserved characters. These are defined (in section 2.3) as "uppercase and lowercase letters, decimal digits, hyphen, period, underscore, and tilde." Percent encoding is meant to prevent browsers, search engines, and the like from misinterpreting special characters in URLs, but since none of the unreserved characters have special meanings in a URL, these can be decoded by anyone at any time.
To construct valid relative URLs with a colon as the first path component, we would either have to encode it some of the time and not others, prefix all such relative URLs with
This part of the URL spec could use some clarification, but given the circumstances I would strongly recommend that if you are going to use colons in your URLs that you always encode them. This removes any possible ambiguity with respect to URL equivalence and ensures that you won't hit this edge case even if you use relative URLs.