Will Google's crawler refuse to follow redirects when the difference between the redirected-from and redirected-to URLs is solely whether specific characters are percent-encoded or not? For example:
Both of these are, per the HTTP specs, valid and equivalent URIs, but our site's code always redirects to a "canonical" URL for each content page-- which in this case is the first URL listed.
Google clearly isn't indexing this page (in either URL variant). Neither of the URLs above show up when I search for "PDF Report Server (install on Linux only)".
Google webmaster tools reports a "redirect error" for the "decoded" variant of the URL: www.splunkbase.com/apps/All/4.x/Add-On/app:PDF+Report+Server+(install+on+Linux+only)
Another problem is that we're currently using a 302 instead of a 301 redirect to handle canonicalization-- we're switching to 301s soon for canonicalizing redirects.
But I'm wondering if the 302 vs. 301 issue may be a red herring-- that the actual underlying issue may be that, in Google's eyes, we're redirecting a URL to itself since, per the HTTP specs, a perecent-encoded and non-percent-encoded URL should be treated the same by clients and servers.
I found a related thread here. It's not the same issue-- in their case the only difference between redirected URLs was the upper/lower case of the percent-encoded hex values. But it's suspiciously similar to our issue.
Finally, my question: Has anyone run into this percent-encoding-plus-redirect issue, and if so can you discuss how you worked around it? Did switching to a 301 fix it, or was more needed?
For workarounds beyond 301-ing, we're looking at a variety of options from using REL=CANONICAL and turning off redirection in this case, to modifying our escaping to turn off escaping of apostrophes, parentheses, and other not-usually-percent-escaped characters.
For long-term fixes, we're looking at:
- like this site does, using a numeric ID as a key, adding REL=CANONICAL to handle changes in SEO text after the title, and not doing any redirection
- like many blogs do, continuing to use the title as the canonical URL, continue redirecting, but switching all problematic characters with dashes so we don't have to worry about encoding/decoding