3

Will Google's crawler refuse to follow redirects when the difference between the redirected-from and redirected-to URLs is solely whether specific characters are percent-encoded or not? For example:

  • www.splunkbase.com/apps/All/4.x/Add-On/app:PDF+Report+Server+%28install+on+Linux+only%29
  • www.splunkbase.com/apps/All/4.x/Add-On/app:PDF+Report+Server+(install+on+Linux+only)

Both of these are, per the HTTP specs, valid and equivalent URIs, but our site's code always redirects to a "canonical" URL for each content page-- which in this case is the first URL listed.

Google clearly isn't indexing this page (in either URL variant). Neither of the URLs above show up when I search for "PDF Report Server (install on Linux only)".

Google webmaster tools reports a "redirect error" for the "decoded" variant of the URL: www.splunkbase.com/apps/All/4.x/Add-On/app:PDF+Report+Server+(install+on+Linux+only)

Another problem is that we're currently using a 302 instead of a 301 redirect to handle canonicalization-- we're switching to 301s soon for canonicalizing redirects.

But I'm wondering if the 302 vs. 301 issue may be a red herring-- that the actual underlying issue may be that, in Google's eyes, we're redirecting a URL to itself since, per the HTTP specs, a perecent-encoded and non-percent-encoded URL should be treated the same by clients and servers.

I found a related thread here. It's not the same issue-- in their case the only difference between redirected URLs was the upper/lower case of the percent-encoded hex values. But it's suspiciously similar to our issue.

Finally, my question: Has anyone run into this percent-encoding-plus-redirect issue, and if so can you discuss how you worked around it? Did switching to a 301 fix it, or was more needed?

For workarounds beyond 301-ing, we're looking at a variety of options from using REL=CANONICAL and turning off redirection in this case, to modifying our escaping to turn off escaping of apostrophes, parentheses, and other not-usually-percent-escaped characters.

For long-term fixes, we're looking at:

  • like this site does, using a numeric ID as a key, adding REL=CANONICAL to handle changes in SEO text after the title, and not doing any redirection
  • like many blogs do, continuing to use the title as the canonical URL, continue redirecting, but switching all problematic characters with dashes so we don't have to worry about encoding/decoding
3
  • REL=CANONICAL is definitely a must do
    – John Conde
    Sep 3, 2010 at 20:36
  • There's no need for that redirect. Those URLs mean the exact same thing; the only difference is how it's displayed to the user, and most browsers will only display as one or the other. E.g. in Firefox, any %28 and %29 in the address bar will just turn into ( and ). In Opera, it's just the opposite. And had you needed a redirect, you would not need a canonical link. Canonical links are a last-resort. Google prefers you simple use a permanent redirect. You wouldn't ever need to use both. Sep 4, 2010 at 1:09
  • re: Has anyone run into... yes, and my answer below did the trick. Loss of PR was a moot point, since the whole site was a re-write and the CMS swapped out. (and really after it was done it gained PR right back and then some because the CMS used SEO better.) Sep 17, 2010 at 7:14

3 Answers 3

4

These URLs are theoretically equivalent, so a redirect is likely being seen as a redirect to itself, which would be a reason for a crawl error. In general, if that is the case (and I'm assuming that is so), then I would recommend not trying to canonicalize the URL on that level, I would not redirect a URL to an alternate representation of the same URL.

Similarly, there's no need to use the rel=canonical link element for these two URLs. It's fine to use if there are alternate versions, such as different capitalization in the path or URL parameters, but just for these two URLs it will not have any effect.

For what it's worth, an easy way to test how Google sees URLs like these is to use the Fetch as Googlebot function in Webmaster Tools. It will not follow redirects, so you should be able to see exactly which URLs are fetched, allowing you to try the different variations and see how it reacts.

In a related note, using URLs such as the one you mentioned seems a bit problematic to me, since it may be difficult for users to link to URLs that use spaces (eg when copy & pasting a URL into a forum). Given a proper link, Google will be able to follow it, but if the server-side software does not recognize the full URL, then that link may end up being broken.

1
  • @Johh - thanks for the expert advice. I confirmed by Splunking (splunk.com) our web server logs that you are correct. Google is hitting only the unencoded version of the page, is being served a redirect (which Google sees as a redirect to itself) and Googlebot never follows that redirect to the encoded version. BTW, as I noted in the question above, the site (which I inherited) will eventually be changed to a better URL scheme which doesn't rely on mirroring all characters including non-alphanumerics, into the actual URL. But for now we will need to figure out a workaround. Sep 24, 2010 at 22:17
3

Firstly, the fact that Google has not indexed either URL variant does not indicate a problem with the URL itself. It's more likely a different reason, like Googlebot hasn't crawled that page yet or doesn't think it's interesting enough.

I would suggest a few steps:

  1. Remove the redirection entirely. As you say the URLs are treated as the same anyway. In fact Google Chrome automatically converts %28 into (. You may find some browsers do the opposite - ( into %28 - which may cause problems.
  2. Link to 'canonical' version. In other words, make sure all links that you have control over point to the correct, parenthesised, version.
  3. Put the canonical version in your sitemap. If you don't already have an XML sitemap, create one and submit it to Google Webmaster Tools.
  4. Use rel=canonical to set the correct URL. If you put the parenthesised version in there, Google should show that in search results instead of the other one.

One final suggestion would be to remove special characters from URLs where possible. Your URLs seem to be based on files names. Perhaps when you upload a file, generate a 'slug' for use on the website, e.g. pdf-report-server-install-on-linux-only and use that in the URL instead.

2

If you are using Apache, I would strongly recommend using mod_rewrite so that it will handle canonical URLs by simply serving the page requested rather than sending a redirect.

Your root problem really is the fact you are using URLs throughout your website that require UTF-8 encoding. While it may not be entirely as pretty looking original to limit your character set, it really does help down the road once other sites start linking to yours. Other sites are most likely going to end up re-encoding the URL and before you know it search engines will be trying to access the canonized link.

My best solution is to change the link to one without UTF-8 characters, then send a 410 GONE or 404 NOT FOUND response and add a 5 second redirect in the header to the corrected URL. Wait a couple of weeks and it will correct itself.

Of course, if the page is older (more than 1 year) this may never correct.

(edit tidbit: 410 GONE seemed to work best on URLs that absolutely should have never been spidered. For example temporary files and URLs with session data in the $_GET.)

1
  • you're right that changing URLs to only use alphanumeric characters and dashes is the right solution. Unfortunately the site (which I inherited) made the decision to use the page names as primary keys at the storage layer, so obvious changes like stripping non-alphanumeric chars is very expensive. So (for now at least) we're stuck with the URL scheme we have. Sep 24, 2010 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.