Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Do Google sitemap URLs need to be double urlencoded?

I am submitting my sitemaps to Google and Google is having difficulty navigating to URLs that contain urlencoded characters. An example of a problematic URL is one that contains here&there (which has been urlencoded to here%26there). In my sitemap, this URL looks like this:

example.com/here%26there

However, Google misreads my sitemap and searches for:

example.com/here&there

which returns a 404 error. Do I need to double encode the URL in my sitemap so that it looks like this:

example.com/here%2526there

(note that % becomes %25)

Although this question is about Google and the way it reads sitemaps ((do Google sitemap URLs need to be double urlencoded)), I'll go ahead and show you guys the code I'm using to double encode my URLs for my sitemaps:

for($j = 1; $j <= $page_count; $j += 1){

   $data .= '<url><loc>'.str_replace('%', '%25', 
      BASE_URL.'products/'.
      urlencode(str_replace('/', '~1', $list['manufacturer']))).
      ($j == 1?'':'/'.$j).'</loc></url>'.PHP_EOL;
}

Note that non-URL '/'s are completely removed and all other special characters are double encoded.

share|improve this question
    
See: stackoverflow.com/questions/1957115/… -- Slash and %2F are often viewed as interchangeable in URLs. –  Stephen Ostermiller Mar 18 at 10:08
    
I have been using the "Fetch as Google" option within Google's web master tools. I'm 99% sure that I need to double encode my URLs. Can anyone else confirm this? –  Hoytman Mar 18 at 21:52
    
You'd need to deal with the special characters with whatever code constructs the URL's in your sitemap. Without seeing that code, it's hard to give advice. If you use preg_replace you can handle characters such as & (it would be recommended to handle these in the URL's as well, even if replacing them with a hyphen or something). –  zigojacko Mar 19 at 8:31

2 Answers 2

Or you could just use preg_replace with urldecode in the URL construction to retain special characters in the URL such as what are currently in your sitemap.

preg_replace("/%u([0-9a-f]{3,4})/i","&#x\\1;",urldecode($[attach_necessary_object_here]));

This way the URL's would be:-

example.com/here/there
share|improve this answer
    
Would you mind explaining your regular expression, I'm not quite sure what it does. –  Hoytman Mar 18 at 12:58
    
It replaces and decodes %252F etc back to special characters such as / (slash) in this example. –  zigojacko Mar 18 at 14:35
    
Although this is not quite the solution I was looking for (the slashes definitely need to be encoded), I would like to know more about this regular expression usage. I understand how ([0-9a-f]{3,4}) works, but the rest is a bit of a mystery. –  Hoytman Mar 18 at 15:06
    
Actually, something seems to be amiss. You regex returns %2F for %252F instead of / –  Hoytman Mar 18 at 15:13
    
Without seeing the code of your website, is hard to give you an exact working line. I recently used my answer to retain slashes in an input field and it works perfectly. –  zigojacko Mar 18 at 16:06
up vote 0 down vote accepted

I when ahead and submitted my site maps to Google with Double urlencoding. In other words, if a URL contained a urlencoded character, such as %26, this became %2526

My 404 errors are now down from 14,000 to 10,000 and are still dropping.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.