Do Google sitemap URLs need to be double urlencoded?

Question

Do Google sitemap URLs need to be double urlencoded?

I am submitting my sitemaps to Google and Google is having difficulty navigating to URLs that contain urlencoded characters. An example of a problematic URL is one that contains here&there (which has been urlencoded to here%26there). In my sitemap, this URL looks like this:

example.com/here%26there

However, Google misreads my sitemap and searches for:

example.com/here&there

which returns a 404 error. Do I need to double encode the URL in my sitemap so that it looks like this:

example.com/here%2526there

(note that % becomes %25)

Although this question is about Google and the way it reads sitemaps ((do Google sitemap URLs need to be double urlencoded)), I'll go ahead and show you guys the code I'm using to double encode my URLs for my sitemaps:

for($j = 1; $j <= $page_count; $j += 1){

   $data .= '<url><loc>'.str_replace('%', '%25', 
      BASE_URL.'products/'.
      urlencode(str_replace('/', '~1', $list['manufacturer']))).
      ($j == 1?'':'/'.$j).'</loc></url>'.PHP_EOL;
}

Note that non-URL '/'s are completely removed and all other special characters are double encoded.

Answer 1

Or you could just use preg_replace with urldecode in the URL construction to retain special characters in the URL such as what are currently in your sitemap.

preg_replace("/%u([0-9a-f]{3,4})/i","&#x\\1;",urldecode($[attach_necessary_object_here]));

This way the URL's would be:-

example.com/here/there

Answer 2

I when ahead and submitted my site maps to Google with Double urlencoding. In other words, if a URL contained a urlencoded character, such as %26, this became %2526

My 404 errors are now down from 14,000 to 10,000 and are still dropping.