I'm going to try and keep this as terse as possible: when it comes to single quotes and commas in the URL, I'm damned if I encode and damned if I don't.
If I leave the single quote unencoded in the sitemap.xml loc
entry, some crawlers (most notably, Bing) truncate the URL up to the point just before the single quote.
If I encode the single quote as '
according to this guide, some crawlers truncate the URL up to and including the ampersand. Bing used to do this until I contacted their tech support.
However now that my sitemap.xml is "proper" according to the guide, Google Webmaster shows a crap-tonne of 404's - most of which show that the Google crawler is using the XML encoded form of the URL (ex., http://example.com/someone'-lucky-day
) instead of the decoded form (http://example.com/someone's-lucky-day
). The other 404'd URLs contain commas (ex., http://example.com/someone,-really-hates-me
becomes http://example.com/someone
).
One thing to note: Whenever my web app raises a 500 server error, I get emailed a copy of the error. The email includes the URL attempted by the visitor (or crawler in this case). After switching my sitemap.xml to encode the single quotes, I haven't received any more of these error reports; for now, it's just Google Webmaster complaining.