2

I'm going to try and keep this as terse as possible: when it comes to single quotes and commas in the URL, I'm damned if I encode and damned if I don't.

If I leave the single quote unencoded in the sitemap.xml loc entry, some crawlers (most notably, Bing) truncate the URL up to the point just before the single quote.

If I encode the single quote as ' according to this guide, some crawlers truncate the URL up to and including the ampersand. Bing used to do this until I contacted their tech support.

However now that my sitemap.xml is "proper" according to the guide, Google Webmaster shows a crap-tonne of 404's - most of which show that the Google crawler is using the XML encoded form of the URL (ex., http://example.com/someone'-lucky-day) instead of the decoded form (http://example.com/someone's-lucky-day). The other 404'd URLs contain commas (ex., http://example.com/someone,-really-hates-me becomes http://example.com/someone).

One thing to note: Whenever my web app raises a 500 server error, I get emailed a copy of the error. The email includes the URL attempted by the visitor (or crawler in this case). After switching my sitemap.xml to encode the single quotes, I haven't received any more of these error reports; for now, it's just Google Webmaster complaining.

2 Answers 2

1

HTML coding in pages can't be used as part of a URL.

You have to use special character coding for symbols that could possibly wreck the URL.

For the encoding, you start the character with a percent sign then a hexadecimal code which I think means the ascii code of the actual character you're trying to use.

Go to this page and use the forms and fill in your URL's in question to see how they should be encoded.

http://www.w3schools.com/tags/ref_urlencode.asp

4
  • 1
    This seems odd to me - considering it's an XML document that should be parsed using the XML schema. It's as if Google says "encode your URLs using ', but don't bother because we won't decode the XML node properly anyway". Feb 17, 2015 at 5:48
  • 1
    Encoded url's like the way you're saying can be used as links in an HTML document to comply with the HTML 4.01 w3c standard. What I suggest is to keep punctuation in URL's to a minimum. Try to make URLs friendly, and limit it to letters, numbers, dots, underscores, and hyphens. Feb 18, 2015 at 0:13
  • "HTML coding in pages can't be used as part of a URL." - What do you mean by this? URLs need to be percent-encoded and XML entity encoded (when used in an XML document) or HTML entity encoded (when used in an HTML document).
    – MrWhite
    Mar 20, 2015 at 12:08
  • @w3d Here's an example of what I mean. You can't have a URL that looks like this: http://www.web<site.com/pa<th/t>o/u!r$l/# Mar 20, 2015 at 20:02
1

FWIW... on the face of it Google would seem to be incorrect in my opinion. Or rather, it's implementation of the standard (RFC 3986) is too strict. (Although systems do vary in this respect.)

URLs always need to be suitably URL encoded / percent-encoded (as @mike states) by encoding characters that have special meaning, and then XML entity encoded when used in an XML document (or HTML entity encoded if used in an HTML document).

Whilst single quotes and commas are considered "reserved characters" in a URL, they have no special meaning in the path part of the URL and can be used as-is, without being percent-encoded. So, a URL such as http://example.com/someone's-lucky-day is perfectly valid as it is - the ' does not need to be encoded here (it would still need to be XML encoded in an XML sitemp). Just to clarify, there is no harm in percent-encoding these characters, in fact you can percent-encode everything if you wanted to!

Reference: StackOverflow quesion - Valid Characters for Directory part of a URL.

Also conflicting with Google's implementation is that the JavaScript method encodeURI() (for encoding the path parts of a URL) does not percent-encode the single quote and comma characters. However, the corresponding PHP function rawurlencode() does. On examining the output of these functions, it seems that JavaScript closely follows the standard; PHP does not.

However, another thought... Is there an encoding issue? Is the XML document UTF-8 encoded and are these characters really apostrophes and commas and not curly quotes or something that "looks" similar?!

1
  • I tend to agree with @w3d. I've got this problem too, getting 404 since URLs with comma are truncated after the comma (http://example.com/someone,-really-hates-me becomes http://example.com/someone). The thing is, if this was a real error on our side I should have seen a LOT MORE of these 404s. The fact that I'm seeing it for only a fraction of the URLs in the sitemap (and most of them contain a comma), makes me think that this is something wrong on Google's side.
    – Valhalla
    Sep 16, 2015 at 8:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.