Do some mail clients automatically urldecode %2b strings?

Question

We have an email address verification process on our website. The site first generates an appropriate key as a string

mykey

It then encodes that key as a bunch of bytes

&$dac~ʌ����!

It then base64 encodes that bunch of bytes

JiRkYWN+yoyIhIQ==

Since this key is going to be given as a querystring value of a URL that is to be placed in an HTML email, we need to first URLEncode it then HTMLEncode the result, giving us (there's no effect of HTMLEncoding here, but I can't be bothered to rework the example)

JiRkYWN%2ByoyIhIQ%3D%3D

This is then embedded in HTML that is sent as part of an email, something like:

click <a href="http://myapp/verify?key=JiRkYWN%2ByoyIhIQ%3D%3D">here</a>. 
Or paste <b>http://myapp/verify?key=JiRkYWN%2ByoyIhIQ%3D%3D</b> into your browser.

When the receiving user clicks on the link, the site receives the request, extracts the value of the querystring 'key' parameter, base64 decodes it, decrypts it, and does the appropriate thing in terms of the site logic.

However on occasion we have users who report that their clicking is ineffective. One such user forwarded us the email he had been sent, and on inspection the HTML had been transformed into (to put it in terms of the example above)

click <a href="http://myapp/verify?key=JiRkYWN+yoyIhIQ%3D%3D">here</a>
Or paste <b>http://myapp/verify?key=JiRkYWN+yoyIhIQ%3D%3D</b> into your browser.

That is, the %2B string - but none of the other percentage encoded strings - had been converted into a plus.

key=JiRkYWN%2ByoyIhIQ%3D%3D
key=JiRkYWN+yoyIhIQ%3D%3D

So I think that there are a couple of possibilities:

1. I'm doing something stupid that I can't see, or

2. Some mail clients try to cope with the problem of people mistakenly URLEncoding plus signs by 'helpfully' converting them all back

In case of 1 - what is it? In case of 2 - what mail clients? Or, alternatively, is there a standard, known way of dealing with this kind of scenario?

Many thanks for any help

Answer 1

It looks like you are doing the encoding correctly. I agree that some email client is doing something funky.

Here are some approaches you could try:

Replace spaces in the key

A plus in the url will get url decoded into a space. When your webserver gets the "key" parameter, have it replace spaces with pluses. That could fix the keys up so they could be read.

Use different characters for Base64 encoding

Instead of using + and = try two different symbols for base64 encoding such as . and - which are safer in the context of URLs and parameters. This would require that you modify or wrap your base64 encoder.

Switch to hex encoding

Hex encoding uses only numbers and the letters a though f. Hex encoded strings will be longer than base64 encoded strings, but it looks like your strings are pretty short and it wouldn't make a lot of difference in this case.

Don't encode your keys at all

When I write email verification process, I have my server generate a string of common letters and numbers that I can stick in a url parameter with no encoding. The algorithm for doing so looks something like this:

array alphabet=['a','b','c',...'A','B','C',...'0','1','2'...]
string key=""
for 0 to desired_key_length
    key += alphabet[random(alphabet.length)]