Handling extended Latin characters in URLs correctly with PHP and percent-encoding

Question

I am currently tying myself up in knots over URL percent-encoding of extended Latin characters.

---

Step 1 of 4

I have the following URL:

https://example.com/fußgängerbrücke/

The offline folder name (which I have uploaded via FTP) exactly corresponds to this: fußgängerbrücke

---

Step 2 of 4

Wherever this URL has an internal link pointing to it anywhere on the site, the link now takes the percent-encoded form:

<a href="/fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/">

If I cut and paste the URL from the URL bar in Firefox, it pastes as: fu%C3%9Fg%C3%A4ngerbr%C3%BCcke

---

Step 3 of 4

But... imagine that URL is a referrer to the current page.

If I now use PHP to grab the URL (and edit it a little):

$My_Reference = str_replace('https://'.$_SERVER['HTTP_HOST'], '', $_SERVER['HTTP_REFERER']);
$My_Reference = explode('?', $My_Reference)[0];
$My_Reference = substr($My_Reference, 1, -1);

and then later, I use $My_Reference to retrieve some related data:

file_get_contents($_SERVER['DOCUMENT_ROOT'].$My_Reference.'/my-data.php');

it doesn't work.

I had a think about this and concluded that this might be what's happening:

• file_get_contents() is (somehow?) detecting the extended Latin characters in $My_Reference and parsing fußgängerbrücke as fu%C3%9Fg%C3%A4ngerbr%C3%BCcke. (Can this be right?)
• It then looks for a folder which literally exists as /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/ and can't find it, because the only folder that exists is /fußgängerbrücke/

So, to test this hypothesis, I tried:

file_get_contents($_SERVER['DOCUMENT_ROOT'].urldecode($My_Reference).'/my-data.php');

which does work (hooray!), but... well, it seems weird.

I felt uncomfortable about this, since I'm not trying (and don't need) to decode percent-encoded URLs anywhere else on the site, in any other context, and this just makes for a weird exception. For the sake of consistency, I'd just rather use percent-encoding everywhere.

---

Step 4 of 4

So... I went back to the original offline folder and renamed it from

fußgängerbrücke

to

fu%C3%9Fg%C3%A4ngerbr%C3%BCcke

and then uploaded it and replaced the old folder.

Guess what...? The new URL doesn't resolve!

Instead, I get a 404, now.

Presumably because the server is now automatically decoding the hardcoded percent-encoding and trying to find the folder /fußgängerbrücke/ in the webspace... which isn't there.

In summary:

If I upload a folder named fußgängerbrücke:

• example.com/fußgängerbrücke resolves to /fußgängerbrücke/index.php
• /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke resolves to /fußgängerbrücke/index.php

Whereas if I upload a folder named fu%C3%9Fg%C3%A4ngerbr%C3%BCcke:

• /fußgängerbrücke goes to a 404
• /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke goes to a 404

---

So what am I missing here? Two questions:

• Does file_get_contents() automatically percent-encode extended Latin characters which then need to be explicitly percent-decoded again?
• Is it impossible to have URL foldernames and filenames which have percent-encoding already hardcoded into them?

Answer 1

You have your own solution by using urldecode() From the PHP manual urldecode decodes any %## encoding in the given string. Plus symbols ('+') are decoded to a space character.

Answer 2

I think I've figured most of this out now.

---

I'll tackle the second question to begin with:

Is it impossible to have URL foldernames and filenames which have percent-encoding already hardcoded into them?

TLDR:

As far as I can tell, yes. It is impossible.

---

Explanation:

If I upload a folder named /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/ to my Apache web server, Apache will regard the foldername as a normal ASCII string which does not need escaping... oh, except for the fact that it contains % signs.

Which it will escape.

So Apache will now regard the folder name as:

/fu%25C3%259Fg%25C3%25A4ngerbr%25C3%25BCcke/

Every % is now transformed into a %25. This is why any attempt to point the browser at

• /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/; or
• /fußgängerbrücke/

will result in a 404.

I finally saw this explained in a discussion between @Marcelo Cantos and @Tom on Stack Overflow in 2011

---

Now for the first question:

Does file_get_contents() automatically percent-encode extended Latin characters which then need to be explicitly percent-decoded again?

Close, but not quite.

TLDR:

It's not file_get_contents() that is the issue here but $_SERVER['HTTP_REFERER'].

---

Explanation:

Once I have uploaded /fußgängerbrücke/ to my Apache server, here's what the PHP super global variable $_SERVER reports when the browser is currently pointing at /fußgängerbrücke/index.php:

• $_SERVER['SCRIPT_NAME'] returns /fußgängerbrücke/index.php

and when the browser is pointing at a new page, after arriving from /fußgängerbrücke/index.php:

• $_SERVER['HTTP_REFERER'] returns https://example.com/fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/

So, yes, the answer is that even though the foldername, really is /fußgängerbrücke/, PHP will report, via $_SERVER['HTTP_REFERER'], that the referrer's name was /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/.

Of course, if you then try to file_get_contents() from that address, Apache will interpret it as /fu%25C3%259Fg%25C3%25A4ngerbr%25C3%25BCcke/ (see above) and then PHP will tell you the latter doesn't exist.