1

I am currently tying myself up in knots over URL percent-encoding of extended Latin characters.


Step 1 of 4

I have the following URL:

https://example.com/fußgängerbrücke/

The offline folder name (which I have uploaded via FTP) exactly corresponds to this: fußgängerbrücke


Step 2 of 4

Wherever this URL has an internal link pointing to it anywhere on the site, the link now takes the percent-encoded form:

<a href="/fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/">

If I cut and paste the URL from the URL bar in Firefox, it pastes as: fu%C3%9Fg%C3%A4ngerbr%C3%BCcke


Step 3 of 4

But... imagine that URL is a referrer to the current page.

If I now use PHP to grab the URL (and edit it a little):

$My_Reference = str_replace('https://'.$_SERVER['HTTP_HOST'], '', $_SERVER['HTTP_REFERER']);
$My_Reference = explode('?', $My_Reference)[0];
$My_Reference = substr($My_Reference, 1, -1);

and then later, I use $My_Reference to retrieve some related data:

file_get_contents($_SERVER['DOCUMENT_ROOT'].$My_Reference.'/my-data.php');

it doesn't work.

I had a think about this and concluded that this might be what's happening:

  • file_get_contents() is (somehow?) detecting the extended Latin characters in $My_Reference and parsing fußgängerbrücke as fu%C3%9Fg%C3%A4ngerbr%C3%BCcke. (Can this be right?)
  • It then looks for a folder which literally exists as /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/ and can't find it, because the only folder that exists is /fußgängerbrücke/

So, to test this hypothesis, I tried:

file_get_contents($_SERVER['DOCUMENT_ROOT'].urldecode($My_Reference).'/my-data.php');

which does work (hooray!), but... well, it seems weird.

I felt uncomfortable about this, since I'm not trying (and don't need) to decode percent-encoded URLs anywhere else on the site, in any other context, and this just makes for a weird exception. For the sake of consistency, I'd just rather use percent-encoding everywhere.


Step 4 of 4

So... I went back to the original offline folder and renamed it from

fußgängerbrücke

to

fu%C3%9Fg%C3%A4ngerbr%C3%BCcke

and then uploaded it and replaced the old folder.

Guess what...? The new URL doesn't resolve!

Instead, I get a 404, now.

Presumably because the server is now automatically decoding the hardcoded percent-encoding and trying to find the folder /fußgängerbrücke/ in the webspace... which isn't there.

In summary:

If I upload a folder named fußgängerbrücke:

  • example.com/fußgängerbrücke resolves to /fußgängerbrücke/index.php
  • /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke resolves to /fußgängerbrücke/index.php

Whereas if I upload a folder named fu%C3%9Fg%C3%A4ngerbr%C3%BCcke:

  • /fußgängerbrücke goes to a 404
  • /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke goes to a 404

So what am I missing here? Two questions:

  • Does file_get_contents() automatically percent-encode extended Latin characters which then need to be explicitly percent-decoded again?
  • Is it impossible to have URL foldernames and filenames which have percent-encoding already hardcoded into them?
  • I've checked my .htaccess to make sure I haven't already included some rewrites for Extended Latin characters and I should be looking at mod_rewrite instead of php, but it turned out that there weren't any and (thankfully) I wasn't as daft as that. – Rounin Jan 27 at 14:20
0

You have your own solution by using urldecode() From the PHP manual urldecode decodes any %## encoding in the given string. Plus symbols ('+') are decoded to a space character.

| improve this answer | |
  • Yes, I already had it working using urldecode(). My issue wasn't getting it to work but figuring out what was going on under the hood that I would ever need to use urldecode() in the first place. It turns out (see my answer) that the culprit is $_SERVER['HTTP_REFERER']. – Rounin Jan 27 at 21:59
0

I think I've figured most of this out now.


I'll tackle the second question to begin with:

Is it impossible to have URL foldernames and filenames which have percent-encoding already hardcoded into them?

TLDR:

As far as I can tell, yes. It is impossible.


Explanation:

If I upload a folder named /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/ to my Apache web server, Apache will regard the foldername as a normal ASCII string which does not need escaping... oh, except for the fact that it contains % signs.

Which it will escape.

So Apache will now regard the folder name as:

/fu%25C3%259Fg%25C3%25A4ngerbr%25C3%25BCcke/

Every % is now transformed into a %25. This is why any attempt to point the browser at

  • /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/; or
  • /fußgängerbrücke/

will result in a 404.

I finally saw this explained in a discussion between @Marcelo Cantos and @Tom on Stack Overflow in 2011


Now for the first question:

Does file_get_contents() automatically percent-encode extended Latin characters which then need to be explicitly percent-decoded again?

Close, but not quite.

TLDR:

It's not file_get_contents() that is the issue here but $_SERVER['HTTP_REFERER'].


Explanation:

Once I have uploaded /fußgängerbrücke/ to my Apache server, here's what the PHP super global variable $_SERVER reports when the browser is currently pointing at /fußgängerbrücke/index.php:

  • $_SERVER['SCRIPT_NAME'] returns /fußgängerbrücke/index.php

and when the browser is pointing at a new page, after arriving from /fußgängerbrücke/index.php:

  • $_SERVER['HTTP_REFERER'] returns https://example.com/fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/

So, yes, the answer is that even though the foldername, really is /fußgängerbrücke/, PHP will report, via $_SERVER['HTTP_REFERER'], that the referrer's name was /fu%C3%9Fg%C3%A4ngerbr%C3%BCcke/.

Of course, if you then try to file_get_contents() from that address, Apache will interpret it as /fu%25C3%259Fg%25C3%25A4ngerbr%25C3%25BCcke/ (see above) and then PHP will tell you the latter doesn't exist.

| improve this answer | |
  • 1
    The list of characters that are valid for a folder name are not the same as the ones that are valid in a URL. Since the referrer header contains a URL, you should expect them to be encoded to make them valid. Reading data from the file system using a path built from unreliable user-provided data such as the referrer is inherently risky, so I hope you have a bit more validation than you've included in your question. – Tim Fountain Jan 28 at 10:40
  • In this case the script is being run in a PHP-powered CSS file. I need to know the HTML page which is calling that file, hence my deployment of $_SERVER['HTTP_REFERER']. – Rounin Jan 28 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.