I think it is something to do with my server configuration, but as google already listed the pages twice...
it happens on all the CMS pages.
If Google has already found and indexed both URLs then it implies there are links (somewhere) to both - either internal or inbound. Since you state that this "happens on all CMS pages" then it implies it is internal to your site. If the visible in-page links look OK, have you checked your sitemap, RSS feed etc.?
I managed to find a permanent [redirect] solution
This is a Magento issue, not a server one. If this is the case, then implementing a redirect is only a band-aid fix and not ideal if users are finding and navigating via these erroneous URLs (users will be constantly redirected and your access logs will be polluted with 301 redirects - this might, however, give you a clue as to how these URLs are being discovered). You will not be helping search engines by linking to URLs that are being redirected (if that is really what is happening).
To fix this properly you need to find where these erroneous URLs are coming from and fix at source. (Still keep the 301 redirect for SEO, since they have already been indexed.)
This isn't a server issue because...
Servers don't magically generate links to your pages.
It's not the server that makes these "virtual/CMS" URLs accessible. It would only be the server if these URLs mapped to physical files/directories on the filesystem (in which case the server controls additional pathname information and directory slashes).
It is the CMS that controls whether "virtual/CMS" URLs (eg.
/foo/bar/) map to the same resource at the time the URL is routed through the CMS. It is the CMS that decides whether to accept or reject one or the other or both. In fact, the CMS needs to take additional steps to ensure this happens.