I am not sure @Henry's reply is accurate nor Yoast's. Googles official announcement when they "rolled out" the x-default clearly states that x-default is to be used for a page that does not target any language or location. They actually mention this twice at their short post.
It seems that every URL gets 1 hreflang the same way it gets 1 canonical. Even though the effect might be similar google will have 2 signals, one says that the page targets English and the other that the page targets nothing.
Obviously google is able to identify the language of the doc itself independently of the hreflang. So to repeat what google said, x-default is for pages designed to be international.
If you want to make your "en" page international well that's fine, but then I don't see the point of adding conflicting "en" tag. On the other hand you could argue that without "x-default" google would pick what it thinks is best for the user which sometimes can be better than what you picked.
For example say you are a French living in Germany that does not speak English (note that you are targeting "de", which is what your browser setting is, not Germany). Without x-default most probably you will see "de" but with x-default you will see the English.
So basically the question is do you want to force English to say Swedish people or you want to let google show whatever thinks is best? For your case it seems to be not so important, but still you might want the control.
X-default is particularly important for sites that utilize same-language multiple locations, x-default is important as you don't want a page that is optimized for say Australia to be showing up for people in India, it makes no sense. So you create an x-default version to make sure google shows the appropriate pages to non-targeted locations.