I found one particular file mootools.js takes a lot of time to load in ylsow, but opening the file directly - there is no delay.
You'll find many related posts within jQuery+MooTools on Stack Overflow, see this exemplary answer to Mootools and JQuery Integration mentioning huge page slow downs specifically.
Given that you are not using many modules I suppose only an isolated area of functionality is asking for MooTools specifically, consequently I suggest you turn that off entirely (i.e. make sure mootools.js is not loaded anymore) and see whether it makes any difference.
You might check out the various answers on a respective question on Stack Overflow - while they do mostly overlap with the ones given here in the meantime, there is some additional insight spread in the comments here and there.
Other than that I'd like to stress one important thing again, as mentioned by DisgruntledGoat and bpeterson76 already to some extent (+1 each):
Despite Drupal not being the fastest CMS in the first place, the page load times you are seeing aren't normal at all (though possible on badly configured systems/servers) - therefore you really should analyze and identify the cause(s) first, before applying optimizations of any kind! Sure, some recommended optimizations might help even without that, but the golden rule for performance (and other) optimizations really is to understand the problem rather than trial and error.
To get you started I'd like to comment on three topics raised specifically and provide a pointer to more thorough documentation regarding the subject matter below:
I have tried switching the VPS, but no performance gain
- Okay, still you should check explicitly, whether the VPS and LAMP stack in use are performing normal in the first place (i.e. without Drupal)?
I think the main culprit is the theme.
- Then you should verify this assumption, which is easily done by switching to another theme, preferably one with known performance characteristics, e.g. the default Garland theme.
I don't think the SQL query is at fault.
- You already realized that you should verify this assumption as well - this can be approached in various way, I'd start with the devel module, which has various offers regarding Performance Logging including SQL execution time (the other stuff is useful as well of course).
- Please note that for more precise SQL execution times you should resort to dedicated tools for the database system in use.
All this, and a whole lot more, is addressed via Drupal caching, speed and performance - this resource hub might look overwhelming at first sight, but as I said understanding the problem is key for solving performance issues - to allow for a jump start still eventually, I specifically recommend to read Server tuning considerations, in particular section Understanding and configuring your stack for performance.