Your assertion that you are better off with one, bigger CSS file is correct. It will likely be only a few KB when gzipped, and should be cached, so not a huge overhead. There are a few things worth checking though.
If some CSS is only ever used on one page, it may be better in that case to put the CSS on the page, in some style tags. (Note: this can make things difficult to maintain, especially when you later decide to use a similar style elsewhere.)
If you take your most popular pages (for example the pages making up 50%+ of your page views) and find that only a very small amount of your CSS is being used on those pages, it may be faster for users to split it into two CSS files. Now, new users visiting your most popular pages have much less to download. On other pages there is one extra HTTP request but that's not a huge deal.
Make sure your CSS is well-optimized. Avoid descendent selectors where possible. If the right hand side of a selector is too generic then it can slow down rendering time. For example .class div {}
would be a little slow because the browser has to check every <div>
element on the page, then look up the DOM tree to the very top to find (or not) an element with the class.