This has to do with a combination of things. Your sitemap doesn't really tell Google what to index. It's more of a guide for the spiders than anything else. In fact, if you didn't have a sitemap, but you did submitted your website to Google, it would still be crawled.
The problem with having site:domain.com showing you so many, it's because, to the search engine spiders, anything and everything that could potentially be opened with an url, is a page. This is most frequent with WordPress sites. For example, if you have a woocomerce WordPress, technically every single product can be open in its own page. It would look like this :
So if you had 40 products divided in 5 categories, and each product has 3 color variations or sizes. You will have 125 pages that are not really in your site structure or even in your sitemap. 120 of them may be of good use for you, but the 5 category pages are useless.