I don't think using separate SVG files for the images would be any better because that would take even more time to load because of separate HTTP requests.
Efficient handling of linked images is something that browsers are specifically designed to do well. By inlining 23 MB of data in the HTML, you're blocking those optimizations.
If you split the icons to individual files, then the browser will cache individual icons and avoid redownloading them when the same icon is used repeatedly, on the same page load and on separate page loads. The browser will also be able to display the full text of the page before receiving all the images, which will improve the user experience.
Also, your SVGs are overcomplex for rendering at typical icon size. 23 MB for 140 images is an average of 168 kB per image. How big are they displayed? A small icon, 32×32, at 3 bytes per pixel is 3 kB even as a completely uncompressed raw bitmap. A larger icon, 128×128, at 3 bytes per pixel, is a 48 kB raw bitmap. And uncompressed is the most pessimistic scenario to compare against. The point is, your SVGs contain more detail than you can possibly see at icon size. If you prerender them to PNGs, it will look identical but be dramatically smaller and lighter.
If you have the option to change the icons, you could also experiment with a different icon style. The current vogue is a flat iconographic shape with one or two colors, which happens to compress much smaller than a glossy gradient skeuomorphic icon. Changing the entire icon style is probably not necessary to fix your current performance problem, but if you needed to support very slow connections, it would help.