It seems like an inherently obvious design choice on an ecommerce website for product discovery to solely take form of the user drilling down through categories and/or using on-site search. Under that assumption, Google must know that there is no reason to link to a bare category-less product from your site's navigation, so they must be expecting canonical directives like this. For that reason, I can't imagine that Google could penalize your site for having canonical pages that aren't linked internally.
Google's guidelines in fact show an example that is very similar to yours. The wording "consolidate link signals" leads me to believe that your usage is an expected use case of the canonical directive:
Why should I choose a canonical URL?
To consolidate link signals for similar or duplicate pages. It helps search engines to be able to consolidate the information they have for the individual URLs (such as links to them) into a single, preferred URL. This means that links from other sites to
http://example.com/dresses/cocktail?gclid=ABCD get consolidated with links to
Finally, Google doesn't say anything about penalties if your canonical page has low authority. All they say is "if the page isn't good enough, we might override your canonical choice". To me that means the very worst that can happen is that Google ignores your choice for canonical page:
Note that even if you explicitly designate a canonical page, Google might choose a different canonical for various reasons, such as performance or content.
If you want to be perfectly sure that it goes smoothly, I would recommend starting out with just one product as a test case. However, from what I read there is nothing in Google's guidelines to suggest that what you're doing will earn you any kind of penalty, and aside from a penalty there is no other possible reason that I know of that your use of canonicals in this case could negatively affect your SEO.