This will not affect your SEO. Any web crawler will be able to find a website's HTML sitemap as long as there is a link pointing to it.
Remember that you can submit your sitemap to most search engines consoles. You do not need to mention it in the
robots.txt. Using any search engines console UI you could submit
sitemap-name-you-would-like.xml you do not need to specify it in your
robots.txt nor a link in your website.
There are many reasons why you would prefer to hide your XML sitemap. For example, you would want to hide the XML Sitemap from your competitors and everyone that automatically or manually wants to learn about the content on your website.