The issue here is the way that the term sitemap has been used in webmastery to mean two different but similar things and this I think is where the confusion may lie.
The sitemap.xml file should not be linked to through a hyperlink on your website. The purpose of the sitemap.xml file is to list every page on your site and make it easier for search engines to index your site, however if you link to it directly with a hyperlink the crawlers will interpret it as a plain text file rather than a sitemap file. Crawlers will automatically look under the root domain name for a file named sitemap.xml and if found will use this to help identify additional pages on your site that it may not have known about yet due to the lack of links.
The sitemap that you find linked to within the site itself should be a separate but similar file. It should be written in HTML and should contain hyperlinks to the different sections of your site. This sitemap is to help users navigate your site.
As an example I will use the Monash University website...
The sitemap.xml file is located at http://www.monash.edu/sitemap.xml but the users sitemap is located at http://www.monash.edu/sitemap. As you can see the two sitemaps are different. The XML file lists out all the pages on the site and is extremely long, the sitemap web page is more designed for users and has links to the various sections.
This answer may be a generalisation and what you add to your two sitemaps is entirely up to you however the point is that a direct link to the sitemap.xml file should not be done from within your website as this is a background file only useful for search engine spiders.