Just to begin a question: do you have certain Search Console properties for each language? If not - consider to establish.
According to my experience it should be enough to have such setup as you have it - one sitemap with all hreflangs. But such setups are pretty error-prone. There are many factors influencing 100% retrieving of sitemap urls - it is not possible to make a remote diagnose for this issue. Maybe you attach your sitemap to make it easier.
But i'll do the first try. According to your question, "Do I need to list the pages in all 5 languages in my sitemap", it seems, that you list in your sitemap only your english urls with corresponding hreflang urls. Like this:
<url>
<loc>http://www.example.com/english/page1.html</loc>
<xhtml:link
rel="alternate"
hreflang="de"
href="http://www.example.com/deutsch/page1.html"/>
<xhtml:link
rel="alternate"
hreflang="de-ch"
href="http://www.example.com/schweiz-deutsch/page1.html"/>
<xhtml:link
rel="alternate"
hreflang="en"
href="http://www.example.com/english/page1.html"/>
</url>
<url>
<url>
<loc>http://www.example.com/english/page2.html</loc>
<xhtml:link
rel="alternate"
hreflang="de"
href="http://www.example.com/deutsch/page2.html"/>
<xhtml:link
rel="alternate"
hreflang="de-ch"
href="http://www.example.com/schweiz-deutsch/page2.html"/>
<xhtml:link
rel="alternate"
hreflang="en"
href="http://www.example.com/english/page2.html"/>
</url>
<url>
But, according to Google, one should list ALL urls in this sitemap - means, if you list an english page as <loc></loc>
with corresponding german hreflang url, so you should list the german url as <loc></loc>
too, with corresponding english hreflang url. So, if you have three different language urls, you should have in the sitemap three <url></url>
blocks, like this:
<url>
<loc>http://www.example.com/english/page1.html</loc>
<xhtml:link
rel="alternate"
hreflang="de"
href="http://www.example.com/deutsch/page1.html"/>
<xhtml:link
rel="alternate"
hreflang="de-ch"
href="http://www.example.com/schweiz-deutsch/page1.html"/>
<xhtml:link
rel="alternate"
hreflang="en"
href="http://www.example.com/english/page1.html"/>
</url>
<url>
<loc>http://www.example.com/deutsch/page1.html</loc>
<xhtml:link
rel="alternate"
hreflang="de"
href="http://www.example.com/deutsch/page1.html"/>
<xhtml:link
rel="alternate"
hreflang="de-ch"
href="http://www.example.com/schweiz-deutsch/page1.html"/>
<xhtml:link
rel="alternate"
hreflang="en"
href="http://www.example.com/english/page1.html"/>
</url>
<url>
<loc>http://www.example.com/schweiz-deutsch/page1.html</loc>
<xhtml:link
rel="alternate"
hreflang="de"
href="http://www.example.com/deutsch/page1.html"/>
<xhtml:link
rel="alternate"
hreflang="de-ch"
href="http://www.example.com/schweiz-deutsch/page1.html"/>
<xhtml:link
rel="alternate"
hreflang="en"
href="http://www.example.com/english/page1.html"/>
</url>