Think about it. Why would Youtube let you use its content and infrastructure if they are hidden? The ads will not show and neither will the suggest videos at the end to direct visitors back to more stuff Youtube. You may have a great idea to use the audio for but unless it also benefits them, expect to be on your own.
While this is the my intuition, here is the official statement from Youtube's term of service:
If you use the Embeddable Player on your website, you may not modify,
build upon, or block any portion or functionality of the Embeddable
Player, including but not limited to links back to the YouTube