Reduce Your File Size
Before I address your CDN concerns, you should really try to reduce your file size for the podcasts. Unless your podcast episodes are 2 to 4 hours long, each, you shouldn't be reaching 50 megs per file. Try reducing the audio quality in your mp3 files (and use .mp3 file compression if you're not already - it's the podcast industry standard). If you have a 45 minute podcast, you should be able to get your .mp3 file down to around 15 to 20 megs, tops, without any significant reduction in sound quality for spoken voice podcasts. Music podcasts may be different, and you probably want higher quality, but those are the exception where voice based podcasts tend to be the rule.
On Using A CDN
No, you don't have to push your entire site through the CDN. You can put only the files you want on it. When you do this, you'll link to your files through the CDN's URL instead of through your site URL.
For example, if I'm getting a copy of jQuery, I can either load
jquery.min.js in to my site directly and have it served from my services, or I can use a CDN like Google's CDN for it, and pull the file from a script that points to Google:
In this case, I am only pulling the one file from the CDN. The same idea can be applied to any file that you host on a CDN. One thing you should know, though, is that a CDN won't serve up dynamically generated content. So even if you wanted to host your entire site on a CDN, it would have to be a completely static site.
Large Files On A CDN
Bandwidth and storage services are ultimately what it comes down to, for CDN use. It's going to get really really really expensive, really quickly. This is the main reason why you don't want to use even a paid CDN for very large files.
Of course there are exceptions to this. Amazon CloudFront, for example, talks about streaming media very specifically. So if you do want to go down the CDN path, you might want to take a look at them.
SoundCloud Has Limits On Free
Regarding the other suggestion of SoundCloud...
StackExchange may use SoundCloud for it's podcast hosting, but they don't use a free account. Perhaps they have some deal with SoundCloud where they don't get charged anything, but this type of discount arrangement is very different than the free account that SoundClod offers. From SoundCloud's website, you can see exactly what limitations you will run in to with the free account:
2 total upload hours
Unlimited playlist creation
Some stats (see the amount of plays, downloads, comments and favorites of your tracks)
Paid Podcast Hosting Services
You may want to consider podcast hosting services. There are a lot of great services available, including my own at SignalLeaf.com, LibSyn, Blubrry, and many more. These services start cheap - very cheap - usually with a free trial before you actually have to pay for anything.
If you want more info on the possible dangers of "free" podcast hosting, check out my blog post on When Free Hosting Isn't Free. The post is a direct response to this question, but takes a podcast hosting point of view, vs directly answering the question here on using a CDN.