You shouldn't switch to a CDN too early. Read the update on hacked.com's experience with a CDN. If you don't have millions of visitors per day and your visitors are not distributed across a wide geography, then a CDN might slow your site down.
Instead, it would be wise to plan for using a CDN at some time in the future when designing your site, but not implementing it until needed.
When will the CND be needed: When two factors collide. Firstly that you tests show that the majority of users will see a speed improvement and secondly when it is financially viable for your site.