CDN Plug: Talking of CDN's Please use CDN-js to load your common JavaScript libraries.
Extra latency while my site fetch the data from the google servers.
This is not the case. The latency between a client (browser) fetching the google chart library from your server and the google server is the same.
Also hosting it on one CDN (i.e. google) increases the chances that the library is already cached and that the load is instant.
(-) Potentially the Google service can be down when my site is up.
Extremely small chance of this occurring but is a threat.
Google charts spefically can not be used locally.
Most other libraries used from a CDN can have a fallback for when they go down.
Fallback technique if CDN is down
Just plug in your own feature detection for whether google charts is loaded.
Actual Downsides:
The main downsides for using external tools is that they are not as flexible. They can do a lot of work for you though.
It also depends on how skilled you are at writing your own tools and how well you want to integrate them into the rest of your website infrastructure and general styling / look and feel.
Advice:
If you do not need to integrate the functionality with other parts of the website nor do you need to style it heavily to make it look integrated with the rest of your website then use google charts (or g.Raphael).
If you do not have the skills to write your own library that achieves similar functionality then use a library.