A snippet of Advanced Gravatars:
Since Gravatar is serving the images
instead of you, the image load time is
out of your hands. Like a lot of good
internet services, Gravatar slowed way
down as it got more popular. A major
update came out earlier this year (2007)
which greatly improved scalability.
For instance, right when you upload an
image, it's scaled to all possible
sizes - from 1x1 to 80x80 - and
mirrored to two Amazon S3 servers.
Despite all that, though, it ran a bit
slow in some of my tests.
The post briefly discusses various approaches and references many real world examples.
Implementing a custom handler allows you to not show any external links for gravatars, but you can still decide where to get the images, with or without using a local cache to improve performance.
There are code samples (ASP.NET) and references for simple caching/reluctant caching/etc., that helped me implement my own version of a caching gravatar handler.