This is my site 7 months graph from Google Webmaster Tools site performance util.
As you can see the graph goes up and down, with big differences: less than 1 second up to 5 sec.
Since I never changed anything (not an image, nor a script, nothing of nothing) during all those months do you think this could be caused by poor hosting, basically an overloaded server?
Are you experiencing the same up/down graph for some/all of your sites on Google Webmaster Tools?
BTW : ths site was also optimized with YSLOW, but optimized or not, shouldn't the graph be still more stright?
EDIT: in Google Official Blog they show a pretty stright graph, but maybe it's just for graphic purpose.