Once again, I'm trying to optimize my site because one test showed the time to first byte is slightly over 200ms, but I need it under 200ms to make google happy.
I was looking at code to my site and for the longest time, I have image generation with name stamping done on-the-fly and for regular website users I have image generation done once on-the-fly and cached for subsequent requests.
Now I'm thinking maybe I should just cache every image thats generated regardless of what requests it, but I'm not 100% sure if this is such a good strategy, because I have so much space to store all of the the cached website files on a ram drive (about 1 GB), and if I use cache for a robot and the robot requests billions of images at once (yes I've seen a few 15Mbits/s spikes in my bandwidth log), then I'm afraid I'll run out of cache space quickly and the site will slow down to a halt until I manually reset the cache.
I could go with another option to reduce image processing but if I do that then legit users that see the image on the wrong site will wonder why the image isn't in the best quality.
If anyone has an idea how I can manage my image cache better so that it doesn't get full when bots try to access the site like crazy I'll be willing to look into it. Heck, I'll even accept anyone who provides code to a less-processor-consuming way to stamp images.
I just need some kind of starting point to solve this. I'll also appreciate answers about the upper limits robots use (such as how many requests bots make in a given time frame, and when etc) because if I can sync my cache with their behavior, then I can reset half my cache and have things run quickly again.