Cache warming: is fetching the url of the pages enough?

Question

Googling on the topic of cache warming, I see that most approaches consist in automated http calls (get requests) to the pages listed in the sitemap, using a script or command line tool.

But I suppose that just sending a GET request to a url doesn't trigger the loading of all the resources of the web page of this url, since the request is not processed further (the script just makes the request and moves on to the next url listed on the sitemap)

If so, caching is far from perfect because resources on the page won't be cached : images, CSS, JS files... In this case, I probably need to emulate a browser to make sure the page loads all the resources "listed" on it. A tool like Selenium seems useful then.

Selenium is a much heavier solution than the usual scripting approaches to cache warming so I prefer to ask experts: has anyone a first hand experience on cache warming and how it should be properly executed?

Answer 1

You need to start by thinking about how much cache you have, where it is and how much time/resource you want to spend in warming the cache. There are many, MANY caches between the disk where your content is stored and the end-users screen, and not all the content appearing at the user's screen is cacheable.

just sending a GET request to a url doesn't trigger the loading of all the resources of the web page of this url

That's not the right starting point.

You can get a list of the URLs (all content types, and only the stuff you are hosting) by looking in your web logs. From this you can identify the content most frequently accessed, sizes and (if you've customized your logs to include response times) slow pages.

Then you can rank this list based on cost. The next step is to create an exclusion list - this would include anything with (e.g.) a varies header, requires authentication, is above your chosen size limit (you don't want to fill your reverse proxy cache with videos and large PDFs).

Then apply the lists as required.

Note that if you have a transactional website using some DBMS, then applying some warming there is also advisable - but that requires a different approach.