CloudFlare Not Sending If-Modified-Since Headers

Question

I'm trying to reduce my server load by providing 304 Not Modified responses for content where appropriate. CloudFlare is my middle man, so they should be sending If-Modified-Since headers whenever a cached page has expired, right?

I'm receiving these responses, client-side:

• CF-Cache-Status: MISS on first page load
• CF-Cache-Status: Hit on page reloads for 20 seconds
• CF-Cache-Status: EXPIRED on page reload after 20 seconds

The expired request is forwarded to my server, but doesn't include an If-Modified-Since header. How can I get this to work?

<?php
$now = time();
header( "ETag: W/\"$now\"" );
header( 'Expires: '.gmdate('D, d M Y H:i:s \G\M\T', $now + 20) );

header( 'Last-Modified: '.time() );
header( 'Cache-Control: public, max-age=20' );

print('<pre>');
print_r($_SERVER);
print('</pre>');

Answer 1

From my understanding and experience it seems to me perhaps you've misunderstood how CDN caching works:

• From the example you've given, the CDN would not ask your web-server when the file was last modified because you've already told it that the file has expired and will therefore need to be re-fetched anyway.

• Web browsers will only send an If-Modified-Since header if the page/resource has been previously cached by the web-browser and came with a Last-Modified header when first requested.

• If you send an E-Tag header as in your example then the handling of this is not consistent across browsers and you may end up with an If-None-Match header in the subsequent request from the browser instead of an If-Modified-Since.

CDN's however do not typically behave the same as web-browsers, in some ways more similar to proxy servers or web accelerators:

• If a CDN-cached page has expired, when next requested by a web-browser, the CDN will simply fetch a fresh copy from the web-server, update the CDN-cached copy and forward the resource back to the web-browser in the HTTP response. There would normally be no requirement for If-Modified-Since headers reaching your web-server at all, though my testing has revealed that CloudFlare do still issue this header.

• To reduce your server load, enable the CDN to provide a level of caching beyond that already done by the web browsers, by using the s-maxage directive. Similar to the max-age directive on your Cache-Control header, however this one will be observed by CDN's (and similar services) in precedence over any other headers, while the max-age will be observed by the web-browser clients.

  header( 'Cache-Control: public, max-age=20, s-maxage=60' );

  If you use this header, the first request from a web-browser will be a CDN MISS but be cached then at both the web-browser and CDN. After the first 20 seconds, the web-browser cached copy will expire. If the page is then reloaded, the CDN will HIT for a further 40 seconds returning a copy from CDN-cache to the web-browser in the HTTP response. Sixty seconds after the first request the CDN-cache has expired and a subsequent request will be a CDN EXPIRED but otherwise be treated the same as a CDN MISS taking a fresh copy from your web-server, and so the loop sequence would continue.

  A more production-ready header, for web-browser caching of 1 hour, and CDN caching of 6 hours might look like:

  header( 'Cache-Control: public, max-age=3600, s-maxage=21600' );

  If you ever needed to publish an update faster than the 6 hour CDN expiry you can always tell the CDN to take a fresh cache from their web-based control panel. With this header each resource would only be fetched from your web-server 4 times a day while the CDN handles all the bulk of the web traffic.

---

Edited on 11-Nov-2014 @ 12:45pm UTC-0:

On a side note that might also have an impact, there are issues with your PHP code that may be impacting on its correct operation - your ETag and Expires header lines of code when I test them produce the following headers:

ETag: W/""
Expires: Thu, 01 Jan 1970 00:00:20 GMT

Perhaps instead try these lines in your test so that you can see your sent headers too:

<?php
$iClientCacheSecs = 20;
$iProxyCacheSecs = 60;
$dtNow = time();
$dtExpires = strtotime( sprintf( '+%s seconds', $iClientCacheSecs ));
$aHeaders = array();
$aHeaders[] = 'ETag: ' . $dtNow;
$aHeaders[] = 'Expires: ' . date( 'r', $dtExpires );
$aHeaders[] = 'Last-Modified: ' . date( 'r', $dtNow );
$aHeaders[] = sprintf( 'Cache-Control: public, max-age=%s, s-maxage=%s',
  $iClientCacheSecs, $iProxyCacheSecs );
foreach( $aHeaders as $sHeader ) header( $sHeader );
echo( 'Now: ' . date( 'r', $dtNow ) . '<br />' );
foreach( $aHeaders as $sHeader ) echo( $sHeader . '<br />' );
echo( '<hr />' );
foreach( $_SERVER as $sParam => $sValue ) {
  if(( strpos( $sParam, 'HTTP_CF' )) !== false ) 
    echo( $sParam . ': ' . $sValue . '<br />' );
  if(( strpos( $sParam, 'HTTP_IF' )) !== false ) 
    echo( $sParam . ': ' . $sValue . '<br />' );
}

Answer 2

First, make sure you've configured Cloudflare with Cache Level: Cache Everything and Origin Cache Control: On, whether in the global settings or as a page rule.

Additionally, you must not set a cookie in the Origin response, max-age=0, or a past expires date. Doing so will prevent Cloudflare from caching the page.

Note that some hosts (usually free or shared hosts) will remove any If-Modified-Since and ETag headers. This is supposedly done because these headers can cause extra requests on the server. If you run the following script twice in a row (6 seconds apart), you should see the If-Modified-Since header printed on the page. Otherwise, your Last-Modified header was removed before the page was sent to Cloudflare. This is done outside of the PHP script, whether in the Apache configuration or other server settings. Note: as of January 2020, 000webhost removes the Last-Modified header and Freehostia does not.

/*if( isset( $_SERVER[ 'HTTP_IF_MODIFIED_SINCE' ] ) )
{
    header( "HTTP/1.1 304 Not Modified" );
    exit();
}*/

header( "Cache-Control: max-age=5");
header( "Last-Modified: " . gmdate( 'D, d M Y H:i:s' ) . ' GMT' );
echo "<h1>Request Headers received by the server (from Cloudflare):</h1>\r\n<pre>";
print_r( getallheaders() );
echo "</pre>"

Answer 3

You should keep in mind that a 304 Not Modified header still requires a request going through to your server.

You properly won't want to send 304 Not Modified headers, you most like want to send a cache expire header, so that the browser doesn't even try to request the resources again. That of course only relevant for static resources (images, scripts etc.).

In my experience CloudFlare only send Last-Modified headers for static resources.

You might want to have a look at CloudFlare's page urls, and give your pages (i.e. not images, scripts, etc.) a Custom caching value other than default?