Our website hosting is with a shared hosting provider. The only configuration we can make is through .htaccess files, we can't touch the apache configuration files that are sourced when apache first starts.

The website root directory is public_html/. We have some content that should only be accessible to members, and this content is all kept under public_html/members_only/ files and subdirectories. Members have a username/password to authenticate when they want to get to the members-only portion of the site. The public_html/members_only/.htaccess file is fairly simple as follows:

AuthUserFile /path/to/.htpasswd
AuthType Basic
AuthName "Password Protected"
Options -Indexes
AuthGroupFile /dev/null
    Require valid-user
    Require method GET POST HEAD

When a particular file public_html/members_only/auto/xyz.pdf is requested in the browser the expected 401 status is returned, the browser prompts the user for their credentials and, if correctly entered, the request is sent again and the file is served. This is as expected. The problem is that when any other user who has not authenticated subsequently requests this same file, it's served to them without authentication, at least for a while. Then, after some interval, the next request is only satisfied if the user is authenticated.

Just to be sure I'm not experiencing some kind of browser caching confusion I have verified this behavior using curl -D headers https://<my website>/members_only/auto/xyz.pdf and examined the headers file.

I noticed another question that sounds very similar to my problem, but there were no answers. I am out of ideas at this point. I've tried tech support for the hosting provider (Endurance International Group), they are worse than useless.


Here is the content of the headers file when things are working as I expect:

HTTP/1.1 401 Unauthorized
Date: Fri, 16 Jul 2021 22:29:11 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 381
Connection: keep-alive
Server: Apache/2
WWW-Authenticate: Basic realm="Password Protected"
Age: 0

By contrast, here's what it looks like when it's serving the file without authentication:

HTTP/1.1 200 OK
Date: Mon, 19 Jul 2021 01:08:59 GMT
Content-Type: application/pdf
Content-Length: 1448787
Connection: keep-alive
Server: Apache/2
Last-Modified: Sun, 11 Jul 2021 00:50:41 GMT
ETag: "161b53-5c6ce64d9506f"
Accept-Ranges: bytes
Cache-Control: max-age=14400
Expires: Mon, 19 Jul 2021 04:52:35 GMT
Age: 985
  • When this happens is it on the same device and in the same browser as when the user was logged in? http-auth will persist in the browser for that session.
    – Steve
    Jul 19, 2021 at 3:59
  • No, it's not necessarily in the same device.
    – Phoenix GS
    Jul 19, 2021 at 12:07
  • From what server do these requests come from (perhaps indicated by a Server HTTP response header)? There might be a front-end (caching/static file) proxy (perhaps Nginx?) sitting in front of your application (Apache) server. This might be bypassing Apache (and your HTTP authentication) for these static file requests? So not a client-cache problem, but a server-side caching problem. (?)
    – MrWhite
    Jul 19, 2021 at 12:26
  • 1
    @MrWhite: I added some info to my question. Here is another clue that it might be some server-side caching: I don't see the problematic GET requests in my http log file. It's as if they never make it to the Apache server process.
    – Phoenix GS
    Jul 19, 2021 at 16:08
  • 1
    @PhoenixGS Yes, this does seem to be a response from a front-end caching proxy. I've updated my answer.
    – MrWhite
    Jul 19, 2021 at 16:56

1 Answer 1

Age: 985

The Age HTTP response header indicates the response is coming from an intermediary proxy server (cache), not your origin server. From the MDN Web Docs:

The Age header contains the time in seconds the object has been in a proxy cache.

The Age header is usually close to zero. If it is Age: 0, it was probably just fetched from the origin server;

This explains why you are not seeing the request logged in your application server's logfile.

Front-end proxy servers are often used for caching and serving static resources. However, since you are using HTTP Basic Authentication on the back-end application server you need to either disable the proxy for this particular file path or perhaps use the Authorization header as part of the cache-key (in other words, set a Vary: Authorization HTTP response header - although this shouldn't be necessary?).

You could try the following in your .htaccess file to see if the proxy takes notice:

# Vary cache based on "Authorization" HTTP request header
Header always set Vary "Authorization"

Or, perhaps:

# Disable all caching for this directory
Header always set Cache-Control "no-store, max-age=0"

Although you may still need to take this up with your shared webhost to get this resolved.

  • Ok, this is good information and gives me some experiments to try to tonight. I wonder that such caching can be implemented without the knowledge of downstream tenants and without respecting the http basic auth headers.
    – Phoenix GS
    Jul 19, 2021 at 17:21
  • Just wondering... what other directives do you have in any parent .htaccess files? You don't have anything that perhaps strips the Vary header from the response? (This is not uncommon, since the Vary header can cause problems with some caching proxies.)
    – MrWhite
    Jul 19, 2021 at 19:05
  • No, nothing like that, there are no Header directives. There is a rewrite rule to turn http requests into https requests, but that's about it.
    – Phoenix GS
    Jul 19, 2021 at 20:07
  • "there are no Header directives" - There doesn't need to be. If you access HTTP request headers using mod_rewrite RewriteRule / RewriteCond directives then Apache automatically includes this in a Vary HTTP response header, unless you state otherwise. This can sometimes cause issues with upstream caching proxies. @PhoenixGS
    – MrWhite
    Jul 19, 2021 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.