What is .well-known/traffic-advice directory?

Question

I'm getting lots of requests from Googlebot on the .well-known/traffic-advice directory on my server. What is it and what are they looking for? Should I add something to this directory, and if yes what?

Answer 1

Googlebot with name "Chrome Privacy Preserving Prefetch Proxy" tries to find instructions if it can preload your website for the user surfing on Chrome (Chrome thinks, for example, the link to your website is going to be clicked on).

Basically what we, webmasters, are interested in is to get rid of 404s caused by this feature. The most simplest way is to create the file traffic-advice (without any extension) in the directory .well-known with the content:

[{
  "user_agent": "prefetch-proxy",
  "google_prefetch_proxy_eap": {
    "fraction": 1.0
  }
}]

Problem is that the bot requires a specific MIME type. On Apache, you can add these lines to the main .htaccess file:

RewriteRule ^\.well-known/traffic-advice$ - [T=application/trafficadvice+json,END]

On Nginx you can alter your config with these lines:

# Private Prefetch Proxy
# https://developer.chrome.com/blog/private-prefetch-proxy/
location /.well-known/traffic-advice {
   types { } default_type "application/trafficadvice+json; charset=utf-8";
}

More info or advanced Nginx config can be found from the source.

Answer 2

As @Maximillian Laumeister mentions the only reference to that .well-known uri is a proposal for prefetch control. I have also come across an obscure reference to it as a middleware layer able to be included in the Plack web application programming framework in order to handdle traffic-advice requests though this seems to have been done in direct response to the Github proposal by @Buettner.

From a practical standpoint unless you have that uri existing on your server for some other purpose (which would be unlikely) then there is no practical reason to be concerned. There is no harm in allowing Googlebot (or any other bot really) from requesting a non existent file and being hit with repeated 404 not found errors. If it is truly the Googlebot then it is like with time these requests will taper off due to the constant 404 not found errors.

If you are really concerned then you could add a custom rewrite rule for your server so that when a request is made for /.well-known/traffic-advice then instead of returning a 404 not found error it instead returns a 410 Gone error which Google generally will recognise for what the standard specifies, that the request uri is no longer available and will not be available again.