14

I'm getting lots of requests from Googlebot on the .well-known/traffic-advice directory on my server. What is it and what are they looking for? Should I add something to this directory, and if yes what?

4
  • 1
    This is the only helpful document that came up in my google search, which appears to be nothing more than a proposal. Looks like it's a proposed standard to allow servers to ask visiting agents not to prefetch resources. What I can say unequivocally is that you are totally fine not having this on your server, and don't give it another thought in terms of website health. However, maybe someone else with more experience with this can provide a fuller answer in the answers section for educational value. Dec 29, 2021 at 4:08
  • 3
    FWIW, the official registry of ".well-known" URLs is at iana.org/assignments/well-known-uris/well-known-uris.xhtml but this specific /traffic-device doesn't exist there which just means it has been done outside of typical IETF framework. Dec 29, 2021 at 4:22
  • 2
    @PatrickMevzek It appears there now.
    – kashmiri
    Sep 18, 2022 at 4:05
  • This IANA registry has a policy of registered which means basically anything can be added once there is a document outlining the case and the process is followed. Sep 18, 2022 at 4:12

2 Answers 2

11

Googlebot with name "Chrome Privacy Preserving Prefetch Proxy" tries to find instructions if it can preload your website for the user surfing on Chrome (Chrome thinks, for example, the link to your website is going to be clicked on).

Basically what we, webmasters, are interested in is to get rid of 404s caused by this feature. The most simplest way is to create the file traffic-advice (without any extension) in the directory .well-known with the content:

[{
  "user_agent": "prefetch-proxy",
  "google_prefetch_proxy_eap": {
    "fraction": 1.0
  }
}]

Problem is that the bot requires a specific MIME type. On Apache, you can add these lines to the main .htaccess file:

RewriteRule ^\.well-known/traffic-advice$ - [T=application/trafficadvice+json,END]

On Nginx you can alter your config with these lines:

# Private Prefetch Proxy
# https://developer.chrome.com/blog/private-prefetch-proxy/
location /.well-known/traffic-advice {
   types { } default_type "application/trafficadvice+json; charset=utf-8";
}

More info or advanced Nginx config can be found from the source.

5
  • 3
    What does setting the fraction to 1.0 cause Chrome to do? Jun 8, 2022 at 16:52
  • 1
    @StephenOstermiller developer.chrome.com/blog/private-prefetch-proxy/#traffic > The fraction field gives you control over the prefetch traffic by specifying how much Private Prefetch Proxy should let through. The value is a float between 0.0 (no prefetch at all) and 1.0 (100% of the prefetch requests get through).
    – Ben
    Jul 3, 2022 at 5:05
  • 1
    The rewrite rule works but not 100% correct IMO, as the dot needs to be escaped, otherwise it just means any character. In addition the first forward slash should not be present. RewriteRule ^\.well-known/traffic-advice$ - [T=application/trafficadvice+json,END]
    – 8ctopus
    Nov 4, 2022 at 10:32
  • 1
    I have updated answer with your edit @8ctopus I'm don't have any server with Apache anymore where I can test it, you I just believe it works well :D Nov 6, 2022 at 7:27
  • If someone else is stuck with Apache 2.2 - as I am - I'd just like to pitch in that you'll have to do this instead of the RewriteRule (since T=... doesn't seem to be supported): <Location /.well-known/traffic-advice> ForceType application/trafficadvice+json </Location>
    – elgholm
    Nov 30, 2022 at 14:39
4

As @Maximillian Laumeister mentions the only reference to that .well-known uri is a proposal for prefetch control. I have also come across an obscure reference to it as a middleware layer able to be included in the Plack web application programming framework in order to handdle traffic-advice requests though this seems to have been done in direct response to the Github proposal by @Buettner.

From a practical standpoint unless you have that uri existing on your server for some other purpose (which would be unlikely) then there is no practical reason to be concerned. There is no harm in allowing Googlebot (or any other bot really) from requesting a non existent file and being hit with repeated 404 not found errors. If it is truly the Googlebot then it is like with time these requests will taper off due to the constant 404 not found errors.

If you are really concerned then you could add a custom rewrite rule for your server so that when a request is made for /.well-known/traffic-advice then instead of returning a 404 not found error it instead returns a 410 Gone error which Google generally will recognise for what the standard specifies, that the request uri is no longer available and will not be available again.

2
  • Okay, thank you all for your comments. I will just leave it the way it is and hope Google will stop hammering my server with unnecessary requests.
    – WendiT
    Dec 30, 2021 at 5:09
  • This is NOT Google bot. This is Chrome prefetch proxy.
    – kashmiri
    Sep 18, 2022 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.