Vary: User-Agent header is used to tell browsers, CDNs, etc., (for caching purposes) that the content differs by user agents (especially for mobile and desktop versions). But do search engine robots use the header to crawl for mobile and desktops separately?
1 Answer
After posting the question a day back, I was researching Google official docs in parallel. Fortunately, I found the below phrases in https://developers.google.com/search/mobile-sites/mobile-seo/dynamic-serving
we recommend that the server send a hint to request that Googlebot for smartphones also crawl the page, and thus discover the mobile content. This hint is implemented using the Vary HTTP header.
It helps Googlebot discover your mobile-optimized content faster, as a valid Vary HTTP header is one of the signals we may use to crawl URLs that serve mobile-optimized content.
So my conclusion is that Google will possibly crawl desktop and mobile pages separately using specific bots.