Vary: User-Agent header is used to tell browsers, CDNs, etc., (for caching purposes) that the content differs by user agents (especially for mobile and desktop versions). But do search engine robots use the header to crawl for mobile and desktops separately?

1 Answer 1


After posting the question a day back, I was researching Google official docs in parallel. Fortunately, I found the below phrases in https://developers.google.com/search/mobile-sites/mobile-seo/dynamic-serving

we recommend that the server send a hint to request that Googlebot for smartphones also crawl the page, and thus discover the mobile content. This hint is implemented using the Vary HTTP header.

It helps Googlebot discover your mobile-optimized content faster, as a valid Vary HTTP header is one of the signals we may use to crawl URLs that serve mobile-optimized content.

So my conclusion is that Google will possibly crawl desktop and mobile pages separately using specific bots.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.