1

Generally I like to disallow search engines to index images on my site.

There is one exception to this: I like the Google Favicon Bot to index images in order to show the icon in search results.

I am adding a HTTP header for images:

X-Robots-Tag: noindex

The header is not added if the request has an User-Agent containing "Favicon" in its name. So when the Google Favicon Bot requests the images then there is no such header set in the response.

However the Favicon seems to not being indexed. I am uncertain why but maybe Google evaluates the HTTP header beforehand with its normal Googlebot and therefore never queues it for the Favicon bot?

What I've already considered and that maybe part of a solution:

  • Adjust the HTTP header itself to disallow all bots except the Google Favicon bot. The problem here seems that the Google Favicon Bot has an unique User-Agent-String but does not have its own User-Agent Token. The Google Favicon Bot seems to use the Tokens Googlebot-Image and Googlebot. But the X-Robots-Tag HTTP header works with the User-Agent-Token and not with the User-Agent-String. So I cannot define different rules for Favicon Bot only.
  • Adding a Vary HTTP header for the User-Agent to indicate that the response varies based on the User-Agent requesting. I am not sure if this would help and I am afraid that this will destroy cache hit rates or even prevent resources from being cached at all.
5
  • Why did you decide to make an exception for the user agent rather than for the file name? If you made it so that you were favicon didn't have the no index regardless of the user agent, it would be very easy for you to verify that it's working. Sep 19, 2023 at 6:43
  • I'm also wondering if you consider using robots that text instead of no index tags? For images I don't believe Google ever indexes anything it can't crawl. He's straight forward to create a robot that text file that disallows all images except for your favorite icons. Sep 19, 2023 at 6:45
  • @StephenOstermiller I would like to enforce the noindex for images in a nginx rule that cannot be changed and is used by several websites on the server. Furthermore the Icon-URL can vary so it is impossible to have a generic rule for favicons based on the URL path. But that's basically what I want: A generic configuration that disallow indexing images except for the favicon. With a dedicated User Agent token for the Favicon bot that would be easy but the Favicon bot only has a dedicated User-Agent String but not token. Sep 21, 2023 at 13:48
  • If you control all these sites, I would just recommend standarizing the file names they use for favicons. Or create a list of all the file names used for favicons by all the sites. Sep 21, 2023 at 14:56
  • Adding the X-Robots-Tag: noindex header to images is the correct approach. Your favicon may not be showing due to several other reasons. Try an online checker like realfavicongenerator.net/favicon_checker to help you diagnose Sep 28, 2023 at 23:31

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.