Web crawler list

What I want

  1. noimageindex (Indexation-controlling parameters) will only work for googlebot (2 : two meta tag - code below)

  2. Others all crawlers (example as a YandexBot, Baiduspider) will work for (1: one meta tag - code below)

Question is:

will below code will work for every crawler? if not, what is the easiest way to do it ? Will need I to write for separate mea tag for every crawlers?

In programming methodology there is a term overriding, That is why I am thinning (want to write meta tag) this way but I am not sure that will it work or not?

Example Code

1: <meta name="robots" content="index, follow" >

2: <meta  name="googlebot"  content="index, follow, noimageindex " >

1 Answer 1

<meta name="robots" content="index, follow">
<meta name="googlebot" content="index, follow, noimageindex">

Yes, this will "work" as intended, but can be simplified. Googlebot will merge the two and "will use the sum of the negative directives", but there's nothing to merge.

However, index and follow are the default actions and will simply be ignored. See the following related question: What is a difference in the order of "follow" and "index" in a robots meta tag?

So, the above could be simplified to a single element:

<meta name="googlebot" content="noimageindex">

If you don't want images indexed by any bot (including Googlebot) that might support the noimageindex directive then you can "simplify" this further:

<meta name="robots" content="noimageindex">

Any bots (incl. Googlebot) that support noimageindex will honour it, those that don't will ignore it. (This does assume that any other bots implementation is similar to that of Googlebot.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.