2

As far as I can tell, in 2017, the <meta name="robots" content="index, follow"> accepts the following directives in the content attribute:

  • index
  • follow
  • all (equivalent to content="index, follow")

and

  • noindex
  • nofollow
  • none (equivalent to content="noindex, nofollow")

and

  • noarchive
  • nosnippet
  • notranslate
  • noimageindex

and also (albeit these two are functionally deprecated):

  • noodp
  • noydir

Is unavailable_after: [RFC-850 date/time] still a legitimate directive?

Are there any other directives which might be included in a fully comprehensive list?

2

There would seem to be only 8 mainstream directives for the robots meta tag (or X-Robots-Tag HTTP response header) that really matter. Google includes 9 directives in its list of valid directives, including all, which is arguably just ignored.

Note that Google does not include index and follow in its list of supported directives. These are most probably just ignored, as I would expect any "unknown" directive to be.

Google lists the following supported directives (which are already included on your list):

  • all - There are no restrictions for indexing or serving. Note: this directive is the default value and has no effect if explicitly listed.
  • noindex - Do not show this page in search results and do not show a "Cached" link in search results.
  • nofollow - Do not follow the links on this page
  • none - Equivalent to noindex, nofollow
  • noarchive - Do not show a "Cached" link in search results.
  • nosnippet - Do not show a text snippet or video preview in the search results for this page. A static thumbnail (if available) will still be visible.
  • notranslate - Do not offer translation of this page in search results.
  • noimageindex - Do not index images on this page.
  • unavailable_after: [RFC-850 date/time] - Do not show this page in search results after the specified date/time. The date/time must be specified in the RFC 850 format.

Are there any other directives which might be included in a fully comprehensive list?

There could be any number of "other" directives, that are (or were) supported by "other" bots. The syntax is designed to be extended and used as required. However, unless you need to support some "other" bot (which you would probably already be made aware of the specific directive to use) then the de-facto standard is probably that stated by Google above.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.