Google is pushing to standardize
robots.txt syntax because it was never standardized in fact; (see their announcement); this is happening right now at the IETF with this draft.
Google help page already references it, so I guess we can safely use it as a formal specification.
The draft includes this relevant grammar (keeping only the parts useful in our case):
rule = *WS ("allow" / "disallow") *WS ":"
*WS (path-pattern / empty-pattern) EOL
path-pattern = "/" *(UTF8-char-noctl) ; valid URI path pattern
; UTF8 derived from RFC3629, but excluding control characters
UTF8-char-noctl = UTF8-1-noctl / UTF8-2 / UTF8-3 / UTF8-4
UTF8-1-noctl = %x21 / %x22 / %x24-7F ; excluding control, space, '#'
UTF8-2 = %xC2-DF UTF8-tail
UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) /
%xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail )
UTF8-4 = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) /
%xF4 %x80-8F 2( UTF8-tail )
UTF8-tail = %x80-BF
So, in short, while you can use UTF-8, it has to be encoded, per RFC3986.
This is spelled out again later in the document:
Octets in the URI and robots.txt paths outside the range of the US-
ASCII coded character set, and those in the reserved range defined by
RFC3986 1, MUST be percent-encoded as defined by RFC3986 1 prior
Hence, I believe that you should use your second form:
* should be percent encoded only if you want to specifically match this character, but if you need its "glob" behavior of matching anything, you need to keep it as is.