2

I have a MediaWiki 1.32.0 RTL site (Hebrew) and I desire to hide some of its URLs from search engines like Google and Bing by robots.txt.

The robots.txt command Disallow: /מדיה_ויקי:* can have two UTF-8 versions for RTL languages (Hebrew in this case); one is decoded and one is encoded;

Decoded:

Disallow: /מדיה_ויקי:*

Encoded:

Disallow: /%D7%9E%D7%93%D7%99%D7%94_%D7%95%D7%99%D7%A7%D7%99:*

Both are same in essence - disabling indexation of everything that starts with מדיה-ויקי:.

Which one should I put in robots.txt?

2

Google is pushing to standardize robots.txt syntax because it was never standardized in fact; (see their announcement); this is happening right now at the IETF with this draft.

Google help page already references it, so I guess we can safely use it as a formal specification.

The draft includes this relevant grammar (keeping only the parts useful in our case):

rule = *WS ("allow" / "disallow") *WS ":"
       *WS (path-pattern / empty-pattern) EOL

path-pattern = "/" *(UTF8-char-noctl) ; valid URI path pattern

; UTF8 derived from RFC3629, but excluding control characters

UTF8-char-noctl = UTF8-1-noctl / UTF8-2 / UTF8-3 / UTF8-4
UTF8-1-noctl = %x21 / %x22 / %x24-7F ; excluding control, space, '#'
UTF8-2 = %xC2-DF UTF8-tail
UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) /
         %xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail )
UTF8-4 = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) /
         %xF4 %x80-8F 2( UTF8-tail )

UTF8-tail = %x80-BF

So, in short, while you can use UTF-8, it has to be encoded, per RFC3986.

This is spelled out again later in the document:

Octets in the URI and robots.txt paths outside the range of the US-
ASCII coded character set, and those in the reserved range defined by RFC3986 1, MUST be percent-encoded as defined by RFC3986 1 prior
to comparison.

Hence, I believe that you should use your second form:

Disallow: /%D7%9E%D7%93%D7%99%D7%94_%D7%95%D7%99%D7%A7%D7%99:*

Regarding :*, * should be percent encoded only if you want to specifically match this character, but if you need its "glob" behavior of matching anything, you need to keep it as is.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.