3

This question is a follow up to this one:
Should one hide RTL encoded URLs in robots.txt or not?

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, such as the following?

Disallow: /מדיה_ויקי:*
Disallow: /%D7%9E%D7%93%D7%99%D7%94_%D7%95%D7%99%D7%A7%D7%99:*

Both are actually "the same" in the sense that the first one is in its original RTL syntax (Hebrew, in this case) and decoded and the second one is encoded.

The "rationale" is to "aim" at any current or possible-in-future standard from Google (or any other major search engine for that matter).
For example:
Today one major search engine development team might prefer decoded versions and tomorrow another might prefer the encoded version;
Hence the question if it's okay to just have both and be done with it?

  • I think this is already answered by (my own answer) at webmasters.stackexchange.com/a/124210/75842. The specification says you need to "%-encode" things in the file. You can not envision everything the future will bring and do things defensively regarding that. So for me, right now, based on the specifications, doing anything else than %-encoding is wrong. – Patrick Mevzek Aug 30 at 22:34
  • Hello ! I think "you cannot" should have been "I think you shouldn't" ^__^ Thanks ! – JohnDoea Aug 30 at 23:13
  • Is @JohnDoea and @JosephElazar (from the linked question) the same person? – MrWhite Aug 31 at 16:55
  • Maybe so, this is for another discussion in meta I believe; – JohnDoea Aug 31 at 20:55
  • @PatrickMevzek by %-encoding do you mean to UTF-8 encoding or something else? I feel confused regarding this... – JohnDoea Aug 31 at 20:56
2
+50

Regardless of whether the proposed draft standard specifies that you should encode or not, to answer your specific question...

No, it is unlikely to cause a problem if you include both URL encoded (%-encoded) and URL decoded (%-decoded) URLs in robots.txt. One or other will simply get ignored, as one or other simply won't match.

UPDATE: To clarify, everything (ie. the entire robots.txt file) should be UTF-8 encoded. This is simply the character encoding of the file. So, both the "URL encoded" and "URL decoded" URLs in robots.txt are in fact UTF-8 encoded. "UTF-8 encoded" and "URL encoded" refer to different things. I'd previously used the phrase "standard UTF-8 encoded" to differentiate from "URL encoded" - this was perhaps misleading, as both are "UTF-8 encoded". Google's webmaster guidelines (quoted below) uses the terms "UTF-8 characters" and "percent-escaped UTF-8 encoded characters" to differentiate the two.

Rather confusingly, Google's robots.txt tester tool only appears to match against %-decoded URLs*1. But, as far as I've been able to determine, this is an (surprising*2) inconsistency in the tool only and not reflected in the real Googlebot. Reference: https://www.screamingfrog.co.uk/search-console-robots-txt-tester-inconsistencies/

(*1 It looks like the tool blindly URL encodes the URL before comparison?)
(*2 If this really is a bug in the tool, why hasn't Google addressed this issue in the last 3+ years?)

FWIW, the Google Webmaster help guidelines regarding robots.txt, which states that it has been updated to reflect the proposed Internet standard (1-July-2019) appears to suggest that whether you %-encode the URL or not is optional:

Non-7-bit ASCII characters in a path may be included as UTF-8 characters or as percent-escaped UTF-8 encoded characters per RFC 3986

And the Robots Exclusion Protocol - draft standard states:

Octets in the URI and robots.txt paths outside the range of the US- ASCII coded character set, and those in the reserved range defined by RFC3986, MUST be percent-encoded as defined by RFC3986 prior to comparison.

Note the last phrase, "prior to comparison" - this is an instruction for implementers of the proposed standard, not necessarily for those writing the robots.txt file itself.


Disallow: /מדיה_ויקי:*
Disallow: /%D7%9E%D7%93%D7%99%D7%94_%D7%95%D7%99%D7%A7%D7%99:*

Aside: You don't need the wildcard * at the end of the URL, since robots.txt is prefix matching by default.

  • "both include many percentage signs, don't they?" - No, the UTF-8 encoded string is the "decoded version" as stated in your question. robots.txt files (the whole file) should be UTF-8 encoded (character encoding), as opposed to some other language specific encoding, such as Western-1250 (Central European languages), ISO-8859-1 (Western Europe) or ISO 8859-8 (Hebrew), etc. – MrWhite Aug 31 at 20:22
  • 1
    "to disallow all webpages of a site one does have to use Disallow /*" - No, you only need Disallow: / (and don't forget the colon). The wildcard * is a later addition to the original robots.txt "standard". It is prefix-matching by default, eg. Disallow: /foo will disallow /foo, /foobar and /foobar/baz/anything.html, etc. You only need to use * if you have a variable part in the middle of the URL. – MrWhite Aug 31 at 20:28
  • Thanks; I must say - I go through a hard period and this topic is extremely confusing and annoying for me. I think this is the first time I hear on "0-7 bit ASCII" VS "full 8 bit UTF-8"; as far as I understand I can indeed combine decoded Hebrew + UTF-8 encoded Hebrew and it should be fine even though encoded UTF-8 Hebrew by itself should be enough. – JohnDoea Sep 1 at 3:29
  • BTW, should I define this entire topic "machine-encoding-for-natural-languages" or something else? – JohnDoea Sep 1 at 3:29
  • 1
    So, to be verbose, it would be "UTF-8 encoded + URL encoded (ie. %-encoded)" vs "UTF-8 encoded + URL decoded (ie. %-decoded)". Or just remove the term "UTF-8 encoded" completely, as that is assumed here. (robots.txt files should always be UTF-8 encoded, as opposed to any other character encoding.) I'll try and clarify this in my answer. – MrWhite Sep 5 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.