Regardless of whether the proposed draft standard specifies that you should encode or not, to answer your specific question...
No, it is unlikely to cause a problem if you include both URL encoded (%-encoded) and URL decoded (%-decoded) URLs in robots.txt
. One or other will simply get ignored, as one or other simply won't match.
UPDATE: To clarify, everything (ie. the entire robots.txt
file) should be UTF-8 encoded. This is simply the character encoding of the file. So, both the "URL encoded" and "URL decoded" URLs in robots.txt
are in fact UTF-8 encoded. "UTF-8 encoded" and "URL encoded" refer to different things. I'd previously used the phrase "standard UTF-8 encoded" to differentiate from "URL encoded" - this was perhaps misleading, as both are "UTF-8 encoded". Google's webmaster guidelines (quoted below) uses the terms "UTF-8 characters" and "percent-escaped UTF-8 encoded characters" to differentiate the two.
Rather confusingly, Google's robots.txt tester tool only appears to match against %-decoded URLs*1. But, as far as I've been able to determine, this is an (surprising*2) inconsistency in the tool only and not reflected in the real Googlebot. Reference: https://www.screamingfrog.co.uk/search-console-robots-txt-tester-inconsistencies/
(*1 It looks like the tool blindly URL encodes the URL before comparison?)
(*2 If this really is a bug in the tool, why hasn't Google addressed this issue in the last 3+ years?)
FWIW, the Google Webmaster help guidelines regarding robots.txt
, which states that it has been updated to reflect the proposed Internet standard (1-July-2019) appears to suggest that whether you %-encode the URL or not is optional:
Non-7-bit ASCII characters in a path may be included as UTF-8 characters or as percent-escaped UTF-8 encoded characters per RFC 3986
And the Robots Exclusion Protocol - draft standard states:
Octets in the URI and robots.txt paths outside the range of the US-
ASCII coded character set, and those in the reserved range defined by
RFC3986, MUST be percent-encoded as defined by RFC3986 prior
to comparison.
Note the last phrase, "prior to comparison" - this is an instruction for implementers of the proposed standard, not necessarily for those writing the robots.txt
file itself.
Disallow: /מדיה_ויקי:*
Disallow: /%D7%9E%D7%93%D7%99%D7%94_%D7%95%D7%99%D7%A7%D7%99:*
Aside: You don't need the wildcard *
at the end of the URL, since robots.txt
is prefix matching by default.
%-encoding
do you mean toUTF-8
encoding or something else? I feel confused regarding this... – user58733 Aug 31 '19 at 20:56