1

I am trying to implement a library to handle robots.txt and I am a bit confused about the meaning of some rules that I believe are generated by WordPress:

Disallow: /?s=*
Disallow: /*/?s=*\
Disallow: /\\?s=*

I have seen these rules on multiple WP sites.

I understand the first line, forbid entries with s parameter. But what do the other lines forbid? What is the meaning of the backslash characters in these lines?

Could someone give me an example or two for these 2 rules?

1 Answer 1

1

Backslashes in disallow rules are literals. Neither the original robots.txt specification, nor Google's extended interpretation treat / in any way specially.

My guess is that some WP plugin is adding those rules erroneously. Backslashes are not safe URL characters so they shouldn't appear as literals in URLs.

1
  • Thought so, but was not sure. Thank you!
    – alturkovic
    Commented May 10, 2023 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.