Take for example


But allow


Is that possible? Would disallowing /about?* work?

  • Do you mostly care about Google or does this have to work for other search engines and crawlers as well? Nov 6, 2020 at 2:13
  • @StephenOstermiller it should work for all crawlers yeah. But I guess I mostly care about Google.
    – Julio
    Nov 6, 2020 at 2:53

1 Answer 1


Yoast's guide to robots.txt has the solution:

User-agent: * 
Disallow: /*?*

(Replace the User-agent as necessary.)

The line above accounts for everything that comes after the first forward slash (after your root domain name), on either side of the question mark parameter delimiter, which needs to be present in the URL.

  • 1
    Wildcards are not standard in disallow rules. Most robots won't be able to use that rule. The major search engines including Google will honor that rule though. Nov 6, 2020 at 10:27
  • Right, @StephenOstermiller - according to Google's dev documentation (developers.google.com/search/reference/robots_txt), "Google, Bing, and other major search engines support a limited form of "wildcards" for path values. These are: * designates 0 or more instances of any valid character. $ designates the end of the URL." Nov 6, 2020 at 15:22
  • @HenryVisotski sorry It's not clear. /*?* would disallow any url with a '?' correct? So I was right in my example that /about?* would work only for /about ?
    – Julio
    Nov 7, 2020 at 12:10
  • @Julio Yes, that would work too. But that one line would disallow all URL's with parameters combined. Nov 8, 2020 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.