1

Take for example

/about?param1=blah
/contact?param1=bleh&param2=bluh

But allow

/about
/contact

Is that possible? Would disallowing /about?* work?

2
  • Do you mostly care about Google or does this have to work for other search engines and crawlers as well? – Stephen Ostermiller Nov 6 '20 at 2:13
  • @StephenOstermiller it should work for all crawlers yeah. But I guess I mostly care about Google. – Julio Nov 6 '20 at 2:53
1

Yoast's guide to robots.txt has the solution:

User-agent: * 
Disallow: /*?*

(Replace the User-agent as necessary.)

The line above accounts for everything that comes after the first forward slash (after your root domain name), on either side of the question mark parameter delimiter, which needs to be present in the URL.

4
  • 1
    Wildcards are not standard in disallow rules. Most robots won't be able to use that rule. The major search engines including Google will honor that rule though. – Stephen Ostermiller Nov 6 '20 at 10:27
  • Right, @StephenOstermiller - according to Google's dev documentation (developers.google.com/search/reference/robots_txt), "Google, Bing, and other major search engines support a limited form of "wildcards" for path values. These are: * designates 0 or more instances of any valid character. $ designates the end of the URL." – Henry Visotski Nov 6 '20 at 15:22
  • @HenryVisotski sorry It's not clear. /*?* would disallow any url with a '?' correct? So I was right in my example that /about?* would work only for /about ? – Julio Nov 7 '20 at 12:10
  • @Julio Yes, that would work too. But that one line would disallow all URL's with parameters combined. – Henry Visotski Nov 8 '20 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.