I use robots.txt to block all robots.txt compliant bots from a directory like this:

User-agent: *
Disallow: /sub/*

How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?

  • 3
    You don't need the * on your Disallow rule. Disallow: /sub/ is equivalent because robots.txt rules are "starts with" rules. Using the * is fine for Googlebot, but many robots don't support it and those bots will see it as a literal star. Your rule is more widely understood without the star. Sep 1, 2019 at 10:19

1 Answer 1


Adding this at the top of the robots.txt works:

User-agent: Mediapartners-Google

N.B. you may need AdSense to download your robots.txt again, which I believe they do on a 24hr basis.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.