I use robots.txt to block all robots.txt compliant bots from a directory like this:

User-agent: *
Disallow: /sub/*

How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?

  • 3
    You don't need the * on your Disallow rule. Disallow: /sub/ is equivalent because robots.txt rules are "starts with" rules. Using the * is fine for Googlebot, but many robots don't support it and those bots will see it as a literal star. Your rule is more widely understood without the star. – Stephen Ostermiller Sep 1 '19 at 10:19

Adding this at the top of the robots.txt works:

User-agent: Mediapartners-Google

N.B. you may need AdSense to download your robots.txt again, which I believe they do on a 24hr basis.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.