I use robots.txt to block all robots.txt compliant bots from a directory like this:
User-agent: *
Disallow: /sub/*
How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?
1 Answer
Adding this at the top of the robots.txt works:
User-agent: Mediapartners-Google
Disallow:
N.B. you may need AdSense to download your robots.txt again, which I believe they do on a 24hr basis.
*on yourDisallowrule.Disallow: /sub/is equivalent because robots.txt rules are "starts with" rules. Using the*is fine for Googlebot, but many robots don't support it and those bots will see it as a literal star. Your rule is more widely understood without the star.