Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I had 2 questions in my mind:

  1. what is the difference between Allow and Disallow?

  2. which one should I use to allow Google bot for AdSense?

User-agent: Mediapartners-Google  

User-agent: Mediapartners-Google*  
Allow: /
share|improve this question
Crucially it's Disallow: without any value and Allow:+/ (with a value). The first means disallow nothing, whilst the second means allow everything. – w3dk Oct 3 '13 at 13:53
up vote 2 down vote accepted

The difference is Allow is not defined in specifications of robots.txt.

However, several web crawlers understand Allow like Disallow. It's the case for Mediapartners-Google bot. Therefore, you can use lines you want but if you want to respect specifications, choose the first ones:

User-agent: Mediapartners-Google  
share|improve this answer
robotstxt.org isn't opening for me. Can you provide any other reference? – user2434 Oct 3 '13 at 9:56
@Simon Walker: this is the website where specifications are defined. However, the website is currently down. – Zistoloen Oct 3 '13 at 10:36
And if iam not using the above code in robots.txt, can google my website?? – sandeep arron Oct 3 '13 at 12:50
If you don't use the code I mentioned, you didn't restrict your website access to Mediapartners-Google bot. Therefore, Mediapartners-Google bot will crawl your website. – Zistoloen Oct 3 '13 at 13:19
...unless there is another, perhaps more general, group in your robots.txt file that catches this crawler? – w3dk Oct 3 '13 at 22:16

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.