Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I had 2 questions in my mind:

  1. what is the difference between Allow and Disallow?

  2. which one should I use to allow Google bot for AdSense?

User-agent: Mediapartners-Google  
Disallow:

User-agent: Mediapartners-Google*  
Allow: /
share|improve this question
    
Crucially it's Disallow: without any value and Allow:+/ (with a value). The first means disallow nothing, whilst the second means allow everything. –  w3d Oct 3 '13 at 13:53
add comment

1 Answer 1

up vote 2 down vote accepted

The difference is Allow is not defined in specifications of robots.txt.

However, several web crawlers understand Allow like Disallow. It's the case for Mediapartners-Google bot. Therefore, you can use lines you want but if you want to respect specifications, choose the first ones:

User-agent: Mediapartners-Google  
Disallow:
share|improve this answer
    
robotstxt.org isn't opening for me. Can you provide any other reference? –  user2434 Oct 3 '13 at 9:56
    
@Simon Walker: this is the website where specifications are defined. However, the website is currently down. –  Zistoloen Oct 3 '13 at 10:36
    
And if iam not using the above code in robots.txt, can google my website?? –  sandeep arron Oct 3 '13 at 12:50
    
If you don't use the code I mentioned, you didn't restrict your website access to Mediapartners-Google bot. Therefore, Mediapartners-Google bot will crawl your website. –  Zistoloen Oct 3 '13 at 13:19
    
...unless there is another, perhaps more general, group in your robots.txt file that catches this crawler? –  w3d Oct 3 '13 at 22:16
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.