Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /account.php
Disallow: /cart.php

Is the above actually blocking crawlers? i just saw the extra "disallow" after the Google

share|improve this question

3 Answers 3

When you have Disallow followed by nothing, that effectively means "allow all":

Disallow: 

If you want to block all, that's:

User-agent: *
Disallow: /

So, in your example, you are allowing Mediapartners-Google access to everything, and then disallowing all other crawlers from these two pages:

User-agent: *
Disallow: /account.php
Disallow: /cart.php
share|improve this answer
    
Thank you, My worry was that by having Disallow: User-agent: *, that meant that i was overiding and telling everyone not to crawl ( in essence, meaning not allow any user agents ( crawlers ) –  jim Aug 21 '13 at 15:17

Note that this is not a valid robots.txt. Parsers may understand it anyway, but you can’t be sure, especially not for the future.

Each record needs to be separated by at least one blank line.

So your robots.txt should look like:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /account.php
Disallow: /cart.php

Meaning of this robots.txt:

  • the bot Mediapartners-Google may crawl all resources
  • all others bots may crawl all resources except those whose URL paths start with account.php resp. cart.php.
share|improve this answer
    
Since the default is to allow all, you don't even need the first two lines. –  DisgruntledGoat Sep 2 '13 at 9:29
    
@DisgruntledGoat: This would change the meaning. With these two lines, Mediapartners-Google is allowed to crawl account.php and cart.php. If you omit these two lines, the Mediapartners-Google bot would fall under User-agent: *, and so it would not be allowed to crawl those URLs anymore. –  unor Sep 2 '13 at 12:30
    
@DisgruntledGoat: The bot Mediapartners-Google doesn’t have to check for the User-agent: * record. User-agent: * is ONLY for those bots that are not matched by any other record. I don’t know if or if not Google implemented their parsers correctly, but this is from the original robots.txt specification: "default access policy for any robot that has not matched any of the other records" –  unor Sep 3 '13 at 9:39
    
@DisgruntledGoat: I tested Google’s implementation in GWT: it works in the way it should (as I described): /cart.php is allowed for Mediapartners-Google and disallowed for all other bots. –  unor Sep 3 '13 at 9:47
    
Sorry you are correct! I misread the output in GWT, my bad. –  DisgruntledGoat Sep 3 '13 at 10:14

Actually, you have three lines here:

User-agent: Mediapartners-Google Disallow: User-agent: *
Disallow: /account.php
Disallow: /cart.php

You can get detailed info about the Google crawler from them

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.