4

I have written the following code inside my robots.txt file:

   User-Agent: Googlebot
   Disallow:

   User-agent: Mediapartners-Google
   Disallow:

   Sitemap: http://example.com/sitemap.xml

Is my robots.txt is correct? I only want two user agent to follow my site (i.e., Googlebot & Mediapartners-Google).

  • 1
    Why do you only want Googlebot to visit your site? ( By the way I rolled back your latest edit because the question doesn't make sense without your explanation at the end.) – DisgruntledGoat Jun 21 '13 at 16:52
  • @DisgruntledGoat Please reply for this-> i have define my robots.xml in this way ansoftsys.com/robots.txt . Is my robots.txt is correct? – ashutosh Jun 21 '13 at 20:00
3

Almost, see the bottom of the section named, "Blocking user-agents" in the following: Google Webmaster Tools: Block or remove pages using a robots.txt file

According that, you should have:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Mediapartners-Google
Allow: /

sitemap: http://example.com/sitemap.xml

See the bottom of the page in the above link for how to test your robots.txt file.

  • If you're wondering what the difference is between the "Disallow" and "Allow" directives in the above answers, see this: Allow directive According to that, for Google, an "Allow" directive with equal or more characters than the path of the "Disallow" directive will win. So the above is specific to Google. – dan Jun 20 '13 at 5:23
  • 2
    Disallow: in moobot's answer is more widely supported than Allow: /. The original robots.txt specification did not have an Allow directive at all. – Stephen Ostermiller Jun 20 '13 at 12:59
  • It was copied directly from the Google Webmaster Tools doc linked to in my answer. Since User-agent: * would block all robots and the OP only wanted Google's bots, I figured I better use what they indicated. – dan Jun 20 '13 at 14:45
2

Nearly, you need to Disallow all the other bots first though.

The wild card (*) below means all bots.

User-agent: *
Disallow: /

User-Agent: Googlebot
Disallow:

User-agent: Mediapartners-Google
Disallow:

Sitemap: http://example.com/sitemap.xml

Although please note, not all web crawlers will obey robots.txt.

  • 1
    @Danny Have a look at my last comment in my answer - not concerned with who is "right" here (both work), but I always follow what Google states to do when I'm focusing on Google. – dan Jun 20 '13 at 15:20
  • @dan i will be very happy if anyone help me for this question webmasters.stackexchange.com/questions/49862/… – ashutosh Jun 20 '13 at 17:13
  • No, not if you still want to block all over crawlers. You have missed the first command. User-agent: * Disallow: / – Max Jun 21 '13 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.