What is the correct way to write my "robots.txt" file?

Question

I have written the following code inside my robots.txt file:

   User-Agent: Googlebot
   Disallow:

   User-agent: Mediapartners-Google
   Disallow:

   Sitemap: http://example.com/sitemap.xml

Is my robots.txt is correct? I only want two user agent to follow my site (i.e., Googlebot & Mediapartners-Google).

Why do you only want Googlebot to visit your site? ( By the way I rolled back your latest edit because the question doesn't make sense without your explanation at the end.) — DisgruntledGoat, Jun 21, 2013 at 16:52
@DisgruntledGoat Please reply for this-> i have define my robots.xml in this way ansoftsys.com/robots.txt . Is my robots.txt is correct? — ashutosh, Jun 21, 2013 at 20:00

dan · Accepted Answer · 2013-06-20 05:03:44Z

3

Almost, see the bottom of the section named, "Blocking user-agents" in the following: Google Webmaster Tools: Block or remove pages using a robots.txt file

According that, you should have:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Mediapartners-Google
Allow: /

sitemap: http://example.com/sitemap.xml

See the bottom of the page in the above link for how to test your robots.txt file.

answered Jun 20, 2013 at 5:03

dan♦

15.1k10 gold badges40 silver badges52 bronze badges

If you're wondering what the difference is between the "Disallow" and "Allow" directives in the above answers, see this: Allow directive According to that, for Google, an "Allow" directive with equal or more characters than the path of the "Disallow" directive will win. So the above is specific to Google.
– dan ♦
Jun 20, 2013 at 5:23
2

Disallow: in moobot's answer is more widely supported than Allow: /. The original robots.txt specification did not have an Allow directive at all.
– Stephen Ostermiller ♦
Jun 20, 2013 at 12:59
It was copied directly from the Google Webmaster Tools doc linked to in my answer. Since User-agent: * would block all robots and the OP only wanted Google's bots, I figured I better use what they indicated.
– dan ♦
Jun 20, 2013 at 14:45

Add a comment |

Zistoloen · Accepted Answer · 2013-07-26 08:03:11Z

2

Nearly, you need to Disallow all the other bots first though.

The wild card (*) below means all bots.

User-agent: *
Disallow: /

User-Agent: Googlebot
Disallow:

User-agent: Mediapartners-Google
Disallow:

Sitemap: http://example.com/sitemap.xml

Although please note, not all web crawlers will obey robots.txt.

Zistoloen

10.1k6 gold badges34 silver badges59 bronze badges

answered Jun 20, 2013 at 5:02

Max

6,08716 silver badges40 bronze badges

1

@Danny Have a look at my last comment in my answer - not concerned with who is "right" here (both work), but I always follow what Google states to do when I'm focusing on Google.
– dan ♦
Jun 20, 2013 at 15:20
@dan i will be very happy if anyone help me for this question webmasters.stackexchange.com/questions/49862/…
– ashutosh
Jun 20, 2013 at 17:13
No, not if you still want to block all over crawlers. You have missed the first command. User-agent: * Disallow: /
– Max
Jun 21, 2013 at 12:26

Add a comment |

2 Answers 2