5

I have written the following code inside my robots.txt file:

   User-Agent: Googlebot
   Disallow:

   User-agent: Mediapartners-Google
   Disallow:

   Sitemap: http://example.com/sitemap.xml

Is my robots.txt is correct? I only want two user agent to follow my site (i.e., Googlebot & Mediapartners-Google).

0

2 Answers 2

4

Almost, see the bottom of the section named, "Blocking user-agents" in the following: Google Webmaster Tools: Block or remove pages using a robots.txt file

According that, you should have:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Mediapartners-Google
Allow: /

sitemap: http://example.com/sitemap.xml

See the bottom of the page in the above link for how to test your robots.txt file.

1
  • If you're wondering what the difference is between the "Disallow" and "Allow" directives in the above answers, see this: Allow directive According to that, for Google, an "Allow" directive with equal or more characters than the path of the "Disallow" directive will win. So the above is specific to Google.
    – dan
    Jun 20, 2013 at 5:23
2

Nearly, you need to Disallow all the other bots first though.

The wild card (*) below means all bots.

User-agent: *
Disallow: /

User-Agent: Googlebot
Disallow:

User-agent: Mediapartners-Google
Disallow:

Sitemap: http://example.com/sitemap.xml

Although please note, not all web crawlers will obey robots.txt.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.