Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file.

User-agent: Googlebot

User-agent: Mediapartners-Google

Sitemap: http://website.com/sitemap.xml

Is the above robots.txt file is correctly written? Yes or no ?

share|improve this question
up vote 1 down vote accepted

Robots.txt permits to you to block some web crawlers (not allowing), if you want to allow a specific crawler like Googlebot, just ignore it in your robots.txt (same for Mediapartners-Google).

Just take a look at robots.txt of Pro Webmasters for example.

If you don't want to block these two web crawlers and you don't want to focus on other web crawlers, your robots.txt must be like this:

User-agent: *
Allow: /

Sitemap: http://example.com/sitemap.xml
share|improve this answer
@Danny: I have edited my answer to allow you to understand. – Zistoloen Jun 21 '13 at 13:28
To not be confused, just put the / after Allow. With the slash, there is no problem with your robots.txt. – Zistoloen Jun 21 '13 at 14:07

No, that is not correct. The default behaviour is to allow everything.

Use this instead:

User-agent: Googlebot
Disallow: / 

If you want to allow everything to all User agents, try this:

User-agent: *
Disallow: / 
share|improve this answer
BlacKEagle is correct - as and easier way to generate them i tend to use this - mcanerin.com/EN/search-engine/robots-txt.asp – sam Jun 21 '13 at 12:58
@ Sam: thanks @Danny: where is your slash? I gave you the correct answer, you don't have to worry anymore. – BlackEagle Jun 21 '13 at 13:15
@BlackEagle The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. I read from this robotstxt.org/robotstxt.html – Ashu Jun 21 '13 at 13:21
@BlackEagle Please do not give wrong answer, other user will learn wrong – Ashu Jun 21 '13 at 13:28
@BlackEagle: Allow exists for robots.txt. – Zistoloen Jun 21 '13 at 13:37

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.