I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file.

User-agent: Googlebot

User-agent: Mediapartners-Google

Sitemap: http://website.com/sitemap.xml

Is the above robots.txt file is correctly written? Yes or no ?


Robots.txt permits to you to block some web crawlers (not allowing), if you want to allow a specific crawler like Googlebot, just ignore it in your robots.txt (same for Mediapartners-Google).

Just take a look at robots.txt of Pro Webmasters for example.

If you don't want to block these two web crawlers and you don't want to focus on other web crawlers, your robots.txt must be like this:

User-agent: *
Allow: /

Sitemap: http://example.com/sitemap.xml
  • @Danny: I have edited my answer to allow you to understand. – Zistoloen Jun 21 '13 at 13:28
  • To not be confused, just put the / after Allow. With the slash, there is no problem with your robots.txt. – Zistoloen Jun 21 '13 at 14:07

No, that is not correct. The default behaviour is to allow everything.

Use this instead:

User-agent: Googlebot
Disallow: / 

If you want to allow everything to all User agents, try this:

User-agent: *
Disallow: / 
  • BlacKEagle is correct - as and easier way to generate them i tend to use this - mcanerin.com/EN/search-engine/robots-txt.asp – sam Jun 21 '13 at 12:58
  • @ Sam: thanks @Danny: where is your slash? I gave you the correct answer, you don't have to worry anymore. – BlackEagle Jun 21 '13 at 13:15
  • @BlackEagle The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. I read from this robotstxt.org/robotstxt.html – ashutosh Jun 21 '13 at 13:21
  • @BlackEagle Please do not give wrong answer, other user will learn wrong – ashutosh Jun 21 '13 at 13:28
  • 2
    @BlackEagle: Allow exists for robots.txt. – Zistoloen Jun 21 '13 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.