Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file.

User-agent: Googlebot
Disallow:  

User-agent: Mediapartners-Google
Disallow:  

Sitemap: http://website.com/sitemap.xml

Is the above robots.txt file is correctly written? Yes or no ?

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

Robots.txt permits to you to block some web crawlers (not allowing), if you want to allow a specific crawler like Googlebot, just ignore it in your robots.txt (same for Mediapartners-Google).

Just take a look at robots.txt of Pro Webmasters for example.

If you don't want to block these two web crawlers and you don't want to focus on other web crawlers, your robots.txt must be like this:

User-agent: *
Allow: /

Sitemap: http://example.com/sitemap.xml
share|improve this answer
    
@Danny: I have edited my answer to allow you to understand. –  Zistoloen Jun 21 '13 at 13:28
    
To not be confused, just put the / after Allow. With the slash, there is no problem with your robots.txt. –  Zistoloen Jun 21 '13 at 14:07
add comment

No, that is not correct. The default behaviour is to allow everything.

Use this instead:

User-agent: Googlebot
Disallow: / 

If you want to allow everything to all User agents, try this:

User-agent: *
Disallow: / 
share|improve this answer
    
BlacKEagle is correct - as and easier way to generate them i tend to use this - mcanerin.com/EN/search-engine/robots-txt.asp –  sam Jun 21 '13 at 12:58
    
@ Sam: thanks @Danny: where is your slash? I gave you the correct answer, you don't have to worry anymore. –  BlackEagle Jun 21 '13 at 13:15
    
@BlackEagle The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. I read from this robotstxt.org/robotstxt.html –  Sundara Jun 21 '13 at 13:21
    
@BlackEagle Please do not give wrong answer, other user will learn wrong –  Sundara Jun 21 '13 at 13:28
2  
@BlackEagle: Allow exists for robots.txt. –  Zistoloen Jun 21 '13 at 13:37
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.