Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I want to allow Googlebot access all my pages with crawl delay. Yandexbot access only index.html. Bingbot access /tools pages.
All other bots will not be accessed my pages.

Is this robots.txt suitable for this?

User-agent: Googlebot
Crawl-delay: 300

User-agent: Yandex
Allow: /index.html
Disallow: /

User-agent: bingbot
Allow: /tools
Disallow: /

User-agent: *
Disallow: /
share|improve this question

1 Answer 1

I would use this code in your case:

User-agent: Googlebot
Crawl-delay: 300
Disallow: 

User-agent: Yandex
Allow: /index.html
Disallow: /

User-agent: bingbot
Allow: /tools
Disallow: /

User-agent: *
Disallow: /

Even if Crawl-delay: 300 directive is not a standard, just add the Disallow: directive for Googlebot and your code is fine.

Just for your information, to give only access to a page or a directory, you need to place Allow directive before Disallow: /.

More information on Wikipedia.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.