2

I want to allow Googlebot access all my pages with crawl delay. Yandexbot access only index.html. Bingbot access /tools pages.
All other bots will not be accessed my pages.

Is this robots.txt suitable for this?

User-agent: Googlebot
Crawl-delay: 300

User-agent: Yandex
Allow: /index.html
Disallow: /

User-agent: bingbot
Allow: /tools
Disallow: /

User-agent: *
Disallow: /
1

I would use this code in your case:

User-agent: Googlebot
Crawl-delay: 300
Disallow: 

User-agent: Yandex
Allow: /index.html
Disallow: /

User-agent: bingbot
Allow: /tools
Disallow: /

User-agent: *
Disallow: /

Even if Crawl-delay: 300 directive is not a standard, just add the Disallow: directive for Googlebot and your code is fine.

Just for your information, to give only access to a page or a directory, you need to place Allow directive before Disallow: /.

More information on Wikipedia.

0

I would recommend to add sitemap directive and Host (for Google and Yandex) in file robots.txt. Host allows websites with multiple mirrors to specify their preferred domain.

Sitemap: http://www.example/sitemap.xml
Host: example.com

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.