Based on your suggestions I made some changes. Let me note that in
addition that we would like not to expose areas which we do not want to
crawl (by disallow / and then just specifying what we allow), we have
big issue that we have over 500 bots which come in and destroy our
caching system. We do not care for any bots except for google, bing
and yahoo.
Anyhow based on you suggestions (thank you all for very good inputs),
this is what I came up with:
User-agent: *
Disallow: /
# http://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
User-Agent: bingbot
User-Agent: Adidxbot
User-Agent: msnbot
User-Agent: msnbot-*
User-Agent: BingPreview
# https://support.google.com/webmasters/answer/1061943?hl=en
User-Agent: Googlebot
User-Agent: Googlebot-News
User-Agent: Googlebot-Image
User-Agent: Googlebot-Video
User-Agent: Googlebot-Mobile
#User-Agent: Mediapartners-Google
#User-Agent: AdsBot-Google
#
http://help.yahoo.com/l/uk/yahoo/mobile/onesearch/onesearchmb/search-69.html
User-Agent: Slurp
User-Agent: YahooSeeker/M1A1-R2D2
Disallow: /
Allow: /index.php
Allow: /person/
Allow: /archive/
Allow: /forkids/
Sitemap: http://yourdomain.com/sitemap.xml
Sitemap: http://yourdomain.com/sitemap.xml?page=1
Sitemap: http://yourdomain.com/sitemap.xml?page=2
Crawl-delay: 10
# GMT (8:00pm pst to 4:00am pst)
Visit-time: 0400-1200
Let me break down my new thinking here.
1)
User-agent: *
Disallow: /
Get all bots which cooperate out of our site
2)
# http://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
User-Agent: bingbot
User-Agent: Adidxbot
User-Agent: msnbot
User-Agent: msnbot-*
User-Agent: BingPreview
# https://support.google.com/webmasters/answer/1061943?hl=en
User-Agent: Googlebot
User-Agent: Googlebot-News
User-Agent: Googlebot-Image
User-Agent: Googlebot-Video
User-Agent: Googlebot-Mobile
#User-Agent: Mediapartners-Google
#User-Agent: AdsBot-Google
#http://help.yahoo.com/l/uk/yahoo/mobile/onesearch/onesearchmb/search-69.html
User-Agent: Slurp
User-Agent: YahooSeeker/M1A1-R2D2
Now that I banned all bots, for bing, google and yahoo bots I want to
make special provisions. Just for above bots do following:
Disallow: /
Allow: /index.php
Allow: /person/
Allow: /archive/
Allow: /forkids/
With "Disallow: /" I am disallowing all links from my site, except for these:
/index.php
/person/
/archive/
/forkids/
Allow: /index.php - I have added because with "Disallow: /" (from what
I understood), I am blocking my home page, which is not what I want to
do. Therefore I have added "Allow /index.php", which redirects to
homepage (just domainname.com).
I have added these to inform crawlers of sitemap.
Sitemap: http://yourdomain.com/sitemap.xml
Sitemap: http://yourdomain.com/sitemap.xml?page=1
Sitemap: http://yourdomain.com/sitemap.xml?page=2
Next two might not have big impact:
Crawl-delay: 10
# GMT (8:00pm pst to 4:00am pst)
Visit-time: 0400-1200
Google (and others) ignore "Crawl-delay", and google does not recognize
"Visit-time" directive.
Let me add that I have used google webmaster tools to check above rules,
and besides Crawl-delay and Visit-time disregard by google, I got:
1) That sitemap is recognized and loaded (not sure if I need to have
just first line or all 3 are needed). But this is ok.
2) Google respected first disallow for all bots, and recognized allows
that I specified in it's bot section.
3) Further more it respected disallow: / as well.
The only thing I could not verify is
Allow: /index.php
To see if I am eliminating my home page.
robots.txt
is completely unrelated to security. You should use proper authentication to make secret parts of your website unaccessible to unauthenticated users.