If I want all bots to not crawl /specific-directory
, but I want them to operate "as usual" (thus: crawl) all other directories of my site, is this robots.txt entry adequate?
# All Bots
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# SemRush
User-agent: SemrushBot
Disallow: /
Or, do I need to specifically allow /allowed-directories
since I deny access to some directories?