If I want all bots to not crawl /specific-directory, but I want them to operate "as usual" (thus: crawl) all other directories of my site, is this robots.txt entry adequate?

# All Bots
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# SemRush
User-agent: SemrushBot
Disallow: /

Or, do I need to specifically allow /allowed-directories since I deny access to some directories?

1 Answer 1


Disallowing specific directory is enough. There is no need to explicitly allow a directory unless it is a subdirectory of the disallowed one.

Your robots.txt looks fine.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.