1

If I want all bots to not crawl /specific-directory, but I want them to operate "as usual" (thus: crawl) all other directories of my site, is this robots.txt entry adequate?

# All Bots
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# SemRush
User-agent: SemrushBot
Disallow: /

Or, do I need to specifically allow /allowed-directories since I deny access to some directories?

Improve this question

1 Answer 1

Reset to default
1

Disallowing specific directory is enough. There is no need to explicitly allow a directory unless it is a subdirectory of the disallowed one.

Your robots.txt looks fine.

Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.