I need to exclude anything that contains /page/ but only at root:

Disallowed: example.com/page/1

Allowed: example.com/category/page/1

How to safely write this in robots.txt?


You almost have it.

This is the correct syntax:

User-agent: *
Disallow: /page


The Disallow: directive is relative-to-root.

So the very first / in that directive indicates the root folder of your website.

Consequently, the directive Disallow: /page will disallow:


but it won't disallow:


because the latter three paths don't begin with /page/.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.