Robots.txt allow directory root while disallowing all deeper URLs

Question

I would like to allow a path and disallow everything else in the directory.

User-Agent: *
Allow: /example/
Disallow: /example/*

Will this work? If not what would be a better way of doing this?

Maximillian Laumeister · Accepted Answer · 2019-04-02 19:00:17Z

4

I actually found this helpful.

User-Agent: *
allow: /example/$
disallow: /example/*

16.4k3 gold badges32 silver badges63 bronze badges

answered Apr 2, 2019 at 18:51

Yaseen

813 bronze badges

3

The "$" matches the end of a URL for anyone curious. More info: developers.google.com/search/reference/robots_txt
– Maximillian Laumeister ♦
Commented Apr 2, 2019 at 19:00
3

Worth pointing out that the rule set only works with user agents that support both the Allow directive and regular expressions. Googlebot will obey that properly, and probably Bingbot. Not many other user agents will though.
– Stephen Ostermiller ♦
Commented Apr 2, 2019 at 19:59

Add a comment |

1 Answer 1