Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
I would like to disallow all directories including a given phrase, i.e.
www.example.com/directory-1-aaa/www.example.com/directory-1-bbb/www.example.com/directory-1-ccc/www.example.com/directory-1-ddd/www.example.com/directory-1-eee/....
www.example.com/directory-1-xxx/Would following disallow rule include all these cases:
Disallow: */directory-1
AFAIK, all major search engines, supports wildcard in robots.txt (an unofficial extension), so Disallow: /directory-1-* should work.
As example, Google does - https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
Alternatively, you could setup htaccess / httpd.conf to return 403 if the URL starts with /directory-1- and the user agent contains the substring "bot".
/directory-1- then you don't need a "wildcard" at all and the trailing * should be removed (as per the Google doc you linked to). However, the question is arguably ambiguous (hence @Stephen's comment under the question)... whilst the examples appear to indicate URLs that start with /directory-1-, it's not actually what the question asks ("including a given phrase") which is further enforced by the OP's code example that uses a "wildcard" at the start of the path value, ie. "*/directory-1".
/foo/directory-1-xxx/?