0

I would like to disallow all directories including a given phrase, i.e.

  • www.example.com/directory-1-aaa/
  • www.example.com/directory-1-bbb/
  • www.example.com/directory-1-ccc/
  • www.example.com/directory-1-ddd/
  • www.example.com/directory-1-eee/

....

  • www.example.com/directory-1-xxx/

Would following disallow rule include all these cases:

Disallow: */directory-1
2
  • To clarify, you only want to disallow crawling? These directories should still be accessible?
    – MrWhite
    Apr 18, 2023 at 14:38
  • Are all these directories in the root of your site? Or are some of them deeper like /foo/directory-1-xxx/? Apr 18, 2023 at 17:47

1 Answer 1

0

AFAIK, all major search engines, supports wildcard in robots.txt (an unofficial extension), so Disallow: /directory-1-* should work.

As example, Google does - https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt

Alternatively, you could setup htaccess / httpd.conf to return 403 if the URL starts with /directory-1- and the user agent contains the substring "bot".

1
  • If you are just blocking URLs that start /directory-1- then you don't need a "wildcard" at all and the trailing * should be removed (as per the Google doc you linked to). However, the question is arguably ambiguous (hence @Stephen's comment under the question)... whilst the examples appear to indicate URLs that start with /directory-1-, it's not actually what the question asks ("including a given phrase") which is further enforced by the OP's code example that uses a "wildcard" at the start of the path value, ie. "*/directory-1".
    – MrWhite
    Apr 19, 2023 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.