10

I would like to allow folder /news/ and disallow all the sub folders under /news/ e.g. /news/abc/, /news/123/. How can I do that please?

I think Disallow: /news/ will block everything in it, including /news/ itself.

Will Disallow: /news/*/ do the job, since there is no easy way to test it, I want to make sure.

2 Answers 2

12
User-agent: *
Allow: /news/$
Disallow: /news/

Explanation:

Google's robots.txt spec (https://developers.google.com/search/reference/robots_txt), which is more up to date than the "official" spec, states that:

/fish/ will match anything in the /fish/ folder but will not match /fish (and, no wildcard necessary, since "The trailing slash means this matches anything in this folder.") If you kinda reverse engineer that:

User-agent: * (or whatever user agent you want to talk to)
Allow: /news/$ (allows /news/ but the $ character says the allow can't go beyond /news/)
Disallow: /news/ (disallows anything in the /news/ folder)

Test it in Google Search Console, or in Yandex (https://webmaster.yandex.com/tools/robotstxt/) to ensure it works for your site.

3
  • I tried this at the root level to allow all webpages to be crawled but to block all directories i.e.: User-agent: * Allow: /$ Disallow: / And tested it via the google search console but mywebsite.com/index.php was blocked. Does this need to be different at root level?
    – rbassett
    Commented Jul 12, 2021 at 15:12
  • 1
    I have answered my own question: User-agent: * Disallow: /*/
    – rbassett
    Commented Jul 12, 2021 at 15:27
  • 1
    Yep. You can even do just: Disallow: /* (because the final slash comes after the wildcard anyway, which covers the slash). Commented Jul 13, 2021 at 16:45
1

I had sort of the same issue. This:

User-agent: *
Allow: /folder/$
Disallow: /folder/

Did not work for me - the url/folder WOULD appear in the google search results - but it would just say NO INFO or something - nothing from the html page would be indexed. So I tried:

User-agent: *
Allow: /folder/index.html
Disallow: /folder/*

Same thing.

What DID work was to put this meta tag in the index.html file in /folder/:

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

With this tag - no links were followed to any other pages or folders but the contents of index.html DID appear.

1
  • Robots.txt doesn't prevent indexing, just crawling. It sounds like it was working it just wasn't doing what you expected. Commented Aug 29, 2019 at 1:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.