I have a project and I would like to disallow everything starting with root.
From what I understand I think I can do so by doing this
Disallow: / Disallow: /*
However I would like to allow 4 subdirectories and everything under those subdirectories.
This is how I think it should be done
Allow: /directory_one/ Allow: /directory_one/* Allow: /directory_two/ Allow: /directory_two/* Allow: /directory_six/ Allow: /directory_six/* Allow: /about/ Allow: /about/*
So how would I go about disallowing everything starting from root but allowing only those 4 directories and everything under them?
Also if I want to allow specific directory and everything under it, do I have to declare it twice?
Will webcrawler be able to navigate to those subdirectories if root is disallowed?