I have a sub-directory that I would like not to be visible to the search engine Web crawlers.
One way to do that is to use a
robots.txt in the root directory of the server but is something that I want to avoid. The reason is that anyone knowing the website URL, could access the robots.txt contents and can explore the disallowed directories, which is something that I want to avoid.
I though a way to avoid this.
X be the name of the sub-directory that I want not to be indexed. One way to stop Web Crawlers indexing the
X directory and at the same time to make harder for someone to identify
X directory from root's
robots.txt, is to add the
robots.txt in the
X directory instead of the root directory.
If I follow this solution I have the following questions:
- Will the Web Crawlers "read" the
robots.txtif is in a sub-directory? (given that, a
robots.txtalready exist and in the root directory)
robots.txtis in the
Xsub-directory, then what shall I use:
User-agent: * Disallow: /X/
User-agent: * Disallow: /