I have a sub-directory that I would like to hide from the search engine web crawlers.
One way to do this is to use a
robots.txt in the root directory of the server (the standard way). However, anyone who knows the website URL and has some basic web knowledge can access the robots.txt contents and can spot the disallowed directories.
I thought a way to avoid this, but I am not sure if will work.
X be the name of the sub-directory that I want to exclude. One way to stop Web Crawlers indexing the
X directory and at the same time to make harder for someone to identify
X directory from root's
robots.txt, is to add the
robots.txt in the
X directory instead of the root directory.
If I follow this solution I have the following questions:
- Will the Web Crawlers find the
robots.txtin the sub-directory? (given that, a
robots.txtalready exist and in the root directory as well)
robots.txtis in the
Xsub-directory, then should I use relative or absolute paths?:
User-agent: * Disallow: /X/
User-agent: * Disallow: /