Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a sub-directory that I would like not to be visible to the search engine Web crawlers.

One way to do that is to use a robots.txt in the root directory of the server but is something that I want to avoid. The reason is that anyone knowing the website URL, could access the robots.txt contents and can explore the disallowed directories, which is something that I want to avoid.

I though a way to avoid this.
Let X be the name of the sub-directory that I want not to be indexed. One way to stop Web Crawlers indexing the X directory and at the same time to make harder for someone to identify X directory from root's robots.txt, is to add the robots.txt in the X directory instead of the root directory.

If I follow this solution I have the following questions:

  1. Will the Web Crawlers "read" the robots.txt if is in a sub-directory? (given that, a robots.txt already exist and in the root directory)
  2. If robots.txt is in the X sub-directory, then what shall I use:

    User-agent: *
    Disallow: /X/
    

    or this

    User-agent: *
    Disallow: /
    
share|improve this question
1  
I'm not sure why you want to avoid people reading your disallowed directories, but if there's sensitive data then it shouldn't just be blocked from robots but also restricted by a login or some other security. – Andrew Lott Jan 31 at 19:56
    
That is not the case, I would like one specific user to have access to it (nothing sensitive in terms of data, just for privacy) and I would like not to lock the directory. – Rafael Jan 31 at 19:58
3  
If you have privacy concerns, then you should secure the data some other way. That's not what robots.txt is for. – Andrew Lott Jan 31 at 19:59
up vote 11 down vote accepted

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site:

Where to put it

The short answer: in the top-level directory of your web server.

or on Google's help pages (emphasis mine):

A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers.

In any case, using robots.txt to hide sensitive pages from search results is a bad idea anyway, since search engines can index pages disallowed in robots.txt if other pages link to them. Or, as described on the Google help page linked above:

You should not use robots.txt as a means to hide your web pages from Google Search results. This is because other pages might point to your page, and your page could get indexed that way, avoiding the robots.txt file.

So what should you do instead?

  • You can let search engines crawl the pages (if they find them), but include a robots meta tag with the content noindex,nofollow. This will tell search engines not to index those pages even if they do find links to them, and not to follow any further links from those pages. (Of course, this will only work for HTML web pages.)

  • For non-HTML resources, you can configure your web server (e.g. using an .htaccess file) to send the X-Robots-Tag HTTP header with the same content.

  • You can set up password authentication to protect the sensitive pages. Besides protecting the pages from unauthorized human visitors, it will also effectively keep web crawlers away.

share|improve this answer
    
Fantastic, is a static HTML page, which by adding the meta tag will do the trick. Thank you. – Rafael Jan 31 at 20:05

Your robots.txt should be in the root directory and should not have any other name. According to the standard specification:

This file must be accessible via HTTP on the local URL "/robots.txt".

share|improve this answer
    
That said, the web crawlers won't look any other directory for robots.txt? – Rafael Jan 31 at 19:59
1  
Not that I've ever seen. /robots.txt is the standard, so how would search engines even know where to look otherwise? – Andrew Lott Jan 31 at 20:00
    
Without being sure, but I had the idea, that they look directory by directory and my hope was that they also look for robots.txt in other directories. – Rafael Jan 31 at 20:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.