Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place.

We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories in the primary domain's root directory.

I'm confused about how I would write the robots.txt for certain directories. First, I wanted to confirm I am right in understanding that for domains and subdomains, crawlers will look to the directory that acts as that urls root directory for the crawling rules(robots.txt). Also, that a directory will not be affected by a robots.txt present in their parent directory if the directory has its own domain/subdomain, and that url is the one being accessed by crawlers. (Am pretty sure, but I wanted to confirm I didnt have a fundamentally flawed understanding of robots.txt)

In the original root directory on the account(where the primary domain was directed before htaccess was put in place) what should the robots.txt contain? When crawlers look to crawl our primary domain, will they look to the original root directory for the robots.txt or will they reference the file contained in the new subdirectory where all the primary domain's site files are located? If so, what should the root's robot.txt include if anything at all.

Would I be right to include a simple 'disallow: /' for all agents, and then include more specific robots.txt files in each subdirectory with more specific instructions. Would that affect the crawling of the directory where the primary domain is now redirected?

Any help is greatly appreciated, Thanks!

share|improve this question
    
"will they reference the file contained in the new subdirectory where all the primary domain's site files are located" - yes. However I've not made that an answer because I can't understand what you are trying to achieve with your robots.txt, so I can't answer the last paragraph. Are you trying to block crawlers? Or something else? –  paulmorriss Aug 2 '11 at 15:33
    
There are certain directories I would like to disallow access from. For the file in the root directory, I dont want the directories to be indexed, I would prefer they remain private and then for each directory(since t hey all have domains/subs) to create it's own robots.txt with directory specific details. –  Jayhal Aug 2 '11 at 21:42

1 Answer 1

The first thing to say is that the crawlers don't know your file structure. They just ask for domain.com/robots.txt or sub.domain.com/robots.txt and get given the file that your server is configured to return.

So to answer

When crawlers look to crawl our primary domain, will they look to the original root directory for the robots.txt or will they reference the file contained in the new subdirectory where all the primary domain's site files are located?

they'll look in the new subdirectory.

So you don't need a robots.txt in your file system root as the crawlers don't have access to it.

However to check I'm understanding your situation properly it would be useful to know the htaccess rule you use to redirect the primary domain.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.