Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If I have a subdomain:
subdir.a.com that redirects to a.com/myfolder/

Should the robots.txt file be in:

  • subdir.a.com/robots.txt

or

  • a.com/myfolder/robots.txt
share|improve this question
up vote 4 down vote accepted

It goes in the root web directory of the subdomain. In your example physically it will be in your a.com/myfolder/ directory but over HTTP it will be accessible at http://subdir.a.com/robots.txt.

share|improve this answer

You can have multiple robots.txt files if you're managing too many subdomains as it helps you manage bots access restriction on specific folders.

If you're managing too many subdomains then it is always better to have robots.txt for each subdomain. You can also write crawling specific instructions in robots.txt of your root level directory. If you don't want to do that then you can just keep the robots.txt in the directory- "a.com/myfolder/" folder. Because you're redirecting permanently to this directory. You're doing permanent redirection to this directory so "robots.txt" should be here.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.