Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

If I have a subdomain:
subdir.a.com that redirects to a.com/myfolder/
should the robots.txt file be in:
subdir.a.com/robots.txt or in
a.com/myfolder/robots.txt

share|improve this question
add comment

2 Answers

up vote 4 down vote accepted

It goes in the root web directory of the subdomain. In your example physically it will be in your a.com/myfolder/ directory but over HTTP it will be accessible at http://subdir.a.com/robots.txt.

share|improve this answer
add comment

You can have multiple robots.txt files if you're managing too many subdomains as it helps you manage bots access restriction on specific folders.

If you're managing too many subdomains then it is always better to have robots.txt for each subdomain. You can also write crawling specific instructions in robots.txt of your root level directory. If you don't want to do that then you can just keep the robots.txt in the directory- "a.com/myfolder/" folder. Because you're redirecting permanently to this directory. You're doing permanent redirection to this directory so "robots.txt" should be here.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.