1

If I have a subdomain:
subdir.a.com that redirects to a.com/myfolder/

Should the robots.txt file be in:

  • subdir.a.com/robots.txt

or

  • a.com/myfolder/robots.txt
4

It goes in the root web directory of the subdomain. In your example physically it will be in your a.com/myfolder/ directory but over HTTP it will be accessible at http://subdir.a.com/robots.txt.

0

You can have multiple robots.txt files if you're managing too many subdomains as it helps you manage bots access restriction on specific folders.

If you're managing too many subdomains then it is always better to have robots.txt for each subdomain. You can also write crawling specific instructions in robots.txt of your root level directory. If you don't want to do that then you can just keep the robots.txt in the directory- "a.com/myfolder/" folder. Because you're redirecting permanently to this directory. You're doing permanent redirection to this directory so "robots.txt" should be here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.