1

Context: sith.edu is mapping the subdirectory sith.edu/rogueone via reverse proxy (/rogueone is using a different CMS and is hosted elsewhere).

My question: If sith.edu adds disallow /rogueone to their robots.txt, can /rogueone override that effect using their own robots.txt? Or will Google honor the parent domain's robots.txt disallow rule?

2

For each domain, Google will follow the robots.txt at the root of the domain, nothing more and nothing less. Google will be unaware of any reverse proxies.

So if sith.edu adds disallow /rogueone to their robots.txt, Google will not crawl any of the URLs under sith.edu/rogueone (though it could crawl the same content at different URLs under the real rogueone domain).

In your case, if rogueone changes their robots.txt, it will be served up at sith.edu/rogueone/robots.txt, and since Google does not recognize robots.txt files unless they are at the root of the domain, it cannot change any of the robots.txt rules in place for sith.edu.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.