We have two sites mainsite.com and mainsite.com/de where /de shows the german site.

We rely on 1 robots.txt file. It contains all the disallow links for mainsite.com that we set-up and copied it to mainsite.com/de.

Disallow: /media/sales/
Disallow: /media/tmp/

Question: doesn't the first slash (/) point to the root? How should we set up this robots to work also on the subdirectory?


First of all /de is not a sub-domain- it is a sub-directory. A sub-domain would be something like de.example.com.

And yes. You will need /de/media/sales/ and /de/media/tmp/. You will always be working from the root directory.

Lastly, the robots.txt file must be in the root directory. You cannot create a robots.txt in the /de directory. It will never be seen. So your robots.txt file in the root will reference:

  • hmmm thanks. Typo. And there is no way to make the first directory variable so it would work for all /subdirectory/ sites? Jul 20 '15 at 15:43
  • @ShirtsofHolland I am not sure I understand the question. However, if I got it right, you may be able to use wild-cards, but generally speaking, you are better off specifying all directories specifically to make sure that bots fully understand. Here is a simple explanation: robotstxt.org/robotstxt.html
    – closetnoc
    Jul 20 '15 at 15:48
  • 3
    In terms of "wildcards", you can do something like /*/media/sales/ for all subdirectories. But note that this is an extension to the "standard". closetnoc's comment still applies. developers.google.com/webmasters/control-crawl-index/docs/…
    – MrWhite
    Jul 20 '15 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.