0

We have two sites mainsite.com and mainsite.com/de where /de shows the german site.

We rely on 1 robots.txt file. It contains all the disallow links for mainsite.com that we set-up and copied it to mainsite.com/de.

Disallow: /media/sales/
Disallow: /media/tmp/

Question: doesn't the first slash (/) point to the root? How should we set up this robots to work also on the subdirectory?

Improve this question

1 Answer 1

Reset to default
3

First of all /de is not a sub-domain- it is a sub-directory. A sub-domain would be something like de.example.com.

And yes. You will need /de/media/sales/ and /de/media/tmp/. You will always be working from the root directory.

Lastly, the robots.txt file must be in the root directory. You cannot create a robots.txt in the /de directory. It will never be seen. So your robots.txt file in the root will reference:

/media/sales/
/media/tmp/
/de/media/sales/
/de/media/tmp/
Improve this answer
3
  • hmmm thanks. Typo. And there is no way to make the first directory variable so it would work for all /subdirectory/ sites?
    – ShirtsofHolland
    Jul 20, 2015 at 15:43
  • @ShirtsofHolland I am not sure I understand the question. However, if I got it right, you may be able to use wild-cards, but generally speaking, you are better off specifying all directories specifically to make sure that bots fully understand. Here is a simple explanation: robotstxt.org/robotstxt.html
    – closetnoc
    Jul 20, 2015 at 15:48
  • 3
    In terms of "wildcards", you can do something like /*/media/sales/ for all subdirectories. But note that this is an extension to the "standard". closetnoc's comment still applies. developers.google.com/webmasters/control-crawl-index/docs/…
    – MrWhite
    Jul 20, 2015 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.