Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

What if I put same robots.txt file in multiple directories and folders?

User-agent: *
Disallow: /otherdirname/dirname2/

Can it'll affect anyhow ??

And how much time it generally takes to refine the search engine results?


share|improve this question

migrated from stackoverflow.com Oct 9 '12 at 17:46

This question came from our site for professional and enthusiast programmers.

No, robots.txt should be placed in root directory only.

Meaning http://www.example.com/robots.txt

General for search bots there is no rule to ban those directories. There are search bots, that will crawl your robots.txt directories as well.

share|improve this answer

First: it's useless to put a robots.txt file anywhere else than the website roots. Web crawler ONLY check at this places: http://www.yourwebsite.com/robots.txt

Your example tell to EVERY web crawler (of course, only "legal" one, bad robots don't care) to not index /otherdirname/dirname2/. It's NOT disalowing to crawl /otherdirname.

You might need MANY Disallow command to reflect your needs.

Please see the "official" web site for help: http://www.robotstxt.org/

share|improve this answer

If you add robot text, then search engine can't crawl that pages as you disallow those pages.

share|improve this answer
The question was about putting the file in several places, not what it does. – paulmorriss Oct 10 '12 at 13:11

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.