Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

What if I put same robots.txt file in multiple directories and folders?

User-agent: *
Disallow: /otherdirname/dirname2/

Can it'll affect anyhow ??

And how much time it generally takes to refine the search engine results?

Regards

share|improve this question

migrated from stackoverflow.com Oct 9 '12 at 17:46

This question came from our site for professional and enthusiast programmers.

3 Answers 3

If you add robot text, then search engine can't crawl that pages as you disallow those pages.

share|improve this answer
    
The question was about putting the file in several places, not what it does. –  paulmorriss Oct 10 '12 at 13:11

No, robots.txt should be placed in root directory only.

Meaning http://www.example.com/robots.txt

General for search bots there is no rule to ban those directories. There are search bots, that will crawl your robots.txt directories as well.

share|improve this answer

First: it's useless to put a robots.txt file anywhere else than the website roots. Web crawler ONLY check at this places: http://www.yourwebsite.com/robots.txt

Your example tell to EVERY web crawler (of course, only "legal" one, bad robots don't care) to not index /otherdirname/dirname2/. It's NOT disalowing to crawl /otherdirname.

You might need MANY Disallow command to reflect your needs.

Please see the "official" web site for help: http://www.robotstxt.org/

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.