1

What if I put same robots.txt file in multiple directories and folders?

User-agent: *
Disallow: /otherdirname/dirname2/

Can it'll affect anyhow ??

And how much time it generally takes to refine the search engine results?

Regards

Improve this question
1
4

No, robots.txt should be placed in root directory only.

Meaning http://www.example.com/robots.txt

General for search bots there is no rule to ban those directories. There are search bots, that will crawl your robots.txt directories as well.

Improve this answer
2

First: it's useless to put a robots.txt file anywhere else than the website roots. Web crawler ONLY check at this places: http://www.yourwebsite.com/robots.txt

Your example tell to EVERY web crawler (of course, only "legal" one, bad robots don't care) to not index /otherdirname/dirname2/. It's NOT disalowing to crawl /otherdirname.

You might need MANY Disallow command to reflect your needs.

Please see the "official" web site for help: http://www.robotstxt.org/

Improve this answer
0

If you add robot text, then search engine can't crawl that pages as you disallow those pages.

Improve this answer
1
  • The question was about putting the file in several places, not what it does.
    – paulmorriss
    Oct 10 '12 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.