1

What if I put same robots.txt file in multiple directories and folders?

User-agent: *
Disallow: /otherdirname/dirname2/

Can it'll affect anyhow ??

And how much time it generally takes to refine the search engine results?

Regards

1
4

No, robots.txt should be placed in root directory only.

Meaning http://www.example.com/robots.txt

General for search bots there is no rule to ban those directories. There are search bots, that will crawl your robots.txt directories as well.

2

First: it's useless to put a robots.txt file anywhere else than the website roots. Web crawler ONLY check at this places: http://www.yourwebsite.com/robots.txt

Your example tell to EVERY web crawler (of course, only "legal" one, bad robots don't care) to not index /otherdirname/dirname2/. It's NOT disalowing to crawl /otherdirname.

You might need MANY Disallow command to reflect your needs.

Please see the "official" web site for help: http://www.robotstxt.org/

0

If you add robot text, then search engine can't crawl that pages as you disallow those pages.

1
  • The question was about putting the file in several places, not what it does. – paulmorriss Oct 10 '12 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.