What if I put same robots.txt file in multiple directories and folders?
User-agent: *
Disallow: /otherdirname/dirname2/
Can it'll affect anyhow ??
And how much time it generally takes to refine the search engine results?
Regards
Tell me more
×
Webmasters Stack Exchange is a question and answer site for
pro webmasters. It's 100% free, no registration required.
|
|
|||||
|
migrated from stackoverflow.com Oct 9 '12 at 17:46
|
If you add robot text, then search engine can't crawl that pages as you disallow those pages. |
|||
|
|
No, robots.txt should be placed in root directory only. Meaning http://www.example.com/robots.txt General for search bots there is no rule to ban those directories. There are search bots, that will crawl your robots.txt directories as well. |
|||
|
|
|
First: it's useless to put a robots.txt file anywhere else than the website roots. Web crawler ONLY check at this places: http://www.yourwebsite.com/robots.txt Your example tell to EVERY web crawler (of course, only "legal" one, bad robots don't care) to not index /otherdirname/dirname2/. It's NOT disalowing to crawl /otherdirname. You might need MANY Disallow command to reflect your needs. Please see the "official" web site for help: http://www.robotstxt.org/ |
|||
|
|
