Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
There is no tag wiki for this tag … yet!
Tag wikis help introduce newcomers to the tag. They contain an overview of the topic defined by the tag, along with guidelines on its usage.
All registered users may propose new tag wikis.
(Note that if you have less than 20000 reputation, your tag wiki will be peer reviewed before it is published.)