Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

27
votes
6answers
397 views

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file? Or can the lack of one be harmful?
4
votes
2answers
352 views

What is the best configuration for robots.txt

What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders? Is there a general setup that should always be used?