Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders?

Is there a general setup that should always be used?

share|improve this question
It's "robots.txt" (plural) –  Adam Tuttle Jul 8 '10 at 19:41
Thanks. I have fixed the title and question to reflect this. –  Mike Jul 8 '10 at 19:47

2 Answers 2

up vote 4 down vote accepted

Google Webmaster tools has a Section called "Crawler access"

This section allows you very easily to create your robots.txt

For example to allow everything except blog a folder called test your robot.txt would look something like

User-agent: *
Disallow: /Test
Allow: /
share|improve this answer
Make sure you also follow the link in Jason's answer for more information. webmasters.stackexchange.com/questions/89/… –  JasonBirch Jul 11 '10 at 5:51

Here's everything you need to know about the robots.txt file

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.