Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders?

Is there a general setup that should always be used?

share|improve this question
1  
It's "robots.txt" (plural) –  Adam Tuttle Jul 8 '10 at 19:41
    
Thanks. I have fixed the title and question to reflect this. –  Mike Jul 8 '10 at 19:47
add comment

2 Answers

up vote 4 down vote accepted

Google Webmaster tools has a Section called "Crawler access"

This section allows you very easily to create your robots.txt

For example to allow everything except blog a folder called test your robot.txt would look something like

User-agent: *
Disallow: /Test
Allow: /
share|improve this answer
    
Make sure you also follow the link in Jason's answer for more information. webmasters.stackexchange.com/questions/89/… –  JasonBirch Jul 11 '10 at 5:51
add comment

Here's everything you need to know about the robots.txt file

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.