Preventing search engines to crawl on specific folders

Question

I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow:

User-agent: *
Allow: /
Disallow: /templates_c

My question is, in the second line I allow every folder, and in the third line I disallow /templates_c folder. So which case will be evaluated by the robots? I mean if I am allowing everything but next line disallow one folder, what will be the result?

Or should I add any folders that I want to be searched to Allow and then call Disallow?

Answer 1

You really only need the disallow. Search engine crawlers will automatically assume they are allowed everywhere that isn't disallowed.

User-agent: *
Disallow: /templates_c

But to answer your question, according to Google:

At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined.

Source

Thus, your current rules should work as intended for properly behaving bots (i.e. allowed everywhere but the templates_c directory).

Note: Disallow: /templates_c will disallow all files and folders in the root that contain that within their name. If you intended to disallow only the specific templates_c directory, add a trailing slash: Disallow: /templates_c/

Answer 2

I'm sorry if this is wrong, I'm not familiar with the Smarty template engine.

Does the template_c folder have to be publicly accessible at all? I imagine that Smarty does use functions like include() to fetch the template code, so you could easily create a .htaccess file in the template_c folder that disallows access to any visitor. It would look like this:

DENY FROM ALL