Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow:

User-agent: *
Allow: /
Disallow: /templates_c

My question is, in the second line I allow every folder, and in the third line I disallow /templates_c folder. So which case will be evaluated by the robots? I mean if I am allowing everything but next line disallow one folder, what will be the result?

Or should I add any folders that I want to be searched to Allow and then call Disallow?

share|improve this question
2  
"robot.txt" - I assume this is a typo, but it should be "robots.txt" (with an "s") –  w3d Jan 25 '13 at 20:49

2 Answers 2

up vote 9 down vote accepted

You really only need the disallow. Search engine crawlers will automatically assume they are allowed everywhere that isn't disallowed.

User-agent: *
Disallow: /templates_c

But to answer your question, according to Google:

At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined.

Source

Thus, your current rules should work as intended for properly behaving bots (i.e. allowed everywhere but the templates_c directory).

Note: Disallow: /templates_c will disallow all files and folders in the root that contain that within their name. If you intended to disallow only the specific templates_c directory, add a trailing slash: Disallow: /templates_c/

share|improve this answer

I'm sorry if this is wrong, I'm not familiar with the Smarty template engine.

Does the template_c folder have to be publicly accessible at all? I imagine that Smarty does use functions like include() to fetch the template code, so you could easily create a .htaccess file in the template_c folder that disallows access to any visitor. It would look like this:

DENY FROM ALL
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.