I am working on a php site that uses Code Igniter framework and has the following folder structure on the server.


  • application
  • public
  • sql
  • style
  • system
  • design
  • images
  • js
  • plugin





The bulleted list of items are subfolders under public_html.

The application folder contains the code consisting of views and controllers.

My question is,

should googlebot be able to crawl folders other than application?


should I Disallow all folders other than application so that no search engine bot gets to crawl other folders?

  • One thing I'm confused about this question is this questioner has put his robot.txt in root directory and I in public_html. What's the exact place?? Nov 6, 2011 at 15:01

2 Answers 2


Your images folder might be useful if you want images to appear in google image search.

Everything else should probably be disallowed.


No search engine spider would even know about the application folder and thus not be able to crawl it anyway. They would only see what URLs you provide from the home page.

Actually IIRC the .htaccess file CI supplies should only allow access to specific folders like images so the application folder should never be accessible. So you don't really need to block anything in robots.txt.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.