Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server.

public_html

  • application
  • public
  • sql
  • style
  • system
  • design
  • images
  • js
  • plugin

.htaccess

robots.txt

sitemap.xml

index.php

The bulleted list of items are subfolders under public_html.

The application folder contains the code consisting of views and controllers.

My question is,

should googlebot be able to crawl folders other than application?

OR

should I Disallow all folders other than application so that no search engine bot gets to crawl other folders?

share|improve this question
    
One thing I'm confused about this question is this questioner has put his robot.txt in root directory and I in public_html. What's the exact place?? –  Santosh Kumar Nov 6 '11 at 15:01
add comment

2 Answers 2

No search engine spider would even know about the application folder and thus not be able to crawl it anyway. They would only see what URLs you provide from the home page.

Actually IIRC the .htaccess file CI supplies should only allow access to specific folders like images so the application folder should never be accessible. So you don't really need to block anything in robots.txt.

share|improve this answer
add comment

Your images folder might be useful if you want images to appear in google image search.

Everything else should probably be disallowed.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.