Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 2 down vote favorite

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server.

public_html

  • application
  • public
  • sql
  • style
  • system
  • design
  • images
  • js
  • plugin

.htaccess

robots.txt

sitemap.xml

index.php

The bulleted list of items are subfolders under public_html.

The application folder contains the code consisting of views and controllers.

My question is,

should googlebot be able to crawl folders other than application?

OR

should I Disallow all folders other than application so that no search engine bot gets to crawl other folders?

share|improve this question
One thing I'm confused about this question is this questioner has put his robot.txt in root directory and I in public_html. What's the exact place?? – Santosh Kumar Nov 6 '11 at 15:01

2 Answers

up vote 2 down vote

Your images folder might be useful if you want images to appear in google image search.

Everything else should probably be disallowed.

share|improve this answer
up vote 0 down vote

No search engine spider would even know about the application folder and thus not be able to crawl it anyway. They would only see what URLs you provide from the home page.

Actually IIRC the .htaccess file CI supplies should only allow access to specific folders like images so the application folder should never be accessible. So you don't really need to block anything in robots.txt.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.