Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Quick and simple question.

I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content using php includes. These php include files are in a separate folder on the server and contain the core content for the individual product pages.

I just wanted to ask, if I use robots.txt or .htaccess to prevent crawling of the directory that holds the php content files, will there be no issue crawling the html pages which include these files?

What I want to achieve is have the html files indexed with the php content included in them, but I don't want visitors landing on the php content pages, nor have these php files indexed as duplicate content. Just clarification needed as to whether it is safe to block spiders from accessing the php folder, without this affecting the html files being indexed with the included content.

Is this the best way to do things? Or should I just leave the content php files to be crawled?

share|improve this question
up vote 2 down vote accepted

You really shouldn't leave the PHP files to be crawled. It is a security hole and those files have no special reason to be crawled.

Block spiders with robots and disable directory list in that includes folder.

share|improve this answer

Move your include files outside of the web directory. The web server never needs to serve up those files.

For example, if you have your website in /somedir/www and your includes in /somedir/www/include, move them to /somedir/include.

share|improve this answer
Thanks for the help. – 21stcn Nov 8 '12 at 19:02

I'd also add that PHP is processed server side so if the files are access directly and the source is viewed its blank. When you include them into HTML files you should be using your .htaccess file to render PHP in .html files. Then when the source of the .html is viewed it'll include the PHP and not look any different. You should be using <?php include("file.php"); ?> or some variation. Not SSI's

share|improve this answer
Thanks for the help. – 21stcn Nov 8 '12 at 19:03

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.