Quick and simple question.

I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content using php includes. These php include files are in a separate folder on the server and contain the core content for the individual product pages.

I just wanted to ask, if I use robots.txt or .htaccess to prevent crawling of the directory that holds the php content files, will there be no issue crawling the html pages which include these files?

What I want to achieve is have the html files indexed with the php content included in them, but I don't want visitors landing on the php content pages, nor have these php files indexed as duplicate content. Just clarification needed as to whether it is safe to block spiders from accessing the php folder, without this affecting the html files being indexed with the included content.

Is this the best way to do things? Or should I just leave the content php files to be crawled?

3 Answers 3


You really shouldn't leave the PHP files to be crawled. It is a security hole and those files have no special reason to be crawled.

Block spiders with robots and disable directory list in that includes folder.


Move your include files outside of the web directory. The web server never needs to serve up those files.

For example, if you have your website in /somedir/www and your includes in /somedir/www/include, move them to /somedir/include.


I'd also add that PHP is processed server side so if the files are access directly and the source is viewed its blank. When you include them into HTML files you should be using your .htaccess file to render PHP in .html files. Then when the source of the .html is viewed it'll include the PHP and not look any different. You should be using <?php include("file.php"); ?> or some variation. Not SSI's


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.