Here is my
User-agent: * Disallow: /head.php Disallow: /head/ Disallow: /head Disallow: /header.php Disallow: /header/ Disallow: /header Disallow: /footer.php Disallow: /footer/ Disallow: /footer Disallow: /cms.php Disallow: /cms/ Disallow: /cms Disallow: /product.php?id= Disallow: /post.php?id=
- I am wondering if it is necessary for me to disallow crawlers from indexing the
cms.phpfiles, as they are included using
include_once, in the
footerfooter sections, respectively, on almost every page of the website (shown below), and therefore they do not make for complete pages by themselves. (
cms.php, a custom CMS, is included before any HTML on every page.)
<html> <head> <!-- TEMPLATE --> <?php include_once('head.php') ?> </head> <body> <?php include_once('header.php') ?> <!-- content goes here --> <?php include_once('footer.php') ?> </body> </html>
I've set up Apache to rewrite all requests for files ending with
.phpto their respective name without
.php, and with the optional forward slash. Is it necessary that, if I do disallow the mentioned parts of pages, also disallow their re-written URLs?
Is it good practice that I hide URLs with query strings if they have their own "custom" re-written URLS? E.g.
Note: I've posted this question on Code Review as well, but I think that this may be a more suitable place for such a problem. Thanks.