1

Here is my robots.txt file:

User-agent: *
Disallow: /head.php
Disallow: /head/
Disallow: /head
Disallow: /header.php
Disallow: /header/
Disallow: /header
Disallow: /footer.php
Disallow: /footer/
Disallow: /footer
Disallow: /cms.php
Disallow: /cms/
Disallow: /cms
Disallow: /product.php?id=
Disallow: /post.php?id=
  1. I am wondering if it is necessary for me to disallow crawlers from indexing the head.php, header.php, footer.php and cms.php files, as they are included using include_once, in the head, header and footer footer sections, respectively, on almost every page of the website (shown below), and therefore they do not make for complete pages by themselves. (cms.php, a custom CMS, is included before any HTML on every page.)
<html>
  <head>
    <!-- TEMPLATE -->
    <?php include_once('head.php') ?>
  </head>
  <body>
    <?php include_once('header.php') ?>
    <!-- content goes here -->
    <?php include_once('footer.php') ?>
  </body>
</html>
  1. I've set up Apache to rewrite all requests for files ending with .php to their respective name without .php, and with the optional forward slash. Is it necessary that, if I do disallow the mentioned parts of pages, also disallow their re-written URLs?

  2. Is it good practice that I hide URLs with query strings if they have their own "custom" re-written URLS? E.g. /product.php?id=23/products/ice-tea/

Note: I've posted this question on Code Review as well, but I think that this may be a more suitable place for such a problem. Thanks.

closed as too broad by dan Aug 10 '18 at 23:49

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Robots.txt rules are "starts with", so most of your rules are redundant. Disallow: /head covers /head/, /head.php and all the header variants as well. You can make do with just 5 rules without losing any meaning. – Stephen Ostermiller Aug 10 '18 at 22:43
  • @critical_mass Multiple questions are really not a good fit for the format of this site since they're difficult to answer and vote on. The last question has been asked many times here, click on the clean-urls tag or use the Search above. If you edit the question to at least the first two, it will be added to the Review queue for consideration to be reopened. – dan Aug 10 '18 at 23:58
  • Lastly, cross-posting to more than one Stack Exchange site is frowned upon since that spreads answers out among them. Please post questions to just one site, and then flag it to request that it be migrated to another site if you feel it belongs there instead. – dan Aug 10 '18 at 23:59
  • @StephenOstermiller thanks for that info, I'll make sure to edit it in. – critical_mass Aug 11 '18 at 23:06
  • @dan yes, I'm fully aware of that, but unfortunately I was in need of this information ASAP, so I needed to post it here as well, in the hopes of getting an answer. I'll edit it as soon as I can to fit the recommended guidelines. Thanks! – critical_mass Aug 11 '18 at 23:07