2

I feel embarrassed asking this question because I've been doing this so long, but I guess you're never too old to learn.

Environment is Rocky Linux 8.8, Virtualmin Pro 7.7, Apache 2.4.37, PHP 8.1 (FPM). Site is hand-coded if that matters. Every page is its own php file and they all live in the Web root, which is (/public_html)/.

The problem became apparent in Google Search Console, where Google was choosing "duplicate" pages in a non-existent /index directory as canonical. In other words, they would choose

https://example.com/index/index.php
https://example.com/index/someotherpage.php
etc.

Instead of

https://example.com/index.php
https://example.com/someotherpage.php
etc.

I figured it was just the usual Google weirdness until I manually accessed the pages in the non-existent /index directory (yes, I checked, and it didn't exist) with a browser, and they came up, albeit with no styling and no JavaScript because the paths to the stylesheets and scripts are relative.

What I eventually did (just a few minutes ago, so it's too soon to know whether Google accepts it) is created an /index directory and placed an .htaccess file in it:

RewriteEngine on
RewriteRule ^(.*)$ https://example.com/$1 [R=301,L]

That redirects the requests to the root when I use it in a browser. I have no idea whether Google with have issues with it.

Why is there an /index directory that doesn't (or didn't) actually exist? Shouldn't requests to a non-existent directory just throw a 404? Or is this some new thing because so few people hand-code anymore that /index is assumed to be the Web root instead of / ?

EDIT 1

.htaccess for the root of the site:

RewriteEngine On

# redirects from example.com to example.aero
RewriteCond %{HTTP_HOST} example.com$ [NC]
RewriteRule ^(.*)$ https://example.aero/$1 [L,R=301]

# removes www
RewriteBase /
RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteRule ^(.*)$ https://%1/$1 [L,R=301]

ErrorDocument 401 https://example.aero/401.php
ErrorDocument 403 https://example.aero/403.php
ErrorDocument 404 https://example.aero/404.php

# GZIP compression for text files: HTML, CSS, JS, Text, XML, fonts
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
AddOutputFilterByType DEFLATE application/x-font
AddOutputFilterByType DEFLATE application/x-font-opentype
AddOutputFilterByType DEFLATE application/x-font-otf
AddOutputFilterByType DEFLATE application/x-font-truetype
AddOutputFilterByType DEFLATE application/x-font-ttf
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE font/opentype
AddOutputFilterByType DEFLATE font/otf
AddOutputFilterByType DEFLATE font/ttf
AddOutputFilterByType DEFLATE image/svg+xml
AddOutputFilterByType DEFLATE image/x-icon
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/xml
</IfModule>

<filesMatch ".(ico|pdf|jpg|jpeg|png|gif|js|swf|webp)$">
Header set Cache-Control "max-age=31536000, public, must-revalidate"
</filesMatch>

<filesMatch ".(xml|txt|css)$">
Header set Cache-Control "max-age=14400, public, must-revalidate"
</filesMatch>

The canonical domain is on the .aero TLD, but I also own the .com and redirect it .aero.

The previously-mentioned .htaccess file in /index/ is the only thing in that directory, which did not exist until I created it to try to fix this issue. I am wondering if I shouldn't change it to redirect all requests on that directory to the site's 404 page?

EDIT 2

Based on MrWhite's comment, I checked and found that MultiViews is in fact enabled in the Apache configuration, presumably by Virtualmin as a default. I don't want to mess with it because there may be some reason it's enabled; but disabling the option in .htaccess for the root of the domain corrects the behavior and causes requests on files in /index/ to return the 404 page.

I will make that change in this site and others affected by the same issue while I further explore why MultiViews is enabled in Apache. I don't want to disable it system-wide until I know why it was enabled in the first place. There may be a good reason.

4
  • I'm not sure this question can be answered with bespoke code. What does your .htaccess say (if it exists?) Is there any load-balancer being used?
    – davidgo
    Commented Jun 12, 2023 at 6:30
  • To clarify, a request for /index/someotherpage.php actually serves/someotherpage.php (from the root) and not /index.php? The later would happen if MultiViews had been enabled (perhaps in the server config). However, how did Google find these URLs in the first place (that suggests some other misconfig)? "Or is this some new thing" - nope.
    – MrWhite
    Commented Jun 12, 2023 at 8:38
  • Requests for any files in the previously-nonexistent /index/ directory served the pages from that directory, and made all the links relative to itself (that is, in the /index/ directory), but without styling or scripts. I created the /index/ directory and added the .htaccess file within it with the single directive to redirect to the root, which is working. I have no idea why that was even necessary, but I'm going to check the Apache config. Maybe Virtualmin enabled MultiViews during the Apache setup. Commented Jun 12, 2023 at 8:51
  • 1
    MultiViews was in fact the problem, MrWhite. I have edited my question with the details. Thank you. Commented Jun 12, 2023 at 9:24

1 Answer 1

1

As suggested by MrWhite, MultiViews being enabled server-wide in the Apache configuration was in fact the problem. I've disabled it in .htaccess for now, and the non-existent pages are now through 404'S as they should. Hopefully this will eliminate Google treating them as duplicates.

I suppose someone posted a bad link to the site at some point and Google followed it. There's no internal link pointing to the non-existent directory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.