1

I have a weird issue where either Nginx or Apache (I can't tell which - but I'm 90% sure it's Apache - Nginx is reverse proxying) is matching a partial URL to a full, random URL.

For example: https://example.com/php

Is being rewritten/redirected to: https://example.com/php-full-article-url

But as far as I'm aware, I don't have anything in my config's that would result in this behaviour!

Nginx conf:

server { 
    server_name example.com; 

    root /var/www/html/examplecom/; 
    index index.php index.html index.htm;

    set $skip_cache 0;
    set $skip_reason "";
    proxy_cache_key $scheme$host$request_uri;

    if ($http_cookie ~* "comment_author_|wordpress_(?!test_cookie)|wp-postpass_" ) {
            set $skip_cache 1;
            set $skip_reason Cookie; 
    }

    if ($request_uri ~* "/wp-admin/|/xmlrpc.php|wp-.*.php|/feed/|sitemap(_index)?.xml") {
        set $skip_cache 1;
        set $skip_reason URI; 
    }
    add_header Cache-BYPASS-Reason  $skip_reason;


    location / {
        try_files                   $uri $uri/ /index.php?$args;
        proxy_buffering             off;
        proxy_cache                 edge-cache;
        proxy_cache_revalidate      on;
        proxy_cache_use_stale       error timeout invalid_header updating http_500 http_502 http_503 http_504;
        proxy_cache_bypass          $skip_cache;
        proxy_no_cache              $skip_cache;
        proxy_cache_valid           200 301 302 500m;
        proxy_cache_valid           404 1m;
        add_header X-Cache-Status   $upstream_cache_status;
        #add_header X-Handled-By    $proxy_host;
        add_header                  Cache-BYPASS-Reason     $skip_reason;
    }

    location ~ \.php$ {
        proxy_set_header            X-Real-IP $remote_addr;
        proxy_set_header            X-Forwarded-For $remote_addr;
        proxy_set_header            X-Forwarded-Host $host;
        proxy_set_header            Host $host;
        proxy_set_header            X-Forwarded-Proto $scheme;
        proxy_pass                  http://127.0.0.1:8080$request_uri;
        add_header                  Front-End-Https on;
        proxy_redirect              off;
        proxy_buffering             off;
        proxy_cache                 edge-cache;
        proxy_cache_revalidate      on;
        proxy_cache_use_stale       error timeout invalid_header updating http_500 http_502 http_503 http_504;
        proxy_cache_bypass          $skip_cache;
        proxy_no_cache              $skip_cache;
        proxy_cache_valid           200 301 302 500m;
        proxy_cache_valid           404 1m;
        add_header X-Cache-Status   $upstream_cache_status;
        #add_header X-Handled-By    $proxy_host;
        add_header                  Cache-BYPASS-Reason     $skip_reason;
    }

    location ~ /\.ht {
        deny all;
    }

    listen 443 ssl http2; # managed by Certbot
    #listen [::]:443 ssl http2 ipv6only=off;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot
    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    #ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_ciphers TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }
    location = /robots.txt {
        log_not_found off;
        access_log off;
        allow all;
    }
    location ~* .(css|gif|ico|jpeg|jpg|js|png)$ {
        expires                     max;
        log_not_found               off;
        proxy_set_header            X-Real-IP $remote_addr;
        proxy_set_header            X-Forwarded-For $remote_addr;
        proxy_set_header            X-Forwarded-Host $host;
        proxy_set_header            Host $host;
        proxy_set_header            X-Forwarded-Proto $scheme;
        proxy_redirect              off;
        proxy_buffering             off;
        proxy_cache                 edge-cache;
        proxy_cache_revalidate      on;
        proxy_cache_use_stale       error timeout invalid_header updating http_500 http_502 http_503 http_504;
        proxy_cache_bypass          $skip_cache;
        proxy_no_cache              $skip_cache;
        proxy_cache_valid           200 301 302 500m;
        proxy_cache_valid           404 1m;
        add_header                  X-Cache-Status  $upstream_cache_status;
        add_header                  Cache-BYPASS-Reason     $skip_reason;
        proxy_pass                  http://127.0.0.1:8080$request_uri;
    }
}
server {

    if ($host = example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = www.example.com) {
        return 301 https://example.com$request_uri; 
    }

    if ($host = dev.example.com) {
        return 301 https://example.com$request_uri; 
    }

    # managed by Certbot

    listen   80;

    server_name example.com;
    return 404; # managed by Certbot

}

Apache conf:

<VirtualHost 127.0.0.1:8080>
    #ServerName example.com

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html/examplecom/

    #LogLevel info ssl:warn

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    #Include conf-available/serve-cgi-bin.conf

    Header unset ETag
    FileETag None

    CacheQuickHandler off

    #CacheRoot               /var/cache/apache2/mod_cache_disk
    #CacheEnable            disk /
    #CacheDirLevels             2
    #CacheDirLength             1
    #CacheMaxFileSize        2000000

    #CacheIgnoreNoLastMod    On
    #CacheDefaultExpire      7200
    #CacheIgnoreCacheControl On
    #CacheLastModifiedFactor 0.5
    CacheIgnoreHeaders      Set-Cookie Cookie
    #CacheHeader            on

    #CacheLock              on

    CacheDisable            /wp-admin
    CacheDisable            /wp-login.php
    CacheDisable            /wp-cron.php

    #SetOutputFilter        CACHE

    <FilesMatch \.php$>
        SetHandler "proxy:unix:/var/run/php/php7.2-fpm.sock|fcgi://localhost/"
    </FilesMatch>

</VirtualHost>

Wordpress .htaccess:

# BEGIN W3TC CDN
<IfModule mod_headers.c>
    Header set Access-Control-Allow-Origin "*"
</IfModule>
# END W3TC CDN

# BEGIN W3TC Browser Cache
<IfModule mod_mime.c>
    AddType text/css .css
    AddType text/x-component .htc
    AddType application/x-javascript .js
    AddType application/javascript .js2
    AddType text/javascript .js3
    AddType text/x-js .js4
    AddType text/html .html .htm
    AddType text/richtext .rtf .rtx
    AddType image/svg+xml .svg
    AddType text/plain .txt
    AddType text/xsd .xsd
    AddType text/xsl .xsl
    AddType text/xml .xml
    AddType video/asf .asf .asx .wax .wmv .wmx
    AddType video/avi .avi
    AddType image/bmp .bmp
    AddType application/java .class
    AddType video/divx .divx
    AddType application/msword .doc .docx
    AddType application/vnd.ms-fontobject .eot
    AddType application/x-msdownload .exe
    AddType image/gif .gif
    AddType application/x-gzip .gz .gzip
    AddType image/x-icon .ico
    AddType image/jpeg .jpg .jpeg .jpe
    AddType image/webp .webp
    AddType application/json .json
    AddType application/vnd.ms-access .mdb
    AddType audio/midi .mid .midi
    AddType video/quicktime .mov .qt
    AddType audio/mpeg .mp3 .m4a
    AddType video/mp4 .mp4 .m4v
    AddType video/mpeg .mpeg .mpg .mpe
    AddType video/webm .webm
    AddType application/vnd.ms-project .mpp
    AddType application/x-font-otf .otf
    AddType application/vnd.ms-opentype ._otf
    AddType application/vnd.oasis.opendocument.database .odb
    AddType application/vnd.oasis.opendocument.chart .odc
    AddType application/vnd.oasis.opendocument.formula .odf
    AddType application/vnd.oasis.opendocument.graphics .odg
    AddType application/vnd.oasis.opendocument.presentation .odp
    AddType application/vnd.oasis.opendocument.spreadsheet .ods
    AddType application/vnd.oasis.opendocument.text .odt
    AddType audio/ogg .ogg
    AddType application/pdf .pdf
    AddType image/png .png
    AddType application/vnd.ms-powerpoint .pot .pps .ppt .pptx
    AddType audio/x-realaudio .ra .ram
    AddType image/svg+xml .svg .svgz
    AddType application/x-shockwave-flash .swf
    AddType application/x-tar .tar
    AddType image/tiff .tif .tiff
    AddType application/x-font-ttf .ttf .ttc
    AddType application/vnd.ms-opentype ._ttf
    AddType audio/wav .wav
    AddType audio/wma .wma
    AddType application/vnd.ms-write .wri
    AddType application/font-woff .woff
    AddType application/font-woff2 .woff2
    AddType application/vnd.ms-excel .xla .xls .xlsx .xlt .xlw
    AddType application/zip .zip
</IfModule>
<IfModule mod_expires.c>
    ExpiresActive On
    ExpiresByType text/css A31536000
    ExpiresByType text/x-component A31536000
    ExpiresByType application/x-javascript A31536000
    ExpiresByType application/javascript A31536000
    ExpiresByType text/javascript A31536000
    ExpiresByType text/x-js A31536000
    ExpiresByType text/html A3600
    ExpiresByType text/richtext A3600
    ExpiresByType image/svg+xml A3600
    ExpiresByType text/plain A3600
    ExpiresByType text/xsd A3600
    ExpiresByType text/xsl A3600
    ExpiresByType text/xml A3600
    ExpiresByType video/asf A31536000
    ExpiresByType video/avi A31536000
    ExpiresByType image/bmp A31536000
    ExpiresByType application/java A31536000
    ExpiresByType video/divx A31536000
    ExpiresByType application/msword A31536000
    ExpiresByType application/vnd.ms-fontobject A31536000
    ExpiresByType application/x-msdownload A31536000
    ExpiresByType image/gif A31536000
    ExpiresByType application/x-gzip A31536000
    ExpiresByType image/x-icon A31536000
    ExpiresByType image/jpeg A31536000
    ExpiresByType image/webp A31536000
    ExpiresByType application/json A31536000
    ExpiresByType application/vnd.ms-access A31536000
    ExpiresByType audio/midi A31536000
    ExpiresByType video/quicktime A31536000
    ExpiresByType audio/mpeg A31536000
    ExpiresByType video/mp4 A31536000
    ExpiresByType video/mpeg A31536000
    ExpiresByType video/webm A31536000
    ExpiresByType application/vnd.ms-project A31536000
    ExpiresByType application/x-font-otf A31536000
    ExpiresByType application/vnd.ms-opentype A31536000
    ExpiresByType application/vnd.oasis.opendocument.database A31536000
    ExpiresByType application/vnd.oasis.opendocument.chart A31536000
    ExpiresByType application/vnd.oasis.opendocument.formula A31536000
    ExpiresByType application/vnd.oasis.opendocument.graphics A31536000
    ExpiresByType application/vnd.oasis.opendocument.presentation A31536000
    ExpiresByType application/vnd.oasis.opendocument.spreadsheet A31536000
    ExpiresByType application/vnd.oasis.opendocument.text A31536000
    ExpiresByType audio/ogg A31536000
    ExpiresByType application/pdf A31536000
    ExpiresByType image/png A31536000
    ExpiresByType application/vnd.ms-powerpoint A31536000
    ExpiresByType audio/x-realaudio A31536000
    ExpiresByType image/svg+xml A31536000
    ExpiresByType application/x-shockwave-flash A31536000
    ExpiresByType application/x-tar A31536000
    ExpiresByType image/tiff A31536000
    ExpiresByType application/x-font-ttf A31536000
    ExpiresByType application/vnd.ms-opentype A31536000
    ExpiresByType audio/wav A31536000
    ExpiresByType audio/wma A31536000
    ExpiresByType application/vnd.ms-write A31536000
    ExpiresByType application/font-woff A31536000
    ExpiresByType application/font-woff2 A31536000
    ExpiresByType application/vnd.ms-excel A31536000
    ExpiresByType application/zip A31536000
</IfModule>
<IfModule mod_brotli.c>
    <IfModule mod_filter.c>
        AddOutputFilterByType BROTLI_COMPRESS text/css text/x-component application/x-javascript application/javascript text/javascript text/x-js text/html text/richtext image/svg+xml text/plain text/xsd text/xsl text/xml image/bmp application/java application/msword application/vnd.ms-fontobject application/x-msdownload image/x-icon image/webp application/json application/vnd.ms-access video/webm application/vnd.ms-project application/x-font-otf application/vnd.ms-opentype application/vnd.oasis.opendocument.database application/vnd.oasis.opendocument.chart application/vnd.oasis.opendocument.formula application/vnd.oasis.opendocument.graphics application/vnd.oasis.opendocument.presentation application/vnd.oasis.opendocument.spreadsheet application/vnd.oasis.opendocument.text audio/ogg application/pdf application/vnd.ms-powerpoint image/svg+xml application/x-shockwave-flash image/tiff application/x-font-ttf application/vnd.ms-opentype audio/wav application/vnd.ms-write application/font-woff application/font-woff2 application/vnd.ms-excel
    <IfModule mod_mime.c>
        # BROTLI_COMPRESS by extension
        AddOutputFilter BROTLI_COMPRESS js css htm html xml
    </IfModule>
    </IfModule>
</IfModule>
<IfModule mod_deflate.c>
    <IfModule mod_filter.c>
        AddOutputFilterByType DEFLATE text/css text/x-component application/x-javascript application/javascript text/javascript text/x-js text/html text/richtext image/svg+xml text/plain text/xsd text/xsl text/xml image/bmp application/java application/msword application/vnd.ms-fontobject application/x-msdownload image/x-icon image/webp application/json application/vnd.ms-access video/webm application/vnd.ms-project application/x-font-otf application/vnd.ms-opentype application/vnd.oasis.opendocument.database application/vnd.oasis.opendocument.chart application/vnd.oasis.opendocument.formula application/vnd.oasis.opendocument.graphics application/vnd.oasis.opendocument.presentation application/vnd.oasis.opendocument.spreadsheet application/vnd.oasis.opendocument.text audio/ogg application/pdf application/vnd.ms-powerpoint image/svg+xml application/x-shockwave-flash image/tiff application/x-font-ttf application/vnd.ms-opentype audio/wav application/vnd.ms-write application/font-woff application/font-woff2 application/vnd.ms-excel
    <IfModule mod_mime.c>
        # DEFLATE by extension
        AddOutputFilter DEFLATE js css htm html xml
    </IfModule>
    </IfModule>
</IfModule>
<FilesMatch "\.(css|htc|less|js|js2|js3|js4|CSS|HTC|LESS|JS|JS2|JS3|JS4)$">
    FileETag MTime Size
    <IfModule mod_headers.c>
         Header unset Set-Cookie
    </IfModule>
</FilesMatch>
<FilesMatch "\.(html|htm|rtf|rtx|svg|txt|xsd|xsl|xml|HTML|HTM|RTF|RTX|SVG|TXT|XSD|XSL|XML)$">
    FileETag MTime Size
    <IfModule mod_headers.c>
        Header append Vary User-Agent env=!dont-vary
    </IfModule>
</FilesMatch>
<FilesMatch "\.(asf|asx|wax|wmv|wmx|avi|bmp|class|divx|doc|docx|eot|exe|gif|gz|gzip|ico|jpg|jpeg|jpe|webp|json|mdb|mid|midi|mov|qt|mp3|m4a|mp4|m4v|mpeg|mpg|mpe|webm|mpp|otf|_otf|odb|odc|odf|odg|odp|ods|odt|ogg|pdf|png|pot|pps|ppt|pptx|ra|ram|svg|svgz|swf|tar|tif|tiff|ttf|ttc|_ttf|wav|wma|wri|woff|woff2|xla|xls|xlsx|xlt|xlw|zip|ASF|ASX|WAX|WMV|WMX|AVI|BMP|CLASS|DIVX|DOC|DOCX|EOT|EXE|GIF|GZ|GZIP|ICO|JPG|JPEG|JPE|WEBP|JSON|MDB|MID|MIDI|MOV|QT|MP3|M4A|MP4|M4V|MPEG|MPG|MPE|WEBM|MPP|OTF|_OTF|ODB|ODC|ODF|ODG|ODP|ODS|ODT|OGG|PDF|PNG|POT|PPS|PPT|PPTX|RA|RAM|SVG|SVGZ|SWF|TAR|TIF|TIFF|TTF|TTC|_TTF|WAV|WMA|WRI|WOFF|WOFF2|XLA|XLS|XLSX|XLT|XLW|ZIP)$">
    FileETag MTime Size
    <IfModule mod_headers.c>
         Header unset Set-Cookie
    </IfModule>
</FilesMatch>
<FilesMatch "\.(bmp|class|doc|docx|eot|exe|ico|webp|json|mdb|webm|mpp|otf|_otf|odb|odc|odf|odg|odp|ods|odt|ogg|pdf|pot|pps|ppt|pptx|svg|svgz|swf|tif|tiff|ttf|ttc|_ttf|wav|wri|woff|woff2|xla|xls|xlsx|xlt|xlw|BMP|CLASS|DOC|DOCX|EOT|EXE|ICO|WEBP|JSON|MDB|WEBM|MPP|OTF|_OTF|ODB|ODC|ODF|ODG|ODP|ODS|ODT|OGG|PDF|POT|PPS|PPT|PPTX|SVG|SVGZ|SWF|TIF|TIFF|TTF|TTC|_TTF|WAV|WRI|WOFF|WOFF2|XLA|XLS|XLSX|XLT|XLW)$">
    <IfModule mod_headers.c>
         Header unset Last-Modified
    </IfModule>
</FilesMatch>
<IfModule mod_headers.c>
    Header set Referrer-Policy "no-referrer-when-downgrade"
</IfModule>
# END W3TC Browser Cache

# BEGIN W3TC Page Cache core
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    RewriteCond %{HTTPS} =on
    RewriteRule .* - [E=W3TC_SSL:_ssl]
    RewriteCond %{SERVER_PORT} =443
    RewriteRule .* - [E=W3TC_SSL:_ssl]
    RewriteCond %{HTTP:X-Forwarded-Proto} =https [NC]
    RewriteRule .* - [E=W3TC_SSL:_ssl]
    RewriteCond %{HTTP:Accept-Encoding} br
    RewriteRule .* - [E=W3TC_ENC:_br]
    RewriteCond %{HTTP_COOKIE} w3tc_preview [NC]
    RewriteRule .* - [E=W3TC_PREVIEW:_preview]
    RewriteCond %{REQUEST_METHOD} !=POST
    RewriteCond %{QUERY_STRING} =""
    RewriteCond %{REQUEST_URI} \/$
    RewriteCond %{HTTP_COOKIE} !(comment_author|wp\-postpass|w3tc_logged_out|wordpress_logged_in|wptouch_switch_toggle) [NC]
    RewriteCond "%{DOCUMENT_ROOT}/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index%{ENV:W3TC_SSL}%{ENV:W3TC_PREVIEW}.html%{ENV:W3TC_ENC}" -f
    RewriteRule .* "/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index%{ENV:W3TC_SSL}%{ENV:W3TC_PREVIEW}.html%{ENV:W3TC_ENC}" [L]
    RewriteCond %{REQUEST_METHOD} !=POST
    RewriteCond %{QUERY_STRING} =""
    RewriteCond %{REQUEST_URI} \/$
    RewriteCond %{HTTP_COOKIE} !(comment_author|wp\-postpass|w3tc_logged_out|wordpress_logged_in|wptouch_switch_toggle) [NC]
    RewriteCond "%{DOCUMENT_ROOT}/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index%{ENV:W3TC_SSL}%{ENV:W3TC_PREVIEW}.xml%{ENV:W3TC_ENC}" -f
    RewriteRule .* "/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index%{ENV:W3TC_SSL}%{ENV:W3TC_PREVIEW}.xml%{ENV:W3TC_ENC}" [L]
</IfModule>
# END W3TC Page Cache core

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

Any help here would be GREATLY appreciated. https://example.com/php should not be rewriting to https://example.com/php-full-article-url - because it's not a legitimate match/binding/whatever it's called.

I have just checked my Apache access logs: 127.0.0.1 - - [21/Feb/2019:14:42:14 +0000] "GET /php HTTP/1.0" 301 - so it is doing a 301 redirect within Apache!!! I just don't know why

I've cleared Nginx caching, restarted both Apache and Nginx (and PHP for good measure).

Happy to give some real world example url's if that helps? I have a set of posts that all start /review- however if I just visit /review (expecting to get a 404 error - which is exactly correct!), it seems to just pick a random post and redirects/rewrites to serve that up instead (200 OK!)

Note: Originally asked on SO (https://stackoverflow.com/questions/54809631/) but got no responses - I feel it's probably more appropriate here

Improve this question
4
  • You have a "front controller" rewrite rule for WordPress: RewriteRule . /index.php [L] This assigns every URL that isn't an actual file or directory to be handled by WordPress. It is WordPress itself that is doing this, not your config files.
    – Stephen Ostermiller ♦
    Feb 22, 2019 at 10:23
  • Thanks @StephenOstermiller - is this standard wordpress behaviour? I'd never noticed this before, but it seems... "dangerous" to me to "match" part of a url to a full resource. For instance, I have a lot of posts that start /review-, but if I just visit example.com/review - I get one random post. If these were products (let's say /blue where I have /blue-tshirt, /blue-trousers & /blue-jumper), how does WP decide which page to use? Surely in that scenario, if there are multiple matches it should just be forwarding to example.com/?s=blue not picking a single page??
    – willstocks_
    Feb 22, 2019 at 12:12
  • The front controller is standard WordPress stuff (almost all content management systems do it.) I know that WordPress has some settings for how your URLs should look, but I'm not well versed in them. Ideally it would show a 404 error or redirect for URLs that aren't in exactly the right format. I'm not sure if there are plugins that would help make that happen.
    – Stephen Ostermiller ♦
    Feb 22, 2019 at 12:40
  • Thanks @StephenOstermiller - do you want to pop that in as an answer and I'll accept it? :)
    – willstocks_
    Feb 22, 2019 at 14:14

1 Answer 1

Reset to default
1

Per @StephenOstermiller's comments - this is actually something WordPress is handling (albeit, not very well).

It's basically guessing the full URI by picking a post/page that happens to have the same word in it!

I asked the question on WP's support forum and as it turns out, fortunately, there's a very easy way to disable this behaviour. One of the forum mod's gave me the following resource: https://www.bloggersignal.com/stop-wordpress-from-guessing-urls/

TL;DR - if you want to retain maximum SEO friendliness, pop the following in your functions.php

function wmse_121146_no_redirect_404($redirect_url)
{
    if ( is_404() ) {
        return false;
    }
    return $redirect_url;
}
add_filter('redirect_canonical', 'wmse_121146_no_redirect_404');

See WordPress forum question: https://wordpress.org/support/topic/wordpress-front-controller-redirecting-partial-urls

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.