0

I use the below htaccess code which appears to function correctly to rewrite URLS, however when a page is shared to social networks or scanned by bots like AHREFs, they all return 404 yet the pages at the site display correctly, and when I print http_response_code() on any page, the response value is 200.

<IfModule mod_rewrite.c>
RewriteEngine on

#RewriteBase /

RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^(.+[^/])$  $1/ [R]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

RewriteCond $0#%{REQUEST_URI} ([^#]*)#(.*)\1$
RewriteRule ^.*$ %2index.php [QSA,L]
</IfModule>

The long URL being rewritten

index.php?component=posts&view=post&category=15-the-blog&post=31-how-blog-well&menuid=25-the-blog

The rewritten URL

/the-blog/how-to-blog-well

The PHP functions that processes the URLS

/* 
Process URLs as sent from links, before converted to SEF to 
retrieve $_GET[] values for use in DB queries and 
other instructions 
*/

function getNonSEF() 
{
    $path = pathinfo(currenturl());
    $filename = $path['filename'];
    $dirnames = explode('/', $path['dirname']);
    preg_match('#[^\/]+$#i', $path['dirname'], $match);


    // product detail and post view
    if( $path['dirname'].'/' == ROOT_URL ) 
    {
        $postid = posts()->where('alias', getCurrentFilename())->value('id');
        $cat_id = categories()->where('alias', getCurrentFilename())->value('id');

        $groupval = getDbo('mod_cmse_products')->select('group_id', 'group_alias')->where('group_alias', $filename)->get()[0];

        if( !empty($postid) ) {
            $queryurl = parse_url(menus()->where([['type', 3], ['src_id', $postid]])->value('url'))['query'];
        }else
        if( !empty($cat_id) ) {
            $url = menus()->where([['src_id', $cat_id], ['type', 2]])->value('url');
            if( !empty($url) ) {
                $queryurl = parse_url($url)['query'];
            }else{
                // category has no menu item
                $queryurl = 'component=posts&view=category&category='.$cat_id.'-'.getCurrentFilename();
            }
        }else
        // product category
        if( $filename == $groupval->group_alias ) {
            $queryurl = 'component=products&view=productgroup&gid='.$groupval->group_id.'-'.$groupval->group_alias;
        //}else
            //index.php?component=products&view=productgroups&menuid=25-all-services
        //if( in_array($filename, menus()->where('type', 5)->pluck('alias')) ) {
        //  $queryurl = parse_url(menus()->where('alias', getCurrentFilename())->value('url'))['query'];
        }else{
            $queryurl = parse_url(menus()->where('alias', getCurrentFilename())->value('url'))['query'];
        }

    }else

    {
        // if view is post and has category alias in url but does not have an assigned menu
        $postval = posts()->select('id', 'catid', 'alias')->where('alias', $filename)->get()[0];

        $prodval = getDbo('mod_cmse_products')
        ->select('prod_alias', 'prod_id', 'prod_gid')
        ->where('prod_alias', $filename)
        ->get()[0];

        if( $filename == $postval->alias ) {
            $posturl = menus()->select('id', 'url', 'src_id')->where([['src_id', $postval->catid],['type', 2]])->get()[0];
            parse_str(parse_url($posturl->url)['query'], $urlparts);

            $queryurl = 'component='.$urlparts['component'].'&view=post&category='.$urlparts['category'].'&post='.$postval->id.'-'.$path['filename'].'&menuid='.$urlparts['menuid'];
        }else

        // component is products
        if( $filename == $prodval->prod_alias ) {
            $prodgroup = getDbo('mod_cmse_products')->where('group_id', $prodval->prod_gid)->value('group_alias');
            $queryurl = 'component=products&view=productdetail&gid='.$prodval->prod_gid.'-'.$prodgroup.'&pid='.$prodval->prod_id.'-'.$prodval->prod_alias;
        }

    }


    $queryurl = html_entity_decode($queryurl);


    // get query elements
    $view       = getInput($queryurl, 'view');
    $category   = getInput($queryurl, 'category');
    $post       = getInput($queryurl, 'post');
    $menuid     = str_replace('-', '', (int)getInput($queryurl, 'menuid'));
    $listpage   = getInput($queryurl, 'listpage');
    $prodid     = getInput($queryurl, 'pid');
    $groupid    = getInput($queryurl, 'gid');
    $parent     = $match[0];

    // registered components
    $com_posts = ['post', 'category', 'categories'];
    $com_product = ['productdetail', 'productgroup', 'productgroups'];

    // the decider (roger)
    if( in_array($view, $com_product) ) {
        $component  = 'products';
    }else
    if( in_array($view, $com_posts) ) {
        $component  = 'posts';
    }

    $list = [$component, $view, $category, $post, $menuid, $listpage, $prodid, $groupid, $parent];

    return $list;
}


/* 
router which wraps URL out put in files
eg: <a href="<?php echo router($modlink.'&view=post&category='.$post->catid.'-'.$cat->alias.'&post='.$post->id.'-'.$post->alias.'&menuid='.$menuid); ?>">View Post</a>
*/

function router($url) 
{
    $parts = parse_url(ROOT_URL.$url);
    parse_str($parts['query'], $q);

    $component  = $q['component'];
    $view       = $q['view'];
    $category   = $q['category'];
    $post       = $q['post'];
    $menuid     = $q['menuid'];
    $prodgroup  = $q['gid'];
    $prodid     = $q['pid'];


    if( isset($component) ) 
    {
        if( $view == 'post' && isset($post) && !isset($category) && isset($menuid) ) {
            $num = (int)$post;
            $route = str_replace($num.'-', '', $post);
        }else
            // posts without menu id uses category alias in url
        if( $view == 'post' && isset($post) && isset($category) ) {
            $num = (int)$post;
            $nums = (int)$category;
            $route = str_replace($nums.'-', '', $category).'/'.str_replace($num.'-', '', $post);
        }else
        // posts category view
        if( $view == 'category' && isset($category) ) {
            $num = (int)$category;
            $route = str_replace($num.'-', '', $category);
        }else
        if( $view == 'categories' ) {
            $num = (int)$menuid;
            $route = str_replace($num.'-', '', $menuid);
        }else

        // product detail view
        if( $view == 'productdetail' && isset($prodgroup) && isset($prodid) ) {
            $route = preg_replace('#\d+\-#', '', strtolower($prodgroup)).'/'.preg_replace('#\d+\-#', '', $prodid);
        }else

        if( isset($view) && $view == 'productgroup' && isset($prodgroup) ) {
            $num = (int)$prodgroup;
            $route = str_replace($num.'-', '', $prodgroup);
        }else
        if( isset($view) && $view == 'productgroups' ) {
            $num = (int)$menuid;
            $route = str_replace($num.'-', '', $menuid);

            if( pathinfo(currenturl())['filename'] == 'cart' && !requestKey('a') ) {
                redirect(str_replace($num.'-', '', $menuid), 301);
            }
        }

    }else{
        // default urls
        $route = $url;
    }

    return $route;
}

What's bizarre is when I use Facebook's object debugger to scrape the pages, though the report 404, the og:image is returned and when I click "see what the scraper sees", the HTML code is correct! So the bot is seeing the right HTML yet reports "bad response code".

I tried adding the following condition to instruct the bots to use the given URL but it does not work.

RewriteCond %{HTTP_USER_AGENT} (facebookexternalhit/[0-9]|Twitterbot|Pinterest|Google.*snippet)
RewriteRule .* [L]

This is the URL type which fails with 404 and sometimes 500, only when seen by bots http://websitedons.net/demo/whmcs/the-blog

But this URL returns 200 http://websitedons.net/demo/whmcs

How must the htaccess be written to deliver consistently good responses?

  • Are you sure it is the htaccess causing the 404's? – John Conde Jun 19 at 20:23
  • @JohnConde No. However since there are no php errors, it seems logical to me that the bots are being instructed by the htaccess and therefore return a false positive. Please present any potential cause you see. – Nadal Jun 19 at 21:15
  • 1
    wget -S http://websitedons.net/demo/whmcs/the-blog gives HTTP/1.1 404 Not Found and exactly the same from my Firefox (which still displays the page). So it seems to always return 404, not only to bots. There is nothing in your PHP code that touches the return code. – Patrick Mevzek Jun 19 at 21:42
  • @PatrickMevzek Do you have any coding suggestion which could resolve this error? The core framework is WHMCS host billing system which is IonCube encrypted so I cannot define its redirection method therefore working in the dark. – Nadal Jun 19 at 22:26
  • @PatrickMevzek and JohnConde are you guys with-holding possible solutions or don't have any ideas for resolving the error? – Nadal Jun 20 at 18:59
0

I was able to resolve the error with a rewrite rule and a bit of php coding to change the error response method used by WHMCS, which is based of fixed values and if those are not defined in the array, an error is sent via Zend apiemitter

I added a condition and rule before all other rules

RewriteCond %{SCRIPT_FILENAME} !-f
RewriteCond %{SCRIPT_FILENAME} !-d
RewriteRule .* exrule.php [L]

which is placed in the host root or where WHMCS is installed, and the coding for the php

(new Zend\Diactoros\Response\SapiEmitter())->emit($statuscode);

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.