URL rewritten pages take much longer to load

Question

(Same issue as http://stackoverflow.com/questions/3987754/how-to-diagnose-htaccess-speed-issues-via-rewritelog but more clearly explained below)

We have a site which has scripts like this:

http://www.oursite.com/abcd/somescript.php

We also have URL rewritten pages like this:

http://www.oursite.com/abcd/events/Anything/Anything/abc/abc.html

All of these share 90% same content in terms of js, css, and image files.

But the URL rewritten pages are extra-ordinary slow to load. Similarly, clicking on ANY link within such a page which leads to another such page is also very slow!

It strongly looks like that the browser is reloading the page completely and not using the cached images which should be present on the browser.

I turned on RewriteLog (at level 2) in Apache's httpd.conf and it shows the below when I:

1. Accessed the homepage at: www.example.com/abcd/
2. Clicked on the link to browse categories

3. Clicked on the category - Language

   URL/sid#ID = www.example.com/sid#2b06c5712d30

   Step 1
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (1) [perdir ar/www/html/abcd/] pass through /var/www/html/abcd/index.php
   Step 2
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (2) [perdir /var/www/html/abcd/] rewrite 'classes-events.html' -> 'browse.php?type=classes-events'
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (2) [perdir /var/www/html/abcd/] trying to replace prefix /var/www/html/abcd/ with /abcd/
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (1) [perdir /var/www/html/abcd/] internal redirect with /abcd/browse.php [INTERNAL REDIRECT]
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56d8528/initial/redir#1] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/browse.php
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/ajax/showUserData.php
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/ajax/showUserData.php
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/ajax/showUserData.php
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/ajax/browse.php
   Step 3
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (2) [perdir /var/www/html/abcd/] rewrite 'classes-events/Language/206.html' -> 'browse.php?type=classes-events&catId=206'
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (2) [perdir /var/www/html/abcd/] trying to replace prefix /var/www/html/abcd/ with /abcd/
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56caf10/initial] (1) [perdir /var/www/html/abcd/] internal redirect with /abcd/browse.php [INTERNAL REDIRECT]
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56d8948/initial/redir#1] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/browse.php
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56ccf20/initial] (2) [perdir /var/www/html/abcd/] rewrite 'classes-events/Language/ajax/showUserData.php' -> 'ajax/showUserData.php'
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56ccf20/initial] (2) [perdir /var/www/html/abcd/] trying to replace prefix /var/www/html/abcd/ with /abcd/
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56ccf20/initial] (1) [perdir /var/www/html/abcd/] internal redirect with /abcd/ajax/showUserData.php [INTERNAL REDIRECT]
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56d59d8/initial/redir#1] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/ajax/showUserData.php
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56ccf20/initial] (2) [perdir /var/www/html/abcd/] rewrite 'classes-events/Language/ajax/showUserData.php' -> 'ajax/showUserData.php'
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56ccf20/initial] (2) [perdir /var/www/html/abcd/] trying to replace prefix /var/www/html/abcd/ with /abcd/
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56ccf20/initial] (1) [perdir /var/www/html/abcd/] internal redirect with /abcd/ajax/showUserData.php [INTERNAL REDIRECT]
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56da9f8/initial/redir#1] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/ajax/showUserData.php
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56cef30/initial] (2) [perdir /var/www/html/abcd/] rewrite 'classes-events/Language/ajax/showUserData.php' -> 'ajax/showUserData.php'
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56cef30/initial] (2) [perdir /var/www/html/abcd/] trying to replace prefix /var/www/html/abcd/ with /abcd/
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56cef30/initial] (1) [perdir /var/www/html/abcd/] internal redirect with /abcd/ajax/showUserData.php [INTERNAL REDIRECT]
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56f4268/initial/redir#1] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/ajax/showUserData.php
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56cef30/initial] (2) [perdir /var/www/html/abcd/] rewrite 'classes-events/Language/ajax/browse.php' -> 'ajax/browse.php'
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56cef30/initial] (2) [perdir /var/www/html/abcd/] trying to replace prefix /var/www/html/abcd/ with /abcd/
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d56cef30/initial] (1) [perdir /var/www/html/abcd/] internal redirect with /abcd/ajax/browse.php [INTERNAL REDIRECT]
   IPADDRESS - - [DATETIME] [URL/sid#ID][rid#2b06d57127d8/initial/redir#1] (1) [perdir /var/www/html/abcd/] pass through /var/www/html/abcd/ajax/browse.php
   

---

Edit 1 - this is the .htaccess that we're using

RewriteEngine On
RewriteBase /abcd/

RewriteRule ^.*/codelibrary/(.*)$ codelibrary/$1 [L]
#RewriteCond %{THE_REQUEST} in_upload/images/(.*) [NC,OR]
RewriteRule ^.*/in_upload/images/(.*)$ in_upload/images/$1 [L]
RewriteCond %{REQUEST_URI} !in_upload/(.*)\.
RewriteRule ^.*/images/(.*)$ images/$1 [L]
RewriteRule ^.*/admin/ajax/(.*)$ admin/ajax/$1 [L]
RewriteCond %{REQUEST_URI} !admin/(.*)\.
RewriteRule ^.*/ajax/(.*)$ ajax/$1 [L]

RewriteRule ^.*/banner_images/(.*)$ banner_images/$1 [L]

#php_flag magic_quotes_gpc off
RewriteRule (.*)invite/([a-zA-Z0-9]+)$ referer.php?in_name=$2
RewriteRule (.*)cms/([a-zA-Z\-]*).html$ static-pages.php?page_url=$2 [L]
RewriteRule ^([a-zA-Z0-9]+)$ t-profile-public.php?name=$1

RewriteRule (.*)cms/([a-zA-Z\-]*).html$ static-pages.php?page_url=$2 [L]

RewriteRule (.*)classes-events.html$ browse.php?type=classes-events [L]
RewriteRule (.*)ins.html$ browse.php?type=ins [L]
RewriteRule (.*)orgs.html$ browse.php?type=orgs [L]
RewriteRule (.*)random-content.html$ browse.php?type=content&random_order=1 [L]
RewriteRule (.*)content.html$ browse.php?type=content [L]

RewriteRule ^([a-zA-Z\-]*)/([a-zA-Z0-9\s]*)/([0-9]*)\.html$ browse.php?type=$1&catId=$3 [L]
RewriteRule ^([a-zA-Z\-]*)/([a-zA-Z0-9\s]*)/([a-zA-Z0-9\s]*)/([0-9]*)/([0-9]*)\.html$ browse.php?type=$1&catId=$4&subCatId=$5 [L]
#RewriteRule (.*)search/all/([a-zA-Z0-9\s]*)$ searchall.php?type=all&srcval=$2 [L]
#RewriteRule (.*)search/([a-zA-Z\s\-]*)/([a-zA-Z0-9\s]*)$ search.php?type=$2&srcval=$3 [L]
#RewriteRule (.*)search/([a-zA-Z\s\-]*)/all/([a-zA-Z0-9\s]*)$ search.php?type=$2&srcval=$3&seeall=all [L]
#RewriteRule (.*)featured/([a-zA-Z\s\-]*)$ featured-listing.php?type=$2&feature=1 [L]
#RewriteRule (.*)featured/([a-zA-Z\-]*)/([a-zA-Z0-9\s]*)/([0-9]*)\.html$ featured-listing.php?type=$2&catId=$4&feature=1 [L]

RewriteRule (.*)search/all/(.*)$ searchall.php?type=all&srcval=$2 [L]
RewriteRule (.*)search/([a-zA-Z\s\-]*)/all/(.*)$ search.php?type=$2&srcval=$3&seeall=all [L]
RewriteCond %{REQUEST_URI} !all/(.*)\.
RewriteRule (.*)search/([a-zA-Z\s\-]*)/(.*)$ search.php?type=$2&srcval=$3 [L]
RewriteRule (.*)featured/(.*)$ featured-listing.php?type=$2&feature=1 [L]
RewriteRule (.*)featured/([a-zA-Z\-]*)/(.*)/([0-9]*)\.html$ featured-listing.php?type=$2&catId=$4&feature=1 [L]
ErrorDocument 404 http://www.oursite.com/abcd/404.php

Answer 1

Whew, that's one hefty .htaccess! One thing you may not be aware of is that the [L] flag doesn't completely stop processing on htaccess rules. Instead, it stops there and makes a new request for the rewritten URL - which starts matching the rules from the beginning again. See here for more details.

Some of your other rules look a little verbose. By default rules match anywhere in the URL, unless you specify ^ at the beginning or $ at the end. Therefore on this rule:

RewriteRule (.*)orgs.html$ browse.php?type=orgs [L]

The (.*) is not necessary and can be removed - Apache will still look for any URL ending in orgs.html.

Also check how necessary those rules are - do you really need to match any URL ending in orgs.html, or is it only ever at example.com/orgs.html ?

Finally, the asterisk modifier searches for "zero or more characters" - so cms/([a-zA-Z\-]*).html$ would match cms/.html. Many of them ought to be replaced with the plus sign to match at least one character - e.g. cms/([a-zA-Z\-]+).html$

Answer 2

How many rewrite rules do you have? If you have many rules, are you using the [L] modifier to make sure the rewrite engine doesn't check rules unnecessarily?