404 errors turning into 500 errors through internal redirects

Question

Today I checked my Apache error.log and I found many:

Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.

...mostly from Bing bot.

So i traced the error to my access log and I found that these errors are being caused by some site/bots requesting a file that no longer exists. Which should create 404 (not found) error, but instead it is redirecting the site to look somewhere else and then that place is telling it to look in the first place, causing an internal redirect which only stops once it reaches the maximum of 10 internal redirects. At this point it puts out a 500 internal error and it's crawled over again and again.

Subdomains are in:

/home/html/myname/public_html/_sub/thisisasubdomain/

Resulting domain is:

thisisasubdomain.myname.com

When I try to access a non existing directory in the _sub directory, then I get the 500 error/Request exceeded error.

My VPS has pre-installed server with this rewrite.conf, defined in :

RewriteEngine On
RewriteMap lowercase int:tolower

RewriteCond  %{REQUEST_URI}  !^/webmail(/|$)?
RewriteCond  %{REQUEST_URI}  !^/horde(/|$)?
RewriteCond  %{REQUEST_URI}  !^/roundcube(/|$)?
RewriteCond  %{REQUEST_URI}  !^/setup(2|-new)?(/|$)?
RewriteCond  %{REQUEST_URI}  !^/webftp(1|2)?(/|$)?
RewriteCond  %{REQUEST_URI}  !^/(db|pg)admin(/|$)?

RewriteCond ${lowercase:%{SERVER_NAME}} ^([0-9a-z-]+)\.([0-9a-z]+)$
RewriteRule ^(.+) ${lowercase:%{SERVER_NAME}}$1 [C]
RewriteRule ([0-9a-z-]+)\.([0-9a-z]+)/(.*) /home/html/$1\.$2/public_html/$3

RewriteCond  %{REQUEST_URI}  !^/webmail(/|$)?
RewriteCond  %{REQUEST_URI}  !^/horde(/|$)?
RewriteCond  %{REQUEST_URI}  !^/roundcube(/|$)?
RewriteCond  %{REQUEST_URI}  !^/setup(2|-new)?(/|$)?
RewriteCond  %{REQUEST_URI}  !^/webftp(1|2)?(/|$)?
RewriteCond  %{REQUEST_URI}  !^/(db|pg)admin(/|$)?

RewriteCond ${lowercase:%{SERVER_NAME}} ^www\.([0-9a-z-]+)\.([0-9a-z]+)$
RewriteRule ^(.+) ${lowercase:%{SERVER_NAME}}$1 [C]
RewriteRule www\.([0-9a-z-]+)\.([0-9a-z]+)/(.*) /home/html/$1\.$2/public_html/$3

RewriteCond  %{REQUEST_URI}  !^/webmail(/|$)?
RewriteCond  %{REQUEST_URI}  !^/horde(/|$)?
RewriteCond  %{REQUEST_URI}  !^/roundcube(/|$)?
RewriteCond  %{REQUEST_URI}  !^/setup(2|-new)?(/|$)?
RewriteCond  %{REQUEST_URI}  !^/webftp(1|2)?(/|$)?
RewriteCond  %{REQUEST_URI}  !^/(db|pg)admin(/|$)?

RewriteCond ${lowercase:%{SERVER_NAME}} !^www\.([0-9a-z-]+)\.([0-9a-z]+)$
RewriteCond ${lowercase:%{SERVER_NAME}} ^([0-9a-z-]+)\.([0-9a-z-]+)\.([0-9a-z]+)$ 
RewriteRule ^(.+) ${lowercase:%{SERVER_NAME}}$1 [C]
RewriteRule ([0-9a-z-]+)\.([0-9a-z-]+)\.([0-9a-z]+)/(.*) /home/html/$2\.$3/public_html/_sub/$1/$4

RewriteCond  %{REQUEST_URI}  !^/webmail(/|$)?
RewriteCond  %{REQUEST_URI}  !^/horde(/|$)?
RewriteCond  %{REQUEST_URI}  !^/roundcube(/|$)?
RewriteCond  %{REQUEST_URI}  !^/setup(2|-new)?(/|$)?
RewriteCond  %{REQUEST_URI}  !^/webftp(1|2)?(/|$)?
RewriteCond  %{REQUEST_URI}  !^/(db|pg)admin(/|$)?

RewriteCond ${lowercase:%{SERVER_NAME}} ^(.+)\.([0-9a-z-]+)\.([0-9a-z-]+)\.([0-9a-z]+)$ 
RewriteRule ^(.+) ${lowercase:%{SERVER_NAME}}$1 [C]
RewriteRule (.+)\.([0-9a-z-]+)\.([0-9a-z-]+)\.([0-9a-z]+)/(.*) /home/html/$3\.$4/public_html/_sub/$2/$5

Any help is appreciated.

Answer 1

You've not yet included enough information in your question in order to give a precise answer. (ie. As requested in comments... what should be served in the case of a 404, how this is being served and whether you have a custom 404 ErrorDocument defined somewhere?) You only appear to have included your rewrite directives in your question.

A common cause for a rewrite loop (500 error) when a 404 occurs is because your custom error document is being rewritten (by your mod_rewrite directives) when a 404 occurs. This appears to be what you are describing, but you've not actually stated what is being rewritten.

When you have a custom error document defined, an internal subrequest is triggered that serves this error document. However, this subrequest can also be rewritten by your mod_rewrite directives and essentially "break" the request.

It is common to have to include an exception for your error documents at the top of your mod_rewrite directives. For example:

RewriteRule ^/?error-documents/ - [L]

(Assuming all your error documents are stored in a directory /error-documents in the document root.)

Alternatively, you can prevent the conflicting RewriteRule directive from being triggered on subrequests by using the NS (nosubreq) flag.

Or, remove your custom error document completely (probably not desirable) either by removing the directive where this is defined, or setting ErrorDocument 404 default in a directory (or .htaccess) context if you don't have access to the server config.