Internal Server Error; enable debugging provides no additional information

Question

I'm working with Mediawiki 1.25.1 on CentOS 7.2 running Apache 2.4.3. I'm working with Mediawiki 1.26.4 on CentOS 7.2 running Apache 2.4.6. I am experiencing an Internal Server Error after editing a page and submitting it:

There is no additional information in /var/log/httpd/error_log. According to Manual:How to debug, I added the following to LocalSettings, and then restarted Apache:

$ sudo tail -7 LocalSettings.php
# Added by JW for debugging (JUN 2016). Keep commented unless needed.
$wgShowExceptionDetails=true;
$wgShowSQLErrors=true;
error_reporting( -1 );
ini_set( 'display_errors', 1 );

?>

Adding $wgShowExceptionDetails, error_reporting and friends does not provide any additional information that we can find. error_log is clean, access_log is clean. We enabled error_log = php_errors.log, but the server does not produce a php_errors.log. We have no idea where else to look.

We ran mysqlcheck my_wiki --auto-repair --user=... --password=... and it reports no errors. apachectl configtest immediately returns Syntax OK. However, the problem persists.

We also performed an upgrade to MediaWiki 1.26.4 (from 1.25.1) in hopes of fixing this internal server error with no joy. It still persists, and I still cannot get additional information about it.

QUESTIONS: How does one really enable debugging information? How can we gather more information about the problem plaguing this server?

(Please provide actionable items in response to the question; and please don't provide off-site links telling us to try some of the stuff on some other page that may or may not work. And please provide answers for the specific question that was asked, and not other questions that were not asked.)

---

Here's the most recent "useless error message" from error_log after upgrading to MediaWiki 1.26. Its from the tail of /var/log/httpd/error_log when a request was submitted to update a wiki page.

...
[Fri Oct 28 21:05:04.456126 2016] [suexec:notice] [pid 1053] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Oct 28 21:05:04.457260 2016] [:notice] [pid 1053] ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/) configured.
[Fri Oct 28 21:05:04.457283 2016] [:notice] [pid 1053] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Fri Oct 28 21:05:04.457290 2016] [:notice] [pid 1053] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Fri Oct 28 21:05:04.457295 2016] [:notice] [pid 1053] ModSecurity: LUA compiled version="Lua 5.1"
[Fri Oct 28 21:05:04.457299 2016] [:notice] [pid 1053] ModSecurity: LIBXML compiled version="2.9.1"
[Fri Oct 28 21:05:04.495066 2016] [auth_digest:notice] [pid 1053] AH01757: generating secret for digest authentication ...
[Fri Oct 28 21:05:04.632965 2016] [mpm_prefork:notice] [pid 1053] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Fri Oct 28 21:05:04.632994 2016] [core:notice] [pid 1053] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

---

A couple of folks have commented about the startup messages. I did not realize it was abnormal since Apache always did it. We recently made the following change to address it (but it did not help out "no information" problem):

# diff /etc/httpd/conf.d/mpm_prefork.conf.bu /etc/httpd/conf.d/mpm_prefork.conf
9,13c9,13
< StartServers       1
< MinSpareServers    1
< MaxSpareServers    5
< ServerLimit       10
< MaxClients        10
---
> StartServers       4
> MinSpareServers    4
> MaxSpareServers    8
> ServerLimit       32
> MaxClients        32

None of us are professional admins, so we don't know if its enough to resolve the issue. We are a bunch of free and open software developers who run a web server and wiki to help users.

---

Here is the "maximum brevity" /etc/php.ini:

# cat /etc/php.ini | egrep -v '(^;|^\[)' | sed '/^$/d'
engine = On
short_open_tag = Off
asp_tags = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 8M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 2M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
cli_server.color = On
date.timezone = "UTC"
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
SMTP = localhost
smtp_port = 25
sendmail_path = /usr/sbin/sendmail -t -i
mail.add_x_header = On
sql.safe_mode = Off
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
mysql.allow_local_infile = On
mysql.allow_persistent = On
mysql.cache_size = 2000
mysql.max_persistent = -1
mysql.max_links = -1
mysql.default_port =
mysql.default_socket =
mysql.default_host =
mysql.default_user =
mysql.default_password =
mysql.connect_timeout = 60
mysql.trace_mode = Off
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
sybct.allow_persistent = On
sybct.max_persistent = -1
sybct.max_links = -1
sybct.min_server_severity = 10
sybct.min_client_severity = 10
bcmath.scale = 0
session.save_handler = files
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.bug_compat_42 = Off
session.bug_compat_warn = Off
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
mssql.allow_persistent = On
mssql.max_persistent = -1
mssql.max_links = -1
mssql.min_error_severity = 10
mssql.min_message_severity = 10
mssql.compatability_mode = Off
mssql.secure_connection = Off
tidy.clean_output = Off
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
ldap.max_links = -1

Answer 1

PHP Parse error: syntax error, unexpected 'error_reporting'...

You are missing a semicolon (;) from the end of the preceding line, ie:

$wgShowExceptionDetails=true;

An "Internal Server Error" (code: 500) is a very generic (catch all) error. Which is why you need the server's error_log for more information. If using the Apache webserver it is often caused by syntax errors in per-directory Apache config files (ie. .htaccess files). Syntax errors can also occur if required server modules are not installed (since the directives cannot be interpreted).

PHP will also trigger a 500 error if a fatal error occurs and error_reporting / display_errors are suppressed.

Answer 2

This type of error is due to mis-confiuration of .htaccess file. php has nothing to do with it. You need to debug apache with some useful diagnostic commands

apachectl configtest

useful help is here https://serverfault.com/questions/232145/command-to-check-validity-of-apache-server-config-files

Answer 3

You say you got an internal server error at 14:20, but this does not appear in your log extract (presumably you didn't see the 500 response in your access log either). This makes me think the child process crashed - on a default centos build, this should have resulted in the core being dumped.

If this is the case, then there's a problem with the apache or PHP build. Code written in PHP cannot cause such a crash (it can only trigger the the binary code which then crashes). Upgrading MediaWiki will not help in such a scenario. You could have a look at the core file in gdb, but really the first step is to ensure that both PHP and Apache are patched up to date (and include the versions of both when asking questions here).

It is somewhat alarming to see you are reaching maxrequestworker twice in your log file. And is there a reason for running -DFOREGROUND?

Answer 4

I think we figured out how to diagnose the issue. All this time we were focusing on error logs because we consider a 500 server crash an error, and we specified errors should be logged to error_log in the configuration files.

Out of desperation, we grep'ed the entire file system for *"Internal Server Error". Lo and behold, mod_security is not honoring the configuration settings:

# grep -IR 'Internal Server Error' /var/log
...
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
...

/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161016:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161016:HTTP/1.0 500 Internal Server Error     
...

<no error_log entries>

I'm OK with mod_security logging to their own file. I take exception that they did not log to where we told them to through php.ini, and it has wasted months of man hours and trouble shooting. All they had to do was write an occasional message into error_log and this would have been fixed months ago.