2

I'm working with Mediawiki 1.25.1 on CentOS 7.2 running Apache 2.4.3. I'm working with Mediawiki 1.26.4 on CentOS 7.2 running Apache 2.4.6. I am experiencing an Internal Server Error after editing a page and submitting it:

enter image description here

There is no additional information in /var/log/httpd/error_log. According to Manual:How to debug, I added the following to LocalSettings, and then restarted Apache:

$ sudo tail -7 LocalSettings.php
# Added by JW for debugging (JUN 2016). Keep commented unless needed.
$wgShowExceptionDetails=true;
$wgShowSQLErrors=true;
error_reporting( -1 );
ini_set( 'display_errors', 1 );

?>

Adding $wgShowExceptionDetails, error_reporting and friends does not provide any additional information that we can find. error_log is clean, access_log is clean. We enabled error_log = php_errors.log, but the server does not produce a php_errors.log. We have no idea where else to look.

We ran mysqlcheck my_wiki --auto-repair --user=... --password=... and it reports no errors. apachectl configtest immediately returns Syntax OK. However, the problem persists.

We also performed an upgrade to MediaWiki 1.26.4 (from 1.25.1) in hopes of fixing this internal server error with no joy. It still persists, and I still cannot get additional information about it.

QUESTIONS: How does one really enable debugging information? How can we gather more information about the problem plaguing this server?

(Please provide actionable items in response to the question; and please don't provide off-site links telling us to try some of the stuff on some other page that may or may not work. And please provide answers for the specific question that was asked, and not other questions that were not asked.)


Here's the most recent "useless error message" from error_log after upgrading to MediaWiki 1.26. Its from the tail of /var/log/httpd/error_log when a request was submitted to update a wiki page.

...
[Fri Oct 28 21:05:04.456126 2016] [suexec:notice] [pid 1053] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Oct 28 21:05:04.457260 2016] [:notice] [pid 1053] ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/) configured.
[Fri Oct 28 21:05:04.457283 2016] [:notice] [pid 1053] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Fri Oct 28 21:05:04.457290 2016] [:notice] [pid 1053] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Fri Oct 28 21:05:04.457295 2016] [:notice] [pid 1053] ModSecurity: LUA compiled version="Lua 5.1"
[Fri Oct 28 21:05:04.457299 2016] [:notice] [pid 1053] ModSecurity: LIBXML compiled version="2.9.1"
[Fri Oct 28 21:05:04.495066 2016] [auth_digest:notice] [pid 1053] AH01757: generating secret for digest authentication ...
[Fri Oct 28 21:05:04.632965 2016] [mpm_prefork:notice] [pid 1053] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Fri Oct 28 21:05:04.632994 2016] [core:notice] [pid 1053] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

A couple of folks have commented about the startup messages. I did not realize it was abnormal since Apache always did it. We recently made the following change to address it (but it did not help out "no information" problem):

# diff /etc/httpd/conf.d/mpm_prefork.conf.bu /etc/httpd/conf.d/mpm_prefork.conf
9,13c9,13
< StartServers       1
< MinSpareServers    1
< MaxSpareServers    5
< ServerLimit       10
< MaxClients        10
---
> StartServers       4
> MinSpareServers    4
> MaxSpareServers    8
> ServerLimit       32
> MaxClients        32

None of us are professional admins, so we don't know if its enough to resolve the issue. We are a bunch of free and open software developers who run a web server and wiki to help users.


Here is the "maximum brevity" /etc/php.ini:

# cat /etc/php.ini | egrep -v '(^;|^\[)' | sed '/^$/d'
engine = On
short_open_tag = Off
asp_tags = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 8M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 2M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
cli_server.color = On
date.timezone = "UTC"
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
SMTP = localhost
smtp_port = 25
sendmail_path = /usr/sbin/sendmail -t -i
mail.add_x_header = On
sql.safe_mode = Off
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
mysql.allow_local_infile = On
mysql.allow_persistent = On
mysql.cache_size = 2000
mysql.max_persistent = -1
mysql.max_links = -1
mysql.default_port =
mysql.default_socket =
mysql.default_host =
mysql.default_user =
mysql.default_password =
mysql.connect_timeout = 60
mysql.trace_mode = Off
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
sybct.allow_persistent = On
sybct.max_persistent = -1
sybct.max_links = -1
sybct.min_server_severity = 10
sybct.min_client_severity = 10
bcmath.scale = 0
session.save_handler = files
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.bug_compat_42 = Off
session.bug_compat_warn = Off
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
mssql.allow_persistent = On
mssql.max_persistent = -1
mssql.max_links = -1
mssql.min_error_severity = 10
mssql.min_message_severity = 10
mssql.compatability_mode = Off
mssql.secure_connection = Off
tidy.clean_output = Off
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
ldap.max_links = -1

migrated from webapps.stackexchange.com Jun 20 '16 at 7:50

This question came from our site for power users of web applications.

  • Even without these debugging statements, you should get useful information in your server's error_log. – Stephen Ostermiller Jun 20 '16 at 7:59
  • Thanks @Stephen. I added error_log information after causing an internal server error. – jww Jul 2 '16 at 18:31
  • Have you addressed the MaxRequestWorkers problem? – Stephen Ostermiller Oct 29 '16 at 1:37
  • "Here's the most recent 'useless error message'" - those aren't error messages. Those are notices, consistent with having additional debug information logged. What version of Apache are you using? Presumably you have access to the server config? Do you know what "log level" Apache is currently set at? – MrWhite Oct 29 '16 at 9:03
  • @w3dk - "those aren't error messages. Those are notices..." - I can't explain why they are provided error_log. "What version of Apache are you using..." - Apache 2.4.6. "Do you know what log level..." - Cat'ing httpd.conf shows LogLevel warn. – jww Oct 29 '16 at 9:20
2

PHP Parse error: syntax error, unexpected 'error_reporting'...

You are missing a semicolon (;) from the end of the preceding line, ie:

$wgShowExceptionDetails=true;

An "Internal Server Error" (code: 500) is a very generic (catch all) error. Which is why you need the server's error_log for more information. If using the Apache webserver it is often caused by syntax errors in per-directory Apache config files (ie. .htaccess files). Syntax errors can also occur if required server modules are not installed (since the directives cannot be interpreted).

PHP will also trigger a 500 error if a fatal error occurs and error_reporting / display_errors are suppressed.

  • Oh, thanks. Let me give it a go and accept if its progress. – jww Jun 20 '16 at 7:56
  • OK, I was missing the semi colon. However, adding all the suggested statements with the semi-colons did not produce any more information. Maybe Apache is broke instead of MediaWiki. It sure would be nice if something was logged somewhere rather than making you and me guess. – jww Jul 2 '16 at 18:37
  • I just upgraded to MediaWiki 1.26.4 in an effort to fix the wiki... Still broken. Nothing displayed on the web page. Nothing written to error_log. I guess it is pick my poison: keep suffering or keep guessing. – jww Oct 29 '16 at 1:03
2
+25

This type of error is due to mis-confiuration of .htaccess file. php has nothing to do with it. You need to debug apache with some useful diagnostic commands

apachectl configtest

useful help is here https://serverfault.com/questions/232145/command-to-check-validity-of-apache-server-config-files

  • Thanks Vikas. You might as well take a look at How to test '.htaccess' from command line at Unix & Linux Stack Exchange. – jww Oct 29 '16 at 8:32
  • Thanks again Vikas. I'm trying to figure out what exactly I should do next since following MediaWiki's Manual:How to debug does not work. apachectl configtest immediately returns Syntax OK. Can you please prescribe something I should do? – jww Oct 30 '16 at 10:21
1

You say you got an internal server error at 14:20, but this does not appear in your log extract (presumably you didn't see the 500 response in your access log either). This makes me think the child process crashed - on a default centos build, this should have resulted in the core being dumped.

If this is the case, then there's a problem with the apache or PHP build. Code written in PHP cannot cause such a crash (it can only trigger the the binary code which then crashes). Upgrading MediaWiki will not help in such a scenario. You could have a look at the core file in gdb, but really the first step is to ensure that both PHP and Apache are patched up to date (and include the versions of both when asking questions here).

It is somewhat alarming to see you are reaching maxrequestworker twice in your log file. And is there a reason for running -DFOREGROUND?

  • Regarding the -DFOREGROUND, I'm guessing its from either (1) nearly 20 years of running a web server so its cruft carried forward, or (2) Confused about DFOREGROUND with Apache and Michael Hampton's answer. Its probably the latter. – jww Oct 30 '16 at 3:51
  • Regarding MaxRequestWorkers, I'm guessing that is CentOS 7 defaults. I changed the defaults as edit'ed into the original question. I guess I assumed that was normal, and Apache was managing its own resources. – jww Oct 30 '16 at 4:00
  • Thanks again symcbean. Please forgive my ignorance... what exactly I should do next? I can't quite parse the actionable item in the answer you provided. I am disgusted I cannot get any useful information out of this server. – jww Oct 30 '16 at 10:25
0

I think we figured out how to diagnose the issue. All this time we were focusing on error logs because we consider a 500 server crash an error, and we specified errors should be logged to error_log in the configuration files.

Out of desperation, we grep'ed the entire file system for *"Internal Server Error". Lo and behold, mod_security is not honoring the configuration settings:

# grep -IR 'Internal Server Error' /var/log
...
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
/var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error
...

/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161016:HTTP/1.0 500 Internal Server Error
/var/log/httpd/modsec_audit.log-20161016:HTTP/1.0 500 Internal Server Error     
...

<no error_log entries>

I'm OK with mod_security logging to their own file. I take exception that they did not log to where we told them to through php.ini, and it has wasted months of man hours and trouble shooting. All they had to do was write an occasional message into error_log and this would have been fixed months ago.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.