Tracking down memory issues affecting a website

Question

I've got a website (Wordpress based) that became unresponsive. I SSH'd into the server and saw that we were out of memory. Errors in my apache log files indicated the same... things failing to be allocated due to lack of memory). Restarting the server fixes it.

So I look in access.log and error.log around the time of the incident but I see nothing strange. No extra traffic, no unusual requests. In fact the only request around the time of the problem was one from Googlebot for an rss feed... at that point I start to see 500 response codes in the logs until the machine was rebooted. I look in message.log hoping to see something but there is nothing at all for that entire day (which is odd as there are entries for every other day).

The site has a large amount of memory allocated to it and normally runs using about 30% of what is available. My question... how would you go about trying to track this down at this point? What are some other log files I could check or strategies I could take?

Answer 1

The general recommendations:

• use cache plugin for your WP installation
• reduce PHP memory_limit to (32..96M) to start seeing PHP memory exceed errors
• disable useless and new plugins
• make sure all reporting settings are on
• reduce max worker processes by hand (3..10)
• set nonzero MaxRequestsPerChild if you think that PHP/Apache may have leaks (errors in compiled code of PHP interpreter or Apache server)
• reduce ServerLimit
• reduce MaxClients
• use PHP in FPM mode

Specific advices:

• write a bash script to measure memory usage or collect peak and put it in cron. You'll have stats for memory usage of a process over time.
• or use munin as more advanced solution.

Answer 2

Check for the wa line in top it will show if you have io wait

If so, check your MySQL error log and try to analyse / repair your tables.

This can be a network or storage problem. The wait average will tell you that .

Hope this helps

Answer 3

You should first disable all your plugins, then restart your apache server. Run top or mtop in terminal and determine if memory leaks have been fixed. If so begin activating plugins and waiting 10-15 minutes and checking your memory usage on the server to try and determine if it is a plugin.

Answer 4

Top and PS are very good tools. I would also look into /proc/meminfo. You can start Valgrind together with Apache to check for memory leaks. But I wouldn't recommend Apache anyway. I can recommend lighttpd or nginx.

Answer 5

You can use something like New Relic, example http://blog.newrelic.com/2010/12/16/measuring-wordpress-performance-with-new-relic-rpm/

---

This script is also handy: http://www.pixelbeat.org/scripts/ps_mem.py

Example usage (taken from here):

-bash-3.2$ wget http://www.pixelbeat.org/scripts/ps_mem.py
-bash-3.2$ sudo python ps_mem.py
 Private  +   Shared  =  RAM used   Program 

 92.0 KiB +  12.0 KiB = 104.0 KiB   qmail-clean
 96.0 KiB +  14.0 KiB = 110.0 KiB   splogger
116.0 KiB +  23.0 KiB = 139.0 KiB   init
128.0 KiB +  12.0 KiB = 140.0 KiB   qmail-rspawn
124.0 KiB +  16.0 KiB = 140.0 KiB   syslogd
132.0 KiB +  12.0 KiB = 144.0 KiB   qmail-lspawn
148.0 KiB +  13.0 KiB = 161.0 KiB   qmail-send
208.0 KiB +  28.5 KiB = 236.5 KiB   dbus-daemon
232.0 KiB +  36.5 KiB = 268.5 KiB   xinetd
240.0 KiB +  32.5 KiB = 272.5 KiB   mysqld_safe
328.0 KiB +  20.5 KiB = 348.5 KiB   udevd
348.0 KiB +  66.0 KiB = 414.0 KiB   courierlogger (4)
444.0 KiB +  85.5 KiB = 529.5 KiB   bash
480.0 KiB +  50.0 KiB = 530.0 KiB   xfs
592.0 KiB +  36.0 KiB = 628.0 KiB   crond
544.0 KiB + 114.0 KiB = 658.0 KiB   couriertcpd (4)
  1.3 MiB +  82.5 KiB =   1.4 MiB   sw-cp-serverd
  1.2 MiB +   1.1 MiB =   2.3 MiB   sshd (3)
  3.1 MiB + 205.5 KiB =   3.3 MiB   named
  3.9 MiB +  48.2 MiB =  52.1 MiB   spamd (2)
 63.7 MiB + 387.0 KiB =  64.1 MiB   mysqld
108.3 MiB +   9.2 MiB = 117.5 MiB   httpd (7)
---------------------------------
                        245.4 MiB