My Drupal 6 site has been running smoothly for years but recently has experienced intermittent periods of extreme slowness (10-60 sec page loads). Several hours of slowness followed by hours of normal (4-6 sec) page loads. The page always loads with no error, just sometimes takes forever.
------Edit----- Updated Question:
How can I troubleshoot this problem?
- windows task manager
- apache and windows logs
- Firewall packet capture
- Windows Server 2003
- Apache/2.2.15 (Win32) Jrun/4.0
- PHP 5 MySql 5.1
- Drupal 6
- Cold fusion 9
- Vmware virtual environment
- DMZ behind a corporate firewall
Traffic: 1-3 hits/sec avg
- No applicable errors in apache error log
- No errors in drupal event log
- Drupal devel module shows 242 queries in 366.23 milliseconds,page execution time 2069.62 ms. (So it looks like queries and php scripts are not the problem)
- NO unusually high CPU, memory, or disk IO
- Cold fusion apps, and other static pages outside of drupal also load slow
- webpagetest.org test shows very high time-to-first-byte
The problem seems to be with Apache responding to requests, but previously I've only seen this behavior under 100% cpu load. Judging solely by resource monitoring, it looks as though very little is going on.
Here is the kicker - roughly half of the site's access comes from our LAN, but if I disable the firewall rule and block access from outside of our network, internal (LAN) access (1000+ devices) is speedy. But as soon as outside access is restored the site is crippled.
Apache config? Crawlers/bots? Attackers? I'm at the end of my rope, where should I be looking to determine where the problem lies?
Attached a waterfall chart from webpagetest.org showing a 15 second load time, I've seen times as high as several minutes. And again, the server runs fine much of the time. The green areas indicate that the browser has sent a request and is waiting to recieve the first byte of data back from the server. This is certainly a back-end delay, but it is puzzling that the CPU is barely used during this slowness.