| Author |
|
Sunburst
Joined: 09 Mar 2012
Posts: 20
|
 Posted: Sat 31 Mar '12 10:18 Post subject: Apache 2.4 Server ran out of threads to serve requests |
 |
|
Hi all
I got this error each day for a week:
| Quote: | | Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting |
So I kept increasing ThreadsPerChild until I came to 1024 and this morning my apache server went offline and httpd.exe had taken 99% of the CPU making the whole server run extremely slow (of course) and no sites where working. I restarted the Apache service and the sites were running again.
I have been working some years with the Apache server but I would not call myself an expert on this field. I really need your advice 
The server:
Dedicated Windows 2008 R2 (64 bit) server
1 Gbit internet connection
8 CPU cores
32 RAM
4 HDDs in RAID10 - operation system + hosting files for 50 Joomla sites, some of the sites are big
2 SSDs in RAID1 - hosting the MySQL databases and Apache disk cache (that I still have to learn more about)
Apache + PHP:
Apache/2.4.1 (Win32) mod_fcgid/2.3.7
php-5.4.0-nts-Win32-VC9-x86
My Apache config file:
Removed
My PHP config file:
Removed
Apache log:
Removed
I think that Apache went offline at 06:55:26 (perhaps was too slowly in the hours before) and I restarted Apache at 08:37:32.
The log copy/paste starts at 01:03:47 where the problem keeps building up and... this is night time so there should be only a few people browsing the sites.
While pasting this in I saw that PHP's max_execution_time and max_input_time were way too high. I must have used the values from my dev computer. So I set them down to (reflected in the php.ini copy/paste above):
| Code: | max_execution_time = 300
max_input_time = 300 |
Could the long PHP timeout get apache to go offline as it did? The Apache time has been on 300 all the time.
Another error is that the browsers' loading icon (hourglass) keeps running even though the page seems to be fully loaded (and can be used). It does this on some of sites and in all browser types, but worst in FF. The pages are loaded very quickly but something seems to be hanging. If I reload the page than the browsers’ loading icon stops and I can browser as I like (until it happens again on another page). It did not do this before I got this new server (and I changed to PHP 5.4 and started using the Apache version from Apache Lounge).
Thank you very much for all your advice
Admin note: conf's and logs removed, see Forum Rules in the left menu. |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
|
| Posted: Sat 31 Mar '12 14:49 Post subject: |
|
|
Worth to try Apache 2.2.22. The 2.4 is oh so new, special in the mpm, disk cache area.
Steffen |
|
| Back to top |
|
glsmith
Moderator
Joined: 16 Oct 2007
Posts: 2268
Location: Sun Diego, USA
|
| Posted: Sat 31 Mar '12 18:26 Post subject: |
|
|
I happened to see the stuff before it got removed.
I wouldn't say (or saw no evidence) that you are running out of threads. What I saw was that you reached the MaxConnectionsPerChild and Apache closed the old and spawned new child processes. You do this cause of memory leaks so your note said, but how much leaking are you actually seeing?
At 10,000 connections per child it must not take too long to run through them, there is only once child on Windows IIRC.
I saw fcgid errors that I thought were gone in 2.3.7-dev, that might have something to do with what is going on, why child processes are getting forced killed when the mpm spawns a new child process.
I saw something to do with memory set (MaxMemFree) for 1GB with a note about too high? Possible.
I have absolutely no suggestions, just thought I'd summarize what I remember seeing at 2am last night.
Well, I'd probably raise the MaxConnectionsPerChild at least if for no other reason than to lessen the log pollution every time Apache hits it, and spawns a new child. I'd probably leave MaxMemFree at it's default.
Joomla is php, and mod_fcgid is dealing with it, I'm not fcgid savvy so cannot say much on that subject, nor do I remember the settings you had. There are however, suggestions on mod_fcgid settings here in this forum, could probably search them out easily.
There's a performance tuning Apache on Server 2008 Word document in the "Additional" downloads section, you might want to check it out if you haven't already. |
|
| Back to top |
|
Sunburst
Joined: 09 Mar 2012
Posts: 20
|
| Posted: Sat 31 Mar '12 18:56 Post subject: |
|
|
Hi Steffen
Thank you for your advice. I will try downgrading if the problem remains.
Hi glsmith
Thank you for your advice. I will lower MaxMemFree and increase MaxConnectionsPerChild and see what happens
I have re-posted my conf + log here: http://apache.pastebin.ca/2134322 |
|
| Back to top |
|
Sunburst
Joined: 09 Mar 2012
Posts: 20
|
| Posted: Sat 31 Mar '12 19:03 Post subject: |
|
|
Here is a screenshot from this morning (my time) when the server was down. Perhaps it can be usefull
 |
|
| Back to top |
|
Sunburst
Joined: 09 Mar 2012
Posts: 20
|
| Posted: Sun 01 Apr '12 0:31 Post subject: |
|
|
Nearly every 10'th minute Apache creates this error in the Windows Server Event Log:
| Code: | Faulting application name: php-cgi.exe, version: 5.4.0.0, time stamp: 0x4f4e6d23
Faulting module name: php5.dll, version: 5.4.0.0, time stamp: 0x4f4e7c5f
Exception code: 0xc0000005
Fault offset: 0x0002760b
Faulting process id: 0x2b94
Faulting application start time: 0x01cd0f7edf36b0a4
Faulting application path: C:\Storage\Webserver\Apache1\PHP\php-5.4.0-nts-Win32-VC9-x86\php-cgi.exe
Faulting module path: C:\Storage\Webserver\Apache1\PHP\php-5.4.0-nts-Win32-VC9-x86\php5.dll
Report Id: 17ccb434-7b73-11e1-b61e-00259049bbd5 |
Can this be of any help?
UPDATE:
Found that error myself. It was a newsletter script that ran every 10'th minute via a batch file. I wounder if this added more to the 100%-cpu-error.
Last edited by Sunburst on Sun 01 Apr '12 9:15; edited 1 time in total |
|
| Back to top |
|
minibayit
Joined: 01 Apr 2012
Posts: 2
|
| Posted: Sun 01 Apr '12 9:13 Post subject: |
|
|
I know this thread has been resolved, but why has no-one suggested "server-status" and "server-info".
This is how I discovered I had a similar issue with a php calendar that the google bot managed to crawl till the year -5068 or something! |
|
| Back to top |
|
Sunburst
Joined: 09 Mar 2012
Posts: 20
|
| Posted: Thu 19 Apr '12 8:29 Post subject: |
|
|
The problem came back but I finally got it to work by setting the LogLevel to Error so it didn't spend resources building up EXTREMELY BIG log file (a 60 GB log file just for one site).
So I am on to the next problem... this gigantic security hole in my settings: http://www.apachelounge.com/viewtopic.php?t=4683 |
|
| Back to top |
|
