Topic: Apache 2.x Memory Leak Mystery Problem ever solved?

[mhilliard]

We have recently moved a PHP-Apache-MySQL application that was running on Debian Linux.

Our current setup is:
Apache/2.0.59 (Win32)
PHP/5.1.6
MySQL 5.0.22
Op Sys: Windows 2003 server SP1 on Dell 2650 with 2GB ram

Our PHP app and Apache starts fine, but soon after 5-6 user connections we see the two Apache.exe processes start to grow and never release it's memory. Soon the Apache.exe process show in Task manager that it is consuming about 290 to 300+ MB of RAM and about this time the Apache server stops responding to new webpage requests.

Restarting the Apache server frees up the trapped memory (unfortunamtely dropping all connections) and then the server starts the process all over again.

We have tried several versions of Apache (2.2.x etc) and PHP 5.x.x and all give us the same memory leak problem.

In researching this problem we have seen it described and confirmed by testing on several other Apache support sites going back many years.

Unfortunately, we can't find anyone who has solved this Apache problem memory issue on Windows.

Does anyone on this site know what causes this perpetual slow memory use?

Are there ANY configuration settings that can help control it?

We urgently need an answer for our non-profit partners who use this app.
Any help or suggestions would be greatly appreciated!!

[Steffen]

First 2.0.59 is a legacy release.

It's a well known issue. PHP is mostly bloating the server and eating memory in time.

I am running PHP as Fastcgi (mod_fcgid at the Apache Lounge) and the memoryusage of httpd.exe is after a long time ~100M and average three php processes of ~10M each, when I was running php as module it was 400-800M.

So running as Fastcgi memory usage of Apache is far less en the Speed is on par when running as module. There are advantages to running PHP with FCGI. Separating the PHP code from the web server removes 'bloat' from the main server, and improves the performance of non-PHP requests. Secondly, having one permanent PHP process as opposed to one per apache process means that shared resources like persistent MySQL connections are used more efficiently. And maybe even more important is stability, since I run FCGI my server never crashed caused by php (extension) errors and out of memory errors.

Together with the author we added lately a fix for memory usage when uploading large files: Now mod_fcgid will swap the http request to disk if it's longer than 64k.

[mhilliard]

Steffen,
Thankyou for your reply! Could you help with three more questions?

This morning with schools starting (and more users) on our Apache 2.0.59 and PHP 5 is already up 400 MB memory and is stopping responding to web requests almost every hour now. We really need to fix our win 2003 server...

Quick Questions for you....

1. We have never used or installed FastCGI or FastCGId, is there a guide or setup doc somewhere we can look at to do this right?

2. What versions of PHP and Apache would you recommend we use to work best with FastCGId?

3. Where is the best place to download software from?

Thanks for help already and thanks in advance for ANY info you can send on how to get and install

Thankyou
Mark H. Minnesota, USA

[galmond]

Mark,
I've been struggling with what seems to be a very similar problem, although I can't give you specific advise about FastCGId. I'm curious if you have made any progress.

I have two very similar servers (apache 2.2.4 php 5.2.4 winxp) and one exhibits the problem and the other doesn't, the problem being that httpd memory is not released after the KeepAliveTimeout period. This occurs whether or not I have the php5_module loaded or not, although that affects the amount of memory that is trapped per request. (~520k when php loaded, ~110k when it is not).

Although FastCGId may help, the problem seems so specific that I would like to solve it before covering it up.
Greg

[mhilliard]

Greg,
I wish I had better news to report, but we have NOT found a good solution yet. we are still on PHP as Module (ISAPI) and Apache 2.0.59 because that has been somewhat stable for us, but the memory leak is still happening daily under a moderate load (10-15+ concurrent users of our web app)

I have been experimenting with changing some Apache settings.
Here are some settings we are trying:

EnableMMAP off
Win32DisableAcceptEx
EnableSendfile off
MaxKeepAliveRequests 100

<IfModule mpm_winnt.c>
ThreadsPerChild 65
MaxRequestsPerChild 10000
</IfModule>

Basically, these settings limit the resources per process/thread (if what read it true) and it does appear that these have helped our server some but I also frequently see the ThreadsPerChild limit almost maxed out when I look at the Apache ServerStutus page (Mod_Statu.so Module).

I have to admit we are experimenting with these settings, based on suggestions from other server admins. they have helped somewhat but are mostly just slowing the leak. I don't think anything has stopped it.

Frankly, I have had it with Apache/PHP on Windows server. They just aren't stable together. If they are, the people who know aren't talking.

My plan is to go back to Linux as soon as my new hardware arrives. I will play with Windows/Apache if I have time, but I will NOT try another production environment until I can verify this problem has been solved.

FastCGI might be a solution, but again this is a solution we don't have time to test and I am not gambling with a production environment again.

Other folks have suggested that PHP caching tools like Xcache, or APC can help, but again, we don't have the luxury of time to install, test them.
We have been using ZEND Optimizer which I have read is not really much help at all.

Good luck Greg, I wish I had more helpful info to share!
Mark H.

[galmond]

Thanks for the update.

I had to set the Win32DisableAcceptEx directive some time ago to prevent very slow request response, and if I could remember what I changed on the server to require that directive I would likely have a clue towards the memory situation.

I've found that setting
MaxRequestsPerChild 100
is a workaround that prevents the httpd child process from eventually running out of memory although it will have an impact on performance.
If each separate request retains 520k of memory and the baseline memory usage of the child process is about 18M, then the child process resets at about 70M (i.e. before running out of memory).

Hope you have better luck with your Linux server.

[mhilliard]

Greg,

Thanks for the tip (e.g. MaxRequestsPerChild 100)

I have to be careful what I change on this system as it is production, but maybe I can do some testing late at night or on weekends to see if this helps.

FYI - Here is a link that has some interesting strategies for Apache Performance tuning tips you might find useful...
http://www.devside.net/articles/apache-performance-tuning

Question for you... How did you test your MaxRequestsPerChild setting?
The default I think is 0 for unlimited right?

So from some reading I did I read about a comparable server (still with mem leak though) using 10,000 so I tried it. pretty much picking a compromise number for something other than "unlimitied"

Did you gradually scale down from unlimited MaxReq..PerChild to the 100 number or did you just move to 100?

Also, how did you monitor your response time, as you mentioned this is impacted?? I'm limited in my ability to "load test"

Finally, maybe you could help me clarify something, I am a newby here..

If you have Apache 2.x or 2.1, or 2.2 you usually (on windows) see just two processes (Apache.exe parent/child with Apache 2.x and HTTPD.EXE parent/child with Apache 2.1 x I think) in the Win Task Mgr.

So when you talk "Process" are we talking just the child process I see in Windows Task Manager? I'm not clear what people are refering to when they speak about "Process" and "Threads" per child.

Is the Child Process you use in your calculations the SAME as the process I am seeing in the Win 23K server Task Manager?

Thanks for any extra info you have time to share.

Mark H.

[galmond]

Hi Mark,

I'm not exactly clear on the distinction between processes and threads either, but as you know, winnt only creates one child process with multiple threads (the one you see in the Win Task Mgr, taskmgr.exe).

I was using the httpd child process mem usage value in taskmgr for the rough calculations I did. I set the MaxRequestsPerChild 100 based observing a consistent memory increase for every page I loaded and wanting to the keep the memory usage below 70M. Then I just kept loading pages until the memory was getting close to 70M, and voila! the child process reset and went back to 18M.

On a heavily loaded server this would be bad for performance because of the interruption as the process resets every 100 requests. As a work-around it prevents me from manually restarting apache periodically.

The slow response time that I observed was based on waiting a few seconds for a page to load...apparently after I installed AVG anti-virus software on the server. Win32DisableAcceptEx resolved this. I've since removed AVG and the server works fine now without the directive.

Something I tried that seems to be effective was to start apache without the mod_cgi.so loadmodule. Now the memory usage is just creeping up by less than 10k periodically. Not sure why that solves things but I can live without serving cgi scripts.

Greg

[mhilliard]

Greg,
Thank you for your information helpful clarification. Your tip sounds like something I need to try myself.

I really appreciate you sharing your process!!

I checked my Apache config and sure enough I have the Load Mod_cgi.so as well on my server.

I don't dare try changing this until my apache guy is back from his vacation but I will certainly test the MaxReq... setting - probably tonight when I dare restart Apache a few times.

Thanks for tips, it really helps to know you're not alone in the dark fighting Apache issues (grin)

Mark H.

[mhilliard]

Greg,
I tried your tip about dropping the MaxRequestsPerchild.
As a compromise I went from 10000 to 5000.

Today I noticed my Apache server gracefully restarted about 11:40am just about when its RAM in Windows Task Mgr was edging upwards of 250MB.

This is usally the point of no return when the Apache server heads to "lockup" or non-response.

The Apache restart was fast (e.g. < 4 seconds) and It appears to keep existing sessions (at least my computers sessions) without losing them.

Like you, I hate to "cover up" problems w/o solving them, but given my setup, I can live with this if it keeps working for my users.

But.... My time in Windows/Apache world maybe short...grin

Thanks again for your help!!!

Thanks to your help I can do something else useful with my day besides watching my server bloat and die.


Mark H.[/quote]

[galmond]

Mark, I'm glad I could help...your efforts have been helpful for me as well. Just to give you another update, it seems that disabling mod_cgi.so was a red herring--I don't think that in itself made any difference. By replicating my installation on another WinXP computer I was able to isolate the main offender: the Win32DisableAcceptEx directive.

Without this directive, the memory bloating is nearly eliminated (reproduced on two computers) although the downside is that I'm expecting to see the error that led me to using this directive in the first place:
The semaphore timeout period has expired. : winnt_accept: Asynchronous AcceptEx failed.

I've seen some suggestions that when Win32DisableAcceptEx is specified, other apache functions are called which may themselves cause memory leaks. More and more I'm thinking that I should be looking for a Linux solution as well!
Greg

[godsyabc]

Our current setup is:

Apache http 2.2.4
PHP 5.2.4 (just upgraded days ago to this release from 5.2.3)
PHPBB 2.0.22
MYSQL 5.0.45
OS/ Win 2k on a Dell with 1.5 GB RAM


I have been experiencing the same sort of problem.
I have 2 dedicated servers having the same issue.
A memory leak that creeps until all available ram has been used (although I am still testing with PHP 5.2.4 and don't know the status of this problem with this version of PHP).

I ran my servers without PHP for years with no problems, but in this past year I added PHP support and began testing with it. This was just as a test at first, but then I decided I really liked PHP and all the great open source software available written in PHP along with the portability and so began to use apps like PHPBB. Upon the addition of PHP to the server is when I noticed this memory issue show up. The problem grew as I used more PHP apps more heavily.

I noticed that the latest version of apache 2.2.6, and the latest version of PHP 5.2.4 had bug fixes that talked about memory leak fixes.

PHP 5.2.4 Change log:
Fixed bug #41713 (Persistent memory consumption on win32 since 5.2). (Dmitry)

Apache 2.2.6 Change log:
*) mod_dbd: Create memory sub-pools for each DB connection and close
DB connections in a pool cleanup function. Ensure prepared statements
are destroyed before DB connection is closed. When using reslists,
prevent segfaults when child processes exit, and stop memory leakage
of ap_dbd_t structures. Avoid use of global s->process->pool, which
isn't destroyed by exiting child processes in most multi-process MPMs.
PR 39985. [Chris Darroch, Nick Kew]

Anybody had any luck with the latest versions of apache 2.2.6, or the newest release of PHP 5.2.4?

[godsyabc]

Here is some more detail on the PHP V 5.2.4 memory leak fix (sounds kind of like what we are seeing?) from the PHP changelog:

Description:
------------
When using php arrays with a lot of entries (about 200000), I figured
out the following problem: PHP sometimes doesn't free all used memory
after completing the request. Apache uses under some circumstances 20-30
MB more RAM after the request. The problem is that this happens per
child. If an Apache runs with 64 threads, it is possible, that httpd.exe
consumes persistently 300-600 MB of RAM, even without any active
request.

Reproduce environment:
- I tested it under an Apache (1.3 and 2.2) environment unter Windows
(XP) (I didn't test it under Linux yet)
- Used PHP-Version: 5.2.3 - the problem was introduced with PHP 5.2.
With PHP 5.1.6 the problem does not appear (but I noticed, that PHP
5.1.6's memory management is much slower than the new one
- Set ThreadsPerChild to 1 in httpd.conf to make sure, you hit always
the same PHP instance and avoid any constraints

[mhilliard]

G,
I read both your emails and it sure looks like it we are on the same problem. I am not an experienced Apache user, so maybe you can explain a bit more what you are doing...

I have a fairly good server, Dell 2650 2GB Ram, dual processors, Fast SCSI HD supporting about 5000 web users. Right now the application is an open source Learning System built in PHP. My requests per minute is not (thankfully) very high yet but I can have 20-30 concurrent users quite frequently.

I have my ThreadsPerChild set to 65 and as I watch the Apache Status page info (refreshed every 5 seconds) I can see that my users frequently use 55 to 60 threads leaving me only a few to use.

My assumption was that it would NOT be good to run out of threads as my users are online. I cannot test this easily on a production system and have been careful to make my changes gradually.

What happens to other users when your threadsPerChild are set to 1?
Do other users wait for service?

Right now I have the server set to recycle-restart the Apache Child process when MaxRequests equals about 5000. This take about a second (if it happens before the RAM leak is too bad e.g. 300+ MB) and usually frees up the lost memory. My tests tell me that existing connections seem to be maintained and users aren't complaining, but I don't like the idea of having to have Apache restrat 2-3 times per day (common for me) on my active customers.

I'd appreciate knowing any info you have about a ThreadsPerChild set to 1 and how it works on a busy server.

Mark
MPls, MN'
USA

[godsyabc]

Mark,

My threadsperchild is set as follows:

# ThreadsPerChild: constant number of worker threads in the server process
# MaxRequestsPerChild: maximum number of requests a server process serves
ThreadsPerChild 250
MaxRequestsPerChild 0

I believe these are default settings.
To date I have not played with this or tracked this in anyway.
I have made adjustments in other areas to defaults, but none that pertain to this problem.

I have played with this apache parameter some
Win32DisableAcceptEx
I am thinking that with it left out/turned off memory management is better, with the penalty that you get a 0.0.0.0 IP in all of your log files rather than the real IP's of users.

I have a number of different applications running, and recently moved some standard apps off onto a shared hosting environment. One of my servers is a little more stripped down, running primarily mail, httpd, and a small amount of PHP. But even this server is showing memory usage that seems higher than I would expect (it seems to have leveled off finally after 3 days of running the newest version of PHP, again higher than I would expect, maybe 150-200 MB higher than without PHP activated). Note on this server Win32DisableAcceptEx was turned ON. Shortly I will trying the same test with it turned OFF.

I am still watching the 2nd dedicated server after upgrading to the latest PHP to see what is happening with it. It is running some custom java apps as well as the Apache, PHP, MYSQL stuff. Note on this server Win32DisableAcceptEx is currently turned OFF.

I guess, at least at the moment that I am taking the approach of searching for software fixes via new releases, because it seems to me that it should be addressed at that level.

It seems you are a number of software versions back from me.
I will try the latest apache when they release the binary installation for it.

G

[godsyabc]

A quick follow up,

I upgraded to apache 2.2.6 and currently have 3 days runtime on 2 servers, with Win32DisableAcceptEx commented out, and memory usage seems to be stable.

Current configuration:
Apache http 2.2.6
PHP 5.2.4
PHPBB 2.0.22
MYSQL 5.0.45
Win 2k OS


I just checked my access logs and they seem to working now!


G