Topic: Apache memory usage after large uploads

[daibach]

Hi,

I've got a setup with Apache 2.2.4 (from this site) running PHP via FastCGI and everything is working fantastically. The only issue I've encountered is when a large file is uploaded (around 300-400MB so far) Apache appeared to swallow up over 1.4Gb of memory (ram + swap) but didn't release it after the upload had finished.

The PHP process handling the request didn't suffer any similar issues.

Does anyone know how to get around this issue or some suggested configurations that might help recover the memory back without affecting the service.

Thanks

[James Blond]

Does PHP handle the Upload or Apache it self?

[daibach]

Well it's a PHP script that processes the upload, but PHP is running via FastCGI. Which makes me think Apache is having the problem.

I'm actually running an application called Moodle and uploading a giant zipped course onto it via it's own admin. From the code it looks like PHP is handling the uploaded file, but it's still going through apache first.

Any ideas?

[James Blond]

The heavy memory usage is caused by the uploads! I think ftp would be a better solution. Of cause PHP can handle that, but that way all incoming data goes through the memory.

[daibach]

Yeah I guessed that the heavy memory usage was caused by the uploads. I do have an alternative method of getting the files up on the server, but eventually I'll need to open up uploads to more people who I can't give access to ftp or my alternative.

Does anyone have any other ideas, is it possible to have apache handle it's uploads differently (like streaming to disk) so it doesn't eat memory?

I don't mind apache using loads of memory, just as long as it gives it back.

Here's a rrd graph of the apache processes memory usage, as you can see the usage jumped after the file upload and then never drops until I restarted the apache service.

[James Blond]

I don't think that 60 MB in memory is large. But depending on your PHP programing you could use PHP wth fast cgi (fcgi). So the process will unload it self after handling that upload.

[daibach]

Yup, your right 60MB isn't large. Unfortunately I didn't have a graph of the other upload of a 400MB which ate 1GB of memory on it's own. Obviously, after a couple of uploads around that size, there isn't going to be much memory left to play with.

I am running PHP with FastCGI and the PHP memory usage appears fine, the processes do unload themselves every so often and free up their memory. Unfortunately it's the httpd.exe process that's being memory hungry.

[James Blond]

Sorry, I missed that from the first post that you already use fastcgi for php.
While or before or after uploading is there anything that apache has to handle?
The first thing you can do to reduce memory usage remove /comment out all
unneeded modules from apache.
Maybe Steffen knows a bit more! I'll askin him soon.

[daibach]

Thanks James,

I did a bit a research and thought I'd post what I found in case anyone else runs into a similar problem and it may help them a little.

I created a bare-bones upload.php file to handle an upload and nothing else, just to eliminate Moodle out of the equation. This still suffered the problem so I removed all possible modules and this still didn't help.

I've since downloaded and installed the Apache.org version to compare and that too had the issue. Then finally I downloaded the latest PHP installer and installed it as a module instead of FastCGI. Surprisingly this didn't eat loads of memory, in fact it barely got over 20Mb (Ram + Swap) total.

So I think my problem lies with either something in mod_fcgid or Apache to do with handling the upload and passing it to the fast cgi process. Unfortunately for me I can't switch to using PHP as a module as I had other problems with stability before switching to FastCGI.

I'll try playing with the various mod_fcgid settings to see if anything works, but other than that I think I'll just give up for now and hope that no one needs to upload large files too often.

Thanks for your help and time though

[tdonovan]

FastCGI (mod_fcgid) does indeed buffer the entire request in memory before sending it along to the php-cgi process.

Looking at the mod_fcgid source code, it does this in a way which can use up to twice the filesize on uploads.

This memory is never released to the OS, but it is (mostly) re-used for subsequent uploads.

This gets very uncomfortable when there are simultaneous uploads because the requests each need large amounts of memory at the same time.

A compromise might be to use mod_fcgid.so for most pages, but use php5apache2_2.dll for the upload page(s).

You could configure the whole directory (the webroot or the moodle directory) to use mod_fcgid,
then use inner <Directory> and <Files> containers to make the upload pages use PHP as a module.
For example:

[Steffen]

I contacted the author of mod_fcgid. Hopefully he comments on this.

Steffen

[daibach]

@tdonovan

Thanks for taking a look, I wouldn't have known what was what looking at the source code for the module. Does explain what's going on though as I am seeing the memory usage jump by an amount almost double than that of the filesize.

Not a bad idea to work around the issue either with the switch between the php module and fastcgi versions just for the upload page. Certainly a good option to take if I haven't a choice and do have to open up the uploads more.

@Steffen

Cheers for contacting the author, I was thinking of asking them next but didn't want to myself without some more knowledge of what was happening.

[Steffen]

From the author Ryan:

[daibach]

Sounds promising, thanks Steffen.

I'll be ready to test it out when you get a version together.

[Steffen]

Please test the new version 2.2.test, I am running it now at the ApacheLounge.

www.apachelounge.com/download/ the file mod_fcgid-2.2-w32.zip

Now mod_fcgid will swap the http request to disk if it's longer than 64k, two directives added:
MaxRequestLen( default 1G byte, return internal server error if http request longer than it)
MaxRequestInMem( default 64k, store the request to tmp file if request longer than it)

Other changes:
Support configuration "PassHeader"
Support apr_shm_remove() in httpd.2.0.X
Support configuration "TimeScore"


Dteffen