Apache :: mod_fcgid 2.2 graceful restart bug back in mod_fcgid 2.3.x

glsmith

Steffen and Tom,

It seems that the old bug is back, or to be more precise, was never fixed.
Tom, your change for fcgid_proc_win.c is in, just not backward compatible as you had made it.

The fixes to fcgid_pm_main.c and fcgid_pm_win.c are not in the current code on SVN.

I have had a problem with mod_fcgid/2.3.5 segfaulting, a child is left hanging so the parent cannot restart itself cause the hanging child is holding onto whatever port. When the child finally dies, the server is dead, for hours one time. I quit using fcgid/2.3.5 because of it.

I just today realized why the segfault has such a nasty outcome, the graceful restart/stop bug in mod_fcgid. I can reproduce the bug with ease (not the segfault) on any machine on any VC version I have at my disposal.

Steffen, it may be prudent to patch 2.3.5 and release an r2?

Tom, it's your bug and your wonderful fix. I would like to do an r2 with your fixes at AH if that is kosher with you.

I cannot find the bug tracker on sourceforge.net, seems to have disappeared, IIRC you had submitted it there. I did however find the email trail on fcgid-users, if that is any help.

http://www.mail-archive.com/mod-fcgid-users@lists.sourceforge.net/msg00225.html

This being sort of on topic, at least the history on the APR side.
https://issues.apache.org/bugzilla/show_bug.cgi?id=43329

I know your time is slim, if I can do any leg work I am happy to. It would be nice to see this in issues.apache.org so it can be fixed, hopefully next release. A quick glance in the SVN and you will see the offending code (or lack thereof of the fix code in the one case) still exists as it did in mod_fcgid/2.2.

Gregg

thijs · Joined: 30 Jul 2010 Posts: 3 Location: Eindhoven, NL

I think I've been experiencing a similar (or related) issue with mod_fcgid 2.3.5 and Apache 2.2.15 (both builds from ApacheLounge).

Apache crashes approximately once every 10 days, always with an access violation, about half of the crashes involve mod_fcgid.so at the following offset:

"Faulting application httpd.exe, version 2.2.15.0, time stamp 0x4b916754, faulting module mod_fcgid.so, version 2.3.5.200, time stamp 0x4b7fc08c, exception code 0xc0000005, fault offset 0x00004ac2, process id 0x11c, application start time 0x01cb185cf3893246."

What's the procedure to reproduce?

tdonovan

Gregg,

re: "I would like to do an r2 with your fixes at AH if that is kosher with you".

No problem. Of course you and Steffen can use any Apache code I ever wrote whenever you like.

Sorry it took me so long to notice your note. I really have been away from Apache for a bit. I'll try to look at getting this into the Apache code base some time next week.

-tom-

glsmith

Thanks Tom.

@thijs, How I reproduce is rather easy, this is not the segfault tho, that I cannot reproduce at will.

Start Apache
Browse a php page, simple phpinfo() works
reload page once or twice
restart Apache right after the reload

Apache will error on restart, show as being stopped in the services window, and two messages in the event log

1. Cannot open logs
2. Cannot open port while in use

At this time, even tho Apache is stopped, the hanging child will still answer requests till the child dies. Till it dies you will see the one child in the process list.

cyclone · Joined: 12 Sep 2006 Posts: 16 Location: St Louis

glsmith, thanks.

I thought I was insane seeing the exact same issue. Was in the process of ripping out fcgid/2.3.5 when I found this post. Any chance of an "r2" with the patch in it?

glsmith

I'll get one built in the next couple days.

glsmith

done finally mod_fcgid 2.3.5b

x86 or x64 take your pick off the download page at
http://www.apachehaus.com

Let me know if it helps.

cyclone · Joined: 12 Sep 2006 Posts: 16 Location: St Louis

cyclone · Joined: 12 Sep 2006 Posts: 16 Location: St Louis

Thanks. Deployed it to all our servers.

Still getting the dreaded 128 errors on occasion from mod_fcgid, but at least now the server does not die (it just gets confused for a couple minutes).

glsmith

confused .... that's funny! Thanks for reporting back.

I've found a bug report #48949 for Linux that sure sounds like the same bug .... they get noisy error messages and hung processes till the machine runs out of ram. I guess that's just as bad as in order of appearance;

The Apache service named reported the following error:
>>> (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : make_sock: could not bind to address 0.0.0.0:80 .

The Apache service named reported the following error:
>>> no listening sockets available, shutting down .

The Apache service named reported the following error:
>>> Unable to open logs

cyclone · Joined: 12 Sep 2006 Posts: 16 Location: St Louis

Confused probably better describes the admin.

Getting a 400 error from "request failed: error reading the headers" that takes 2 minutes to timeout. When it does timeout it crashes Apache and orphans all running php-cgi.exe's

It looks like a valid request that is incomplete versus some really long request as an attempted buffer overflow. So 2.2.16 is not bug free on broken request headers. Curious that it only happens on Core i7 machines and not on any server with other processors.

But as I said with mod_fcgid 2.3.5b at least Apache restarts and gets back to work after a few seconds.

glsmith

Sounds kinda like Slowloris, but even if it's not, mod_reqtimeout should help in this situation as it handles both request headers and bodies. I think it's a tad buggy yet as it's new, but getting cleaned up.

http://marc.info/?l=apache-httpd-dev&m=126735239106154&w=2

flatcircle · Joined: 27 Jun 2006 Posts: 79

I want to switch to mod_fcgid (instead of mod_php).

So even with (latest) mod_fcgid 2.3.5b there are still glitches?

- Is there a version of mod_fcgid which is working as it should be/stable?
- Do you advise to wait for newer version of mod_fcgid before I switch to mod_fcgid?

Regards.

glsmith

I am so not the one to ask as I have an almost php free server. I have two things php and neither garner enough traffic to require moving them off to mod_fcgid. I pass this question off to someone more capable of answering.

However, what use I've put it through, 2.3.5b is a huge improvement. I (personally) do not see the crashes but more importantly, even if it does crash, Apache now recovers instead of just shutting down. So what if a user gets a 500 or whatever on occasion cause most likely php barfed which then made mod_fcgid barf. The user survives it to see another day and now Apache survives it to see another request. I personally am good with that.

As far as waiting, that's your choice. I see no hurry to get a 2.3.6 out the door from their point of view. There is not much that has changed and most of what has been done to 2.3.5b has not been done to the code base ... yet. I gave greetings and all credit to Tom and posted all I know to the bug linked above. That it happens on both Linux & Windows, and Tom has explained what is going on that makes it happen in the mail list archive, maybe it will get some attention soon and a 2.3.6 will be not that far off after all.

who knows?

flatcircle · Joined: 27 Jun 2006 Posts: 79

Anyway, I'll try mod_fcgid on my staging server to see how it works out.

My Mysql and Apache is fine-tuned, the only bottleneck on my server is mod_php which is very resource hungry. I hope that mod_fcgid will reduce the CPU load a littlebit.

- Any idea if Wincache (http://www.iis.net/download/wincacheforphp) does work with mod_fcgid? Some reviews say that Wincache will boost the performance of your website quite a bit.

- Could you provide the config settings of mod_fcgid which will be a good default to start on a Win/Apache server?

Thanks a lot.

glsmith

sample config in the .zip

Kanashii · Joined: 17 Jul 2006 Posts: 155 Location: Porando

eq wincache

I use in my server and work wery fine

OK i test ab.exe my server for some high usage and now i had proble

FcgidIdleScanInterval 0

process is ~96 when idle max 256 and die when reach to 500 has graceful kil
#MaxRequestsPerProcess [ not work ]

When reach to 300 second idle
FcgidIdleScanInterval 300

In /server-status

Total FastCGI processes: 0

Errro 503 and can't run php-cgi
also
mod_fcgid: can't apply process slot
Exiting(connect error)

Eany ide now i use
FcgidZombieScanInterval 900000000000000000000000
FcgidIdleScanInterval 90000000000000000000000000
and work but somtimes i need restart apache to free memory

Server Version: Apache/2.2.16 (Win32) mod_fcgid/2.3.5
Server Built: Jul 25 2010 13:06:46

Parent Server Generation: 0
Server uptime: 10 minutes 51 seconds
Total accesses: 300029 - Total Traffic: 1.7 GB
461 requests/sec - 2.7 MB/second - 5.9 kB/request
2 requests currently being processed, 1698 idle workers

[Offtopic]
I benchmark IIS 7.5 vs Apache on static file and PHP-CGI
+ Apach is only 2 second slower Smile

- IIS had 5 worker by php-cgi.exe and Apache 250 worker Sad

reenfoo · Joined: 12 Aug 2010 Posts: 2

i'm downloading these and i will test it on different servers.. then post here again..

thanks

wm003 · Joined: 24 Mar 2006 Posts: 88

mmh, could you please check, if your patch is included within the new 2.3.6? I don't think so, because i recently had the same issue again after apache reaches MaxRequestPerChild -> Trying Graceful Restart -> Apache dies

When going back to your 2.3.5b it does not happen.

It would be very nice, if you could do a 2.3.6b aswell if possible Smile

James Blond

The patch is not included in 2.3.6. The patch nerver made it into the offical code. Appling the patch aginst 2.3.6 doesn't work cause the code has changed too much.
I whish my C was bettter so I could figure it out myself how to fix that.