Topic: Apache 2.4 graceful restarts - Immediate thread termination

[doyle]

Hi,

When restarting Apache using httpd -k restart, the Timeout period is not honoured and threads are terminated immediately. This, I believe, leads to lingering workers, or a compromised child process with less threads available than it needs to service requests. After some time and multiple restarts, mixed with spikes in traffic, this leads to Apache downtime. The pattern appears to be:

Full restart of Apache
The child process (httpd.exe) starts with 2408 threads
h2_workers: created with min=1200 max=1800 idle_ms=600000
Starting 1200 worker threads

Graceful restart 1
httpd -k restart
New child process now has ~ 2408 threads
Timeout period is honoured and countdown begins
AH00362:Child: Waiting 270 more seconds for 43 worker threads to finish....
h2_workers: created with min=1200 max=1800 idle_ms=600000
AH00363: Child: Terminating 1 threads that failed to exit.
AH00364: Child: All worker threads have exited.
h2_workers: cleanup 1200 workers (1199 idle)
AH10290: h2_workers: waiting for workers to close, still seeing 1 workers (0 idle) living
AH10291: h2_workers: cleanup, 1 workers (0 idle) did not exit after 5 seconds.
h2_workers: cleanup all workers terminated
h2_workers: cleanup zombie workers joined

Graceful restart 2 (plus all subsequent restarts)
httpd -k restart
Timeout period is NOT honoured and threads are terminated immediately
AH00363: Child: Terminating 459 threads that failed to exit
AH00364: Child: All worker threads have exited.

The new child process now has ~ 2310 threads (should we expect 2408?)

Sometime after (could be hours or days)
Spike in traffic 1 - server runs out of threads and recovers
Spike in traffic 2 - Apache downtime, sometimes requiring a full server restart

Technology
Windows 2019 / Apache 2.4.63 using mod_mpm
PHP 8.2
Reverse proxying from web servers -> app servers for file uploads / downloads

Config
Timeout 300
ThreadsPerChild 1200
MaxConnectionsPerChild 0

Reproducing the issue in a dev environment on a single server
If we start with a small number of threads:
Set ThreadsPerChild 5
Set Timeout 15
Stop/Start Apache fully
Start a couple of slow uploads using curl

[tangent]

This is a complex query.

Your post makes mention of PHP, as well as a proxy configuration, but you don't detail configuration settings relating to those components, nor how they relate to service connection properties, or resource limits.

Moreover, your testing specifically refers to using h2 protocol, which may well be directly relevant to this thread issue. What if any h2 configuration directives have you changed?

Historically, I have some experience with connection (thread) timeout issues when connecting to proxy services, but those systems were primarily Unix based using mpm_worker/mpm_event, rather than mpm_winnt as you will be using on a Windows server.

A quick search of the net highlights the following mod_h2 discussion https://github.com/icing/mod_h2/issues/212 over Apache graceful restarts, which although 4 years old, would appear to be directly relevant to your query, and is detailed. I note that discussion relates to Unix based services rather than Windows, but there may be some useful pointers in there.

You don't say what the reason is for needing the graceful restarts in the first place. Is this due to creeping memory resource issues or what?

Good luck with your investigation, and let us know your findings.