Topic: Apache Httpd process hangs, no new connections accepted

[asafsteier]

Hi,

I'm using httpd 2.4.52 and openssl 1.1.1m on windows. Very often the httpd process hangs and does not accept any connection. Nothing gets written to the logs. Sometimes the parent process crashes but usually it has to be restarted.

It seems to be happening at times when child process exists (graceful restart on max connections) and there's a number of websocket connections that are present at that time. MaxConnectionsPerChild is set to 2M.

error: HEAP_CORRUPTION

Backtrace:
ntdll.dll!NtWaitForSingleObject () Unknown
ntdll.dll!RtlpWaitOnCriticalSection() Unknown
ntdll.dll!RtlpEnterCriticalSectionContended () Unknown
ntdll.dll!RtlpFreeHeap() Unknown
ntdll.dll!RtlFreeHeap() Unknown
ucrtbase.dll!_free_base() Unknown
libapr-1.dll!apr_allocator_destroy(apr_allocator_t * allocator) Line 193 C
libapr-1.dll!apr_pool_destroy(apr_pool_t * pool) Line 1054 C
libapr-1.dll!apr_pool_destroy(apr_pool_t * pool) Line 997 C
libhttpd.dll!child_main(apr_pool_t * pconf, unsigned long parent_pid) Line 1283 C
libhttpd.dll!winnt_run(apr_pool_t * _pconf, apr_pool_t * plog, server_rec * s) Line 1721 C
libhttpd.dll!ap_run_mpm(apr_pool_t * pconf, apr_pool_t * plog, server_rec * s) Line 95 C
httpd.exe!main(int argc, const char * const * argv) Line 819 C
[Inline Frame] httpd.exe!invoke_main() Line 64 C++
httpd.exe!__scrt_common_main_seh() Line 253 C++
kernel32.dll!BaseThreadInitThunk () Unknown
ntdll.dll!RtlUserThreadStart () Unknown

Log snippet:
ssl_engine_io.c(2411): [client 10.10.10.10:65555] OpenSSL: I/O error, 5 bytes expected to read on BIO#a2093f0ab0 [mem: a2130776f3]
ssl_engine_io.c(2411): [client 10.10.10.10:59999] OpenSSL: I/O error, 5 bytes expected to read on BIO#a209591490 [mem: a218f07023]
child.c(1175): AH00361: Child: 1452 threads blocked on the completion port
ssl_engine_io.c(2411): [client 10.10.10.10:58888] OpenSSL: I/O error, 5 bytes expected to read on BIO#a2093f1130 [mem: a216fe82f3]
ssl_engine_io.c(2401): [client 10.10.10.10:65555] OpenSSL: write 31/31 bytes to BIO#a2093efcb0 [mem: a21b75d133]
ssl_engine_io.c(2401): [client 10.10.10.10:59999] OpenSSL: write 31/31 bytes to BIO#a209592010 [mem: a2078cf583]
ssl_engine_kernel.c(2223): [client 10.10.10.10:65555] OpenSSL: Write: SSL negotiation finished successfully
ssl_engine_kernel.c(2223): [client 10.10.10.10:59999] OpenSSL: Write: SSL negotiation finished successfully
ssl_engine_io.c(1147): [client 10.10.10.10:65555] AH02001: Connection closed to child 833 with standard shutdown
ssl_engine_io.c(1147): [client 10.10.10.10:59999] AH02001: Connection closed to child 4177 with standard shutdown
ssl_engine_io.c(2401): [client 10.10.10.10:58888] OpenSSL: write 31/31 bytes to BIO#a2093f1830 [mem: a216fe82f3]
ssl_engine_kernel.c(2223): [client 10.10.10.10:58888] OpenSSL: Write: SSL negotiation finished successfully
ssl_engine_io.c(1147): [client 10.10.10.10:58888] AH02001: Connection closed to child 609 with standard shutdown
AH00363: Child: Terminating 1408 threads that failed to exit.
AH00364: Child: All worker threads have exited.
mod_socache_shmcb.c(570): AH00837: socache_shmcb_remove (0x41 -> subcache 1)
mod_socache_shmcb.c(585): AH00839: leaving socache_shmcb_remove successfully
ssl_engine_kernel.c(2061): Inter-Process Session Cache: request=REM status=OK id=41fa32c2f3782d8db1894c3bffdbb7b20bca8ba9bc200dae45e318240df0050e (session dead)
ssl_engine_io.c(1853): [client 10.10.10.10:54444] coalesce: have 0 bytes, adding 6 more (buckets=3)
ssl_engine_io.c(1916): [client 10.10.10.10:54444] coalesce: passing on 6 bytes
ssl_engine_io.c(1853): [client 10.10.10.10:62222] coalesce: have 0 bytes, adding 15 more (buckets=3)
ssl_engine_io.c(1916): [client 10.10.10.10:62222] coalesce: passing on 15 bytes
ssl_engine_io.c(1853): [client 10.10.10.10:59999] coalesce: have 0 bytes, adding 15 more (buckets=3)
ssl_engine_io.c(1916): [client 10.10.10.10:59999] coalesce: passing on 15 bytes
ssl_engine_io.c(1853): [client 10.10.10.10:57777] coalesce: have 0 bytes, adding 6 more (buckets=3)
ssl_engine_io.c(1916): [client 10.10.10.10:57777] coalesce: passing on 6 bytes
ssl_engine_io.c(1853): [client 10.10.10.10:63333] coalesce: have 0 bytes, adding 6 more (buckets=3)
ssl_engine_io.c(1916): [client 10.10.10.10:63333] coalesce: passing on 6 bytes


Has anyone encountered a similar issue? Any ideas how to resolve?

Thanks!

[James Blond]

Is this the same as https://www.apachelounge.com/viewtopic.php?t=8877 ?

[asafsteier]

[James Blond]

Continues here https://www.apachelounge.com/viewtopic.php?t=8877