Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: Apache child process crashing frequently |
|
| Author |
|
tfn
Joined: 23 Jan 2018
Posts: 3
|
 Posted: Tue 23 Jan '18 22:48 Post subject: Apache child process crashing frequently |
 |
|
We have four Windows Servers running Apache 2.4.27 acting as load balancers for our application server cluster running Tomcat. Recently, we have started to experience a high number of crashes with the web servers. Within the Apache error log, we see the following:
| Code: |
[Mon Jan 15 15:12:08.271099 2018] [mpm_winnt:notice] [pid 1696:tid 432] AH00428: Parent: child process 38240 exited with status 3221225477 -- Restarting.
[Mon Jan 15 15:12:08.944108 2018] [mpm_winnt:notice] [pid 1696:tid 432] AH00455: Apache/2.4.27 (Win64) OpenSSL/1.0.2l configured -- resuming normal operations
[Mon Jan 15 15:12:08.944108 2018] [mpm_winnt:notice] [pid 1696:tid 432] AH00456: Apache Lounge VC11 Server built: Jul 10 2017 14:15:02
[Mon Jan 15 15:12:08.957110 2018] [mpm_winnt:notice] [pid 1696:tid 432] AH00418: Parent: Created child process 43540
|
Between the four web servers, we often see over a dozen such events during the day - sometimes more, sometimes less. In some cases Apache will crash after the child process was restarted only 5 minutes before. The number of crashes goes down significantly during the night and weekends, but it still happens. As far as we can tell, we have not made any major changes to the configuration recently.
We were able to get a core dump as one of them was crashing. The following are some pieces extracted from it:
| Code: |
FAULTING_IP:
libaprutil_1!apr_brigade_writev+37a
00000000`6f8f21da 488908 mov qword ptr [rax],rcx
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 000000006f8f21da (libaprutil_1!apr_brigade_writev+0x000000000000037a)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 0000000000000000
Attempt to write to address 0000000000000000
STACK_TEXT:
libaprutil_1!apr_brigade_writev+0x37a
libapr_1!apr_pool_destroy+0x6e
libaprutil_1!apr_brigade_cleanup+0x43
mod_ssl!ssl_run_init_server+0x2ddf
mod_ssl!ssl_run_init_server+0x1cf5
libhttpd!ap_process_request_after_handler+0x5c
libhttpd!ap_process_request+0x17
libhttpd!ap_sys_privileges_handlers+0x3953
libhttpd!ap_run_process_connection+0x35
libhttpd!ap_process_connection+0x45
libhttpd!ap_regkey_value_set+0x21f3
kernel32!BaseThreadInitThunk+0x22
ntdll!RtlUserThreadStart+0x34
|
Looking at the Windows Event Viewer, we see the modules "libaprutil-1" and "libapr-1" as the faulting modules when the crashes occur. On a rare occasion we will see "ntdll" and "libhttpd" as the faulting module.
We have tried to increase the thread stack size (based on similar reports online) but that didn't help. We've enabled forensic logging, trying to determine if there was some sort of rogue request that could be knocking us over, but nothing seemed really out of place.
Is there something we can do to alleviate this issue or is this a bug within Apache? |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3094
Location: Hilversum, NL, EU
|
| Posted: Wed 24 Jan '18 12:09 Post subject: |
|
|
Crashes are mostly caused by by external modules.
Running PHP or other external module ?
Updating to 2.4.29, same issue ? |
|
| Back to top |
|
tfn
Joined: 23 Jan 2018
Posts: 3
|
| Posted: Wed 24 Jan '18 15:03 Post subject: |
|
|
Based on the Apache configuration file, these are the modules we have loaded:
| Code: |
access_compat_module
actions_module
alias_module
allowmethods_module
asis_module
auth_basic_module
authn_core_module
authn_file_module
authz_core_module
authz_groupfile_module
authz_host_module
authz_user_module
autoindex_module
cgi_module
deflate_module
dir_module
env_module
filter_module
headers_module
include_module
isapi_module
lbmethod_byrequests_module
logio_module
log_config_module
log_forensic_module
mime_module
negotiation_module
proxy_module
proxy_balancer_module
proxy_http_module
proxy_wstunnel_module
reqtimeout_module
rewrite_module
setenvif_module
slotmem_shm_module
socache_shmcb_module
ssl_module
status_module
unique_id_module
|
We are not using PHP or anything similar. Is there a way to know which, if any, module is causing the issue? It looks like the module mod_ssl is on the exception stack; is it fair to say that the SSL module is causing the problems?
We have not upgraded to the latest version of Apache yet. Since this is a production environment, we've been hesitant to upgrade especially since the release notes don't give any indication that it solved something along this issue. Additionally, we've been running the same version of Apache for a while now in production, but these issues only started becoming a major issue in the last few weeks. |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3094
Location: Hilversum, NL, EU
|
|
| Back to top |
|
tfn
Joined: 23 Jan 2018
Posts: 3
|
| Posted: Wed 24 Jan '18 16:23 Post subject: |
|
|
No, nothing as far as we can tell configuration wise has changed recently.
We will post our situation to the Apache mailing list.
Thanks for the help. |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7373
Location: Germany, Next to Hamburg
|
| Posted: Thu 08 Feb '18 16:13 Post subject: |
|
|
| Code: |
c0000005 (Access violation)
|
I had such an issue with a loadbalancer setup. I had to run apache with a different local user (adminstrator permission) (not the system account) to access the needed shared memory. |
|
| Back to top |
|
|
|
|
|
|
