Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: Child process restarting 2021 Edition |
|
| Author |
|
Simbiat
Joined: 22 Feb 2013
Posts: 35
Location: Moscow
|
 Posted: Wed 17 Mar '21 9:38 Post subject: Child process restarting 2021 Edition |
 |
|
Hi!
Recently I've started experiencing an issue with Apache (originally 2.4.42, but present on 2.4.46 as well), that its child process on Windows 10 are restarting leaving this in log (cold start, just trying to open a page on test server):
| Quote: | [Wed Mar 17 10:12:41.219377 2021] [:notice] [pid 9344:tid 660] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Wed Mar 17 10:12:41.219873 2021] [:notice] [pid 9344:tid 660] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"
[Wed Mar 17 10:12:41.219873 2021] [:notice] [pid 9344:tid 660] ModSecurity: PCRE compiled version="8.43 "; loaded version="8.44 2020-02-12"
[Wed Mar 17 10:12:41.219873 2021] [:warn] [pid 9344:tid 660] ModSecurity: Loaded PCRE do not match with compiled!
[Wed Mar 17 10:12:41.219873 2021] [:notice] [pid 9344:tid 660] ModSecurity: LUA compiled version="Lua 5.2"
[Wed Mar 17 10:12:41.219873 2021] [:notice] [pid 9344:tid 660] ModSecurity: YAJL compiled version="2.1.0"
[Wed Mar 17 10:12:41.219873 2021] [:notice] [pid 9344:tid 660] ModSecurity: LIBXML compiled version="2.9.9"
[Wed Mar 17 10:12:41.219873 2021] [:notice] [pid 9344:tid 660] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Wed Mar 17 10:12:42.014631 2021] [mpm_winnt:notice] [pid 9344:tid 660] AH00455: Apache/2.4.46 (Win64) OpenSSL/1.1.1j configured -- resuming normal operations
[Wed Mar 17 10:12:42.014631 2021] [mpm_winnt:notice] [pid 9344:tid 660] AH00456: Apache Lounge VS16 Server built: Feb 17 2021 13:11:14
[Wed Mar 17 10:12:42.014631 2021] [core:notice] [pid 9344:tid 660] AH00094: Command line: 'C:\\Users\\simbi\\OneDrive\\Documents\\!Personal\\Coding\\WebServer\\apache\\bin\\httpd.exe -d C:/Users/simbi/OneDrive/Documents/!Personal/Coding/WebServer/apache -f C:\\Users\\simbi\\OneDrive\\Documents\\!Personal\\Coding\\WebServer\\gfc\\apa.gfc'
[Wed Mar 17 10:12:42.021574 2021] [mpm_winnt:notice] [pid 9344:tid 660] AH00418: Parent: Created child process 5368
[Wed Mar 17 10:12:42.277381 2021] [:notice] [pid 5368:tid 560] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Wed Mar 17 10:12:42.277877 2021] [:notice] [pid 5368:tid 560] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"
[Wed Mar 17 10:12:42.277877 2021] [:notice] [pid 5368:tid 560] ModSecurity: PCRE compiled version="8.43 "; loaded version="8.44 2020-02-12"
[Wed Mar 17 10:12:42.277877 2021] [:warn] [pid 5368:tid 560] ModSecurity: Loaded PCRE do not match with compiled!
[Wed Mar 17 10:12:42.277877 2021] [:notice] [pid 5368:tid 560] ModSecurity: LUA compiled version="Lua 5.2"
[Wed Mar 17 10:12:42.277877 2021] [:notice] [pid 5368:tid 560] ModSecurity: YAJL compiled version="2.1.0"
[Wed Mar 17 10:12:42.277877 2021] [:notice] [pid 5368:tid 560] ModSecurity: LIBXML compiled version="2.9.9"
[Wed Mar 17 10:12:42.277877 2021] [:notice] [pid 5368:tid 560] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Wed Mar 17 10:12:42.327477 2021] [mpm_winnt:notice] [pid 5368:tid 560] AH00354: Child: Starting 150 worker threads.
[Wed Mar 17 10:13:25.183834 2021] [mpm_winnt:notice] [pid 9344:tid 660] AH00428: Parent: child process 5368 exited with status 255 -- Restarting.
[Wed Mar 17 10:13:25.222461 2021] [mpm_winnt:notice] [pid 9344:tid 660] AH00455: Apache/2.4.46 (Win64) OpenSSL/1.1.1j configured -- resuming normal operations
[Wed Mar 17 10:13:25.222461 2021] [mpm_winnt:notice] [pid 9344:tid 660] AH00456: Apache Lounge VS16 Server built: Feb 17 2021 13:11:14
[Wed Mar 17 10:13:25.222461 2021] [core:notice] [pid 9344:tid 660] AH00094: Command line: 'C:\\Users\\simbi\\OneDrive\\Documents\\!Personal\\Coding\\WebServer\\apache\\bin\\httpd.exe -d C:/Users/simbi/OneDrive/Documents/!Personal/Coding/WebServer/apache -f C:\\Users\\simbi\\OneDrive\\Documents\\!Personal\\Coding\\WebServer\\gfc\\apa.gfc'
[Wed Mar 17 10:13:25.227916 2021] [mpm_winnt:notice] [pid 9344:tid 660] AH00418: Parent: Created child process 31404
[Wed Mar 17 10:13:25.473891 2021] [:notice] [pid 31404:tid 564] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Wed Mar 17 10:13:25.474387 2021] [:notice] [pid 31404:tid 564] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"
[Wed Mar 17 10:13:25.474387 2021] [:notice] [pid 31404:tid 564] ModSecurity: PCRE compiled version="8.43 "; loaded version="8.44 2020-02-12"
[Wed Mar 17 10:13:25.474387 2021] [:warn] [pid 31404:tid 564] ModSecurity: Loaded PCRE do not match with compiled!
[Wed Mar 17 10:13:25.474387 2021] [:notice] [pid 31404:tid 564] ModSecurity: LUA compiled version="Lua 5.2"
[Wed Mar 17 10:13:25.474387 2021] [:notice] [pid 31404:tid 564] ModSecurity: YAJL compiled version="2.1.0"
[Wed Mar 17 10:13:25.474387 2021] [:notice] [pid 31404:tid 564] ModSecurity: LIBXML compiled version="2.9.9"
[Wed Mar 17 10:13:25.474387 2021] [:notice] [pid 31404:tid 564] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Wed Mar 17 10:13:25.526947 2021] [mpm_winnt:notice] [pid 31404:tid 564] AH00354: Child: Starting 150 worker threads.
|
And this in Event Viewer:
| Quote: | Faulting application name: httpd.exe, version: 2.4.46.0, time stamp: 0x602d081d
Faulting module name: VCRUNTIME140.dll, version: 14.28.29910.0, time stamp: 0x602478b9
Exception code: 0xc0000005
Fault offset: 0x0000000000001404
Faulting process id: 0x7aac
Faulting application start time: 0x01d71afd0582f073
Faulting application path: C:\Users\simbi\OneDrive\Documents\!Personal\Coding\WebServer\apache\bin\httpd.exe
Faulting module path: C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll
Report Id: a66a589d-5fa1-437f-b2a6-14be1adee9df
Faulting package full name:
Faulting package-relative application ID: |
I've googled this already and tried:
1. Disabling APC/Opcache
2. Placing Apache library from PHP package to System32
3. Adjusting MPM settings
4. Moving to FCGI mode (which brought this error in event viewer (did not save the error in Apache log): | Quote: |
Faulting application name: php-cgi.exe, version: 8.0.3.0, time stamp: 0x604cba38
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000096
Fault offset: 0x00000000400009d6
Faulting process id: 0x11ec
Faulting application start time: 0x01d71a96ff927b73
Faulting application path: C:\Users\simbi\OneDrive\Documents\!Personal\Coding\WebServer\php\php-cgi.exe
Faulting module path: unknown
Report Id: 20a8ea70-c494-4318-bfdf-bfbeb7488c84
Faulting package full name:
Faulting package-relative application ID:
Windows cannot access the file for one of the following reasons: there is a problem with the network connection, the disk that the file is stored on, or the storage drivers installed on this computer; or the disk is missing. Windows closed the program CGI / FastCGI because of this error.
Program: CGI / FastCGI
File:
The error value is listed in the Additional Data section.
User Action
1. Open the file again. This situation might be a temporary problem that corrects itself when the program runs again.
2. If the file still cannot be accessed and
- It is on the network, your network administrator should verify that there is not a problem with the network and that the server can be contacted.
- It is on a removable disk, for example, a floppy disk or CD-ROM, verify that the disk is fully inserted into the computer.
3. Check and repair the file system by running CHKDSK. To run CHKDSK, click Start, click Run, type CMD, and then click OK. At the command prompt, type CHKDSK /F, and then press ENTER.
4. If the problem persists, restore the file from a backup copy.
5. Determine whether other files on the same disk can be opened. If not, the disk might be damaged. If it is a hard disk, contact your administrator or computer hardware vendor for further assistance.
Additional Data
Error value: 00000000
Disk type: 0 |
So far I've backed to mod_php (it allows proper transparent use of .htaccess which I have to use on PROD hosting server). In addition to above I've noticed 2 things:
1. Sometimes a child process is shown in Task Manager as "SUSPENDED"
2. This seems to be true for PHP 7.4.16 as well
3. Guaranteed restart if code has connection to MySQL (MariaDB in my case, using mysqli and pdo_mysql libraries)
4. If there is no connection to MySQL, it generally does not restart, but there were some cases, when it did restart when accessing phpmyadmin's main page (with login screen), so unclear, if something else may be causing the issue.
5. This may be related to KB5000802, but I am not sure, since there were a bunch of other updates that came to me during the last week, when this started occurring.
6. On browser side this looks like Connection Reset, but it's not always caught. I have a page with several dynamically served images (around 50) and from them only 1 or 2 may fail with "Connection reset"
I am trying to use WinDBG to get some kind of dump, but so far am failing at that: the processes just hangs as soon as I attach WinDBG to it and no pages get loaded at all. I will continue trying, but while I am at it, perhaps, someone else has some ideas, that I can try? Removing update is not exactly an option (short-term at best). |
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 348
Location: UK
|
| Posted: Wed 17 Mar '21 18:50 Post subject: |
|
|
Based on what you've detailed, this does rather suggest one of the recent Windows updates is involved.
Looking at the exception codes in your event log entries, the first is a plain Access Violation (0xc0000005), whilst the one when using FCGI with php-cgi.exe is a Privileged Instruction exception with a process offset of over 1GB (0x00000000400009d6). Both are probably caused by a corrupted stack or a messed up function pointer.
However, recent Windows update KB4589211 (and related others) were "Intel microcode updates for Windows 10", and some people have reported code issues as a result, so if one of your updates was for microcode, it might be worth backing that out to see if you get Apache stability again?
You could also enable Minidumps, and hopefully produce a stack trace of where things are going astray - https://docs.microsoft.com/en-gb/windows/win32/wer/collecting-user-mode-dumps
It may help. |
|
| Back to top |
|
Simbiat
Joined: 22 Feb 2013
Posts: 35
Location: Moscow
|
| Posted: Wed 17 Mar '21 19:46 Post subject: |
|
|
That update is one in the list, it came in yesterday, and issues started before that. I actually hoped, that it will fix this 
Thank you for the link, I will try to get a proper dump tomorrow using it |
|
| Back to top |
|
Simbiat
Joined: 22 Feb 2013
Posts: 35
Location: Moscow
|
| Posted: Thu 18 Mar '21 11:50 Post subject: |
|
|
| I got the dump. Sadly, I do not see symbols for Apache anywhere... Regardless, results of "!analyze -v" command are in https://apaste.info/xifk |
|
| Back to top |
|
Simbiat
Joined: 22 Feb 2013
Posts: 35
Location: Moscow
|
| Posted: Thu 18 Mar '21 11:54 Post subject: |
|
|
Huh... The dump mentioned "realpath". I've tried loading a page without realpath (just a simple echo) - it still crashed. But then I commented out open_basedir... It stopped crashing. What's more it's now literally flying.
My open_basedir was:
| Code: | | open_basedir = "C:/Users/simbi/OneDrive/Documents/!Personal/Coding/WebServer/;C:/Users/simbi/AppData/Local/Temp/;" |
Reported to https://bugs.php.net/bug.php?id=80881 |
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 348
Location: UK
|
| Posted: Thu 18 Mar '21 23:26 Post subject: |
|
|
Interesting. I did raise an eyebrow when your posted Apache paths included a reference to OneDrive.
Based on your findings, and looking at your reported PHP bug, and their follow-ups, it would appear the exceptions occur when open_basedir is set to a path that includes reparse points.
Pending any patch updates they may make to PHP, presume the problem goes away if you serve Apache from a purely local filesystem? |
|
| Back to top |
|
Simbiat
Joined: 22 Feb 2013
Posts: 35
Location: Moscow
|
| Posted: Fri 19 Mar '21 6:53 Post subject: |
|
|
| tangent wrote: |
Pending any patch updates they may make to PHP, presume the problem goes away if you serve Apache from a purely local filesystem? |
Firstly I will try to check the same on official build, as agreed in bug report. Testing with htdocs outside of OneDrive can be next. |
|
| Back to top |
|
Simbiat
Joined: 22 Feb 2013
Posts: 35
Location: Moscow
|
| Posted: Fri 19 Mar '21 10:28 Post subject: |
|
|
Crashed on official build as well: https://apaste.info/utsD (analyzed from WinDBG)
When I moved htdocs to C:/htdocs - it did not fail. But... It looks like it crashes during stopping of the service. No logs in Apache, but here's the debug: https://apaste.info/tX0l
Reports from DebugDiag for both cases - https://1drv.ms/u/s!AsaK_C2Xv0I4gunxGl8Cn2AOkewerww?e=aw1tGd
BTW, where can I get symbols for Apache downloaded from here? |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7371
Location: Germany, Next to Hamburg
|
| Posted: Sun 21 Mar '21 17:35 Post subject: |
|
|
Your error report shows that you use mod_php instead of mod_fcgid
| Code: |
php8apache2_4!php_apache_server_shutdown+0x24
|
See https://www.apachelounge.com/viewtopic.php?t=2394 how to use php over mod_fcgid. |
|
| Back to top |
|
Simbiat
Joined: 22 Feb 2013
Posts: 35
Location: Moscow
|
| Posted: Sun 21 Mar '21 17:43 Post subject: |
|
|
| James Blond wrote: | Your error report shows that you use mod_php instead of mod_fcgid
|
James, because I've tried using it when I initially encountered the error, and it still failed. This is mentioned in original post. I moved back to mod_php because of .htaccess directives, that stop working in fcgid, and I need them enabled same as on PROD, where they do work (test environment has Apache, and PROD one Lightspeed).
Regardless, this is turned out to be an issue with open_basedir in PHP, as covered in respective bug report. |
|
| Back to top |
|
|
|
|
|
|
