| Author |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
 Posted: Sun 05 Apr '20 10:21 Post subject: Apache 2.4.43 / Windows 10 / Crash of libhttpd.dll |
 |
|
Hi,
since a while I use Apache x64 2.4.41 for the staging of my website. It runs on latest Windows 10 (all updates) with PHP 7.4.4 and Perl 5.30.2. It's configured for TLS 1.3 only with "SSLProtocol -all TLSv1.3" and HTTP/2.
Yesterday I updated to Apache x64 2.4.43. I copied the whole "conf" directory from .41 to .43. Most of my site still works. But *all* https calls to Perl URLs now fail. Regardless of direct or using PHP cURL. http works fine. PHP/https also works fine. Perl is configured as "ScriptInterpreterSource registry-strict" using .pl and .cgi settings in the registry.
There are *no* error messages at all in the browser. Also *no* error messages when I instruct cURL to write a verbose logfile. Windows event log shows an app crash of httpd.exe (in detail libhttpd.dll). Exception is "0xc0000005" (access violation) at "Error offset: 0x00000000000266f9".
Where do I start to look for errors? Why fails a fully working .41 environment in .43? If required, I'll upload (parts of) my config files.
Thanks for any help
mdrmdr |
|
| Back to top |
|
glsmith
Moderator
Joined: 16 Oct 2007
Posts: 2268
Location: Sun Diego, USA
|
| Posted: Sun 05 Apr '20 22:17 Post subject: |
|
|
Odd.
Try adding to httpd.conf
ScriptLog logs/script-error.log
See if script-error.log hints at something. |
|
| Back to top |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
| Posted: Mon 06 Apr '20 0:19 Post subject: |
|
|
The ScriptLog file does not even get created.
I now monitored system activity with Sysinternals "process monitor". Immediately after reading the path to perl.exe from the .cgi registry value, it starts "C:\Windows\system32\WerFault.exe" because httpd.exe has crashed. This does not happen with .41.
Here's an excerpt of the output filtered for httpd.exe:
| Code: | 23:56:13,8866572 httpd.exe 12404 RegQueryValue HKCR\.cgi\Shell\ExecCGI\Command\(Default) SUCCESS Type: REG_SZ, Length: 60, Data: X:\Apps\Perl\bin\perl.exe -WT
23:56:13,8866719 httpd.exe 12404 RegCloseKey HKCR\.cgi SUCCESS
23:56:13,8867205 httpd.exe 12404 QueryNameInformationFile X:\Apps\Apache24\bin\httpd.exe SUCCESS Name: \Apps\Apache24\bin\httpd.exe
23:56:13,9697247 httpd.exe 12404 Process Create C:\Windows\system32\WerFault.exe SUCCESS PID: 6652, Command line: C:\Windows\system32\WerFault.exe -u -p 12404 -s 2184 |
|
|
| Back to top |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
| Posted: Mon 06 Apr '20 0:29 Post subject: |
|
|
A call to httpd.exe from the commandline with "httpd.exe -X -e debug" just exits the programm without any further message once I call a https perl script in the browser. Guess this is the same httpd.exe hard crash as if I run it via Apache Monitor.
I'm currently giving up and switching back to .41 which works fine. |
|
| Back to top |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
|
| Back to top |
|
admin
Site Admin
Joined: 15 Oct 2005
Posts: 692
|
| Posted: Wed 08 Apr '20 10:31 Post subject: |
|
|
What you can try:
Replace libhttpd.dll with the one from 41, and see if it errors.
Replace a used module and mentioned in the change log 42/43 one by one with a 41 one, and see if it errors. |
|
| Back to top |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
| Posted: Wed 08 Apr '20 11:59 Post subject: |
|
|
Tried that already but did it again rigth now.
No change. httpd.exe still crashes. Excerpt from the event viewer:
| Code: | Name of the faulty application: httpd.exe, version: 2.4.43.0, timestamp: 0x5e834d57
Name of the faulty module: libhttpd.dll, version: 2.4.41.0, timestamp: 0x5d4d8749
Exception code: 0xc0000005
Error offset: 0x0000000000000015d9
ID of the faulty process: 0x2028 |
|
|
| Back to top |
|
Jan-E
Joined: 09 Mar 2012
Posts: 1266
Location: Amsterdam, NL, EU
|
| Posted: Wed 08 Apr '20 15:36 Post subject: |
|
|
The problem might be a change in an Apache module.
Try disabling as much of the changed modules as possible or replace them by the 2.4.41 versions. Start with replacing mod_cgi with the 2.4.41 version and/or disable mod_http2. If you are using mod_md: disable it as well.
See the changelog: https://www.apachelounge.com/Changelog-2.4.html |
|
| Back to top |
|
glsmith
Moderator
Joined: 16 Oct 2007
Posts: 2268
Location: Sun Diego, USA
|
| Posted: Wed 08 Apr '20 21:17 Post subject: |
|
|
I have a problem believing it is a problem with mod_cgi because I use that all day long and all my cgi scripts work just fine. I use old school #!shebang however and have perl in the system path so I only have to use #!perl. ScriptInterpreterSource is in core, not in mod_cgi.
Comment out ScriptInterpreterSource (defaults to "Script") and add the shebang to the script. Does it then work?
If it does then something's been borked in the core.
Or, did you switch to perl64? It installs into c:\program files\perl64 (unless you tell it different instead of c:\program files\perl and you would need to modify the registry key/s you've set up. |
|
| Back to top |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
| Posted: Wed 08 Apr '20 22:43 Post subject: |
|
|
The replace of the modules uncovered the evil program 
It's mod_http2. Either when I disable http2 in httpd.conf or when I use the .41 module, everything works again.
So one of the changes from .41 to .43 must cause the crash in libhttpd.dll when I call a Perl script and http/2 is active.
Commenting out ScriptInterpreterSource did by the way not help...
And just for complete information: I'm using the latest x64 Strawberry Perl 5.30.2.1. But after my finding, this should not matter anymore.
What to do next? Can you possibly re-create the error yourself now? Should I test with a special debug version of mod_http2? |
|
| Back to top |
|
glsmith
Moderator
Joined: 16 Oct 2007
Posts: 2268
Location: Sun Diego, USA
|
| Posted: Thu 09 Apr '20 4:14 Post subject: |
|
|
Looking into a few files I see trace logging of 1, 2 & 4 so;
LogLevel warn http2:trace4
However, if you have a debugger go ahead and use it as it will find the where it's crashing and show the road taken the bits took to get there.
I'll try to duplicate. I do not use ScriptInterpreterSource but I do use TLS/1.3 and http2.
Just curious, what happens if you allow TLS/1.2? |
|
| Back to top |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
| Posted: Thu 09 Apr '20 9:15 Post subject: |
|
|
Lowering from TLS1.3 to TLS1.2 does not help. I do not have a debugger to run httpd.exe.
I added the LogLevel statement and the result was:
| Code: | [Thu Apr 09 09:07:14.618745 2020] [http2:debug] [pid 8820:tid 608] mod_http2.c(111): AH03089: initializing post config dry run
[Thu Apr 09 09:07:14.642767 2020] [http2:info] [pid 8820:tid 608] AH03090: mod_http2 (v1.15.8, feats=, nghttp2 1.40.0), initializing...
[Thu Apr 09 09:07:14.642767 2020] [http2:trace1] [pid 8820:tid 608] h2_h2.c(447): h2_h2, child_init
[Thu Apr 09 09:07:14.642767 2020] [http2:trace1] [pid 8820:tid 608] h2_switch.c(44): h2_switch init
[Thu Apr 09 09:07:14.651775 2020] [mpm_winnt:notice] [pid 8820:tid 608] AH00455: Apache/2.4.43 (Win64) OpenSSL/1.1.1f PHP/7.4.4 configured -- resuming normal operations
[Thu Apr 09 09:07:14.651775 2020] [mpm_winnt:notice] [pid 8820:tid 608] AH00456: Apache Lounge VS16 Server built: Mar 31 2020 16:01:02
[Thu Apr 09 09:07:14.651775 2020] [core:notice] [pid 8820:tid 608] AH00094: Command line: 'X:\\Apps\\Apache24\\bin\\httpd.exe -d X:/Apps/Apache24'
[Thu Apr 09 09:07:14.653777 2020] [mpm_winnt:notice] [pid 8820:tid 608] AH00418: Parent: Created child process 10560
[Thu Apr 09 09:07:14.918017 2020] [http2:debug] [pid 10560:tid 668] mod_http2.c(111): AH03089: initializing post config dry run
[Thu Apr 09 09:07:14.937034 2020] [http2:info] [pid 10560:tid 668] AH03090: mod_http2 (v1.15.8, feats=, nghttp2 1.40.0), initializing...
[Thu Apr 09 09:07:14.937034 2020] [http2:trace1] [pid 10560:tid 668] h2_h2.c(447): h2_h2, child_init
[Thu Apr 09 09:07:14.937034 2020] [http2:trace1] [pid 10560:tid 668] h2_switch.c(44): h2_switch init
[Thu Apr 09 09:07:14.945041 2020] [http2:trace3] [pid 10560:tid 668] h2_conn.c(133): h2_workers: min=64 max=96, mthrpchild=64, idle_secs=600
[Thu Apr 09 09:07:14.945041 2020] [http2:trace2] [pid 10560:tid 668] h2_workers.c(102): h2_workers: new thread for slot 95
<start of workers deleted>
[Thu Apr 09 09:07:14.948044 2020] [http2:trace2] [pid 10560:tid 668] h2_workers.c(102): h2_workers: new thread for slot 0
[Thu Apr 09 09:07:14.949046 2020] [mpm_winnt:notice] [pid 10560:tid 668] AH00354: Child: Starting 64 worker threads.
<crash below>
[Thu Apr 09 09:07:48.725754 2020] [mpm_winnt:notice] [pid 8820:tid 608] AH00428: Parent: child process 10560 exited with status 3221225477 -- Restarting.
[Thu Apr 09 09:07:48.750777 2020] [http2:info] [pid 8820:tid 608] AH03090: mod_http2 (v1.15.8, feats=, nghttp2 1.40.0), initializing...
<following lines identical to above after "initializing..."> |
The error code 3221225477 is 0xC0000005 in hex, which on Windows is:
#define STATUS_ACCESS_VIOLATION ((NTSTATUS)0xC0000005L)
[read this on stackoverflow] |
|
| Back to top |
|
admin
Site Admin
Joined: 15 Oct 2005
Posts: 692
|
|
| Back to top |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
| Posted: Thu 09 Apr '20 10:42 Post subject: |
|
|
| Great, thanks! I'll check here and at the mailing list regularily... |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3094
Location: Hilversum, NL, EU
|
| Posted: Sat 11 Apr '20 12:21 Post subject: |
|
|
Stefan wrote at dev-list to try the revisions between .41.and .43 :
r1861247
r1864126
r1872230
So here the first Win64 one : Removed
when not ok then build the next
Last edited by Steffen on Sun 12 Apr '20 20:34; edited 1 time in total |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3094
Location: Hilversum, NL, EU
|
| Posted: Sat 11 Apr '20 13:08 Post subject: |
|
|
Just figured out that r1861247 and r1864126 are already in .41 so that should work at you. You can check to be sure.
At the dev-list Stefan says: if we do not get a minimal configuration where you can reproduce.
So. Maybe it helps that you post at dev-list more info of your configuration and you Perl script.
ps.
I do not know why you put things in the registry. I run all kind of scripts just only using the #!shebang line. |
|
| Back to top |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
| Posted: Sat 11 Apr '20 13:50 Post subject: |
|
|
| Steffen wrote: | | Just figured out that r1861247 and r1864126 are already in .41 so that should work at you. You can check to be sure. |
Hm, I only can use .41 (which works) and .43 (which fails). I cannot compile the releases inbetween myself, as I do not have a development environment on my Windows system.
| Steffen wrote: | | At the dev-list Stefan says: if we do not get a minimal configuration where you can reproduce. |
What is this? An excerpt of httpd config files? Which one? All?
| Steffen wrote: | | So. Maybe it helps that you post at dev-list more info of your configuration and you Perl script. |
The Perl script is just a 3-liner direct called in the browser:
| Code: | #!/usr/bin/perl
print "Content-Type: text/html\n\n";
print substr($^V,1);
|
| Steffen wrote: | ps.
I do not know why you put things in the registry. I run all kind of scripts just only using the #!shebang line. |
Because pathes on my prod system (hosted & unix) and my dev system (@home & W10) are different... And, with the correct shebang and registry .cgi disabled, it crashes as well... |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3094
Location: Hilversum, NL, EU
|
| Posted: Sat 11 Apr '20 14:21 Post subject: |
|
|
Just trying to help.
You wrote: Hm, I only can...
You missed my post above that I already build one for you ? Download link to http2-1861247 in above post.
You wrote: What is this? ...
Ask Stefan what info he wants more.
Better to discuss with Stefan at the dev-list, he is the author. |
|
| Back to top |
|
mdrmdr
Joined: 05 Apr 2020
Posts: 16
|
| Posted: Sat 11 Apr '20 15:18 Post subject: |
|
|
I'm terribly sorry. Yes, I've overseen the rar archive.
Just tested it. And ... IT WORKS!
Are you posting this result to the mailing list or should I and ask Stefan what he needs for analysis?
Thanks a lot for your help!
Edit:
Just sent it to the mailing.list... |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3094
Location: Hilversum, NL, EU
|
| Posted: Sun 12 Apr '20 15:43 Post subject: |
|
|
You can report:
By Stefan Advised revisions
r1861247 - works
r1864126 - works
r1872230 - works
Test some more Revisions:
r1873368 - works
r1874286 - works
r1874347 - works
r1874909 (=2.4.43) - fails! |
|
| Back to top |
|
