Topic: Getting ntdll!ExpInterlockedPopEntrySList on Windows 2003

[xyzcoder]

Built Apache 2.2.6 with openssl 0.98g on Windows XP SP2 with VC8 and RC 2 SDK and it works fine. When I try to run it on Windows 2003 Server I get a DrWatson c0000005 with ntdll!ExpInterlockedPopEntrySList for httpd. When I remove mod_ssl it works fine as well. No errors in the Apache logs, but dies after:
[Mon Nov 19 16:06:33 2007] [debug] ssl_engine_init.c(747): Configuring RSA server private key
[Mon Nov 19 16:06:33 2007] [info] Loading certificate & private key of SSL-aware server
[Mon Nov 19 16:06:33 2007] [debug] ssl_engine_pphrase.c(469): unencrypted RSA private key - pass phrase not required

Any idea? Thanks!

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\system32\ntdll.dll -
function: ntdll!ExpInterlockedPopEntrySListFault
7c81bcf0 90 nop
7c81bcf1 53 push ebx
7c81bcf2 55 push ebp
7c81bcf3 8be9 mov ebp,ecx
ntdll!ExpInterlockedPopEntrySListResume:
7c81bcf5 8b5504 mov edx,[ebp+0x4]
7c81bcf8 8b4500 mov eax,[ebp]
7c81bcfb 0bc0 or eax,eax
7c81bcfd 740c jz ntdll!ExpInterlockedPopEntrySListEnd+0x7 (7c81bd0b)
7c81bcff 8d4aff lea ecx,[edx-0x1]
FAULT ->ntdll!ExpInterlockedPopEntrySListFault:

[tdonovan]

A couple of questions:

* Is there a Stack Back Trace section of your DrWatson log? This section is more useful than the State Info section you posted.
Make sure you run Drwtsn32 and check the "Dump All Thread Contexts" box before running Apache to get a Stack Back Trace.

* Does the problem still occur if you run OpenSSL without Apache?

Code:

C:\Apache2\bin> openssl.exe  s_server  -cert ..\conf\server.crt  -key ..\conf\server.key  -www


If this runs without error, try browsing to https://localhost:4433/ while it is running.

-tom-

[xyzcoder]

Openssl as a web server w/o Apache works fine.
"Dump All Thread Contexts" was checked but there is not much info in the Stack Back Trace.

*----> Stack Back Trace <----*
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
009d0718 006d0026 01000089 00000963 0000091b ntdll!ExpInterlockedPopEntrySListFault
00000263 00000000 00000000 00000000 00000000 0x6d0026

Thanks!

[tdonovan]

Mysterious!

The function ntdll!ExpInterlockedPopEntrySListFault claims to have been called from address 00000000.
I doubt that could really be true.

FWIW - I built Apache 2.2.6 & openssl 0.98g with the same tools (VC8-SP1/WinSDK-RC2) and tried it on Windows 2003 Server (Enterprise) with SSL.
I didn't have any problems, so that doesn't help you directly
It does show the problem is something in your specific environment though, not a general Win2003-Svr/Apache problem.

Does your Win2003 Server have the Data Execution Prevention (DEP) feature?
If so, you might try turning it off for httpd.exe and see if this makes any difference.

Anything relevant in the Windows Event Viewer for the time of the failure?

Same symptoms when run from the command-line and as a Service?

-tom-

[xyzcoder]

Turned DEP off, now the DrWatson is gone but httpd still does not start.
Now I get these errors in the Apache error log, the certificates are good since they work with openssl and the same Apache on a different system:
[Tue Nov 20 13:19:23 2007] [info] Configuring server for SSL protocol
[Tue Nov 20 13:19:23 2007] [debug] ssl_engine_init.c(384): Creating new SSL context (protocols: SSLv2, SSLv3, TLSv1)
[Tue Nov 20 13:19:23 2007] [debug] ssl_engine_init.c(580): Configuring permitted SSL ciphers [ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL:!EXP-DES-CBC-SHA:!EXP-RC2-CBC-MD5:!EXP-RC4-MD5:!DES-CBC-SHA]
[Tue Nov 20 13:19:23 2007] [debug] ssl_engine_init.c(708): Configuring RSA server certificate
[Tue Nov 20 13:19:23 2007] [error] Unable to import RSA server certificate
[Tue Nov 20 13:19:23 2007] [error] SSL Library Error: 218529960 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Tue Nov 20 13:19:23 2007] [error] SSL Library Error: 218595386 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error

There certificates were created with the openssl binary from this build. Running an openssl rsa -noout -modulus -in certname | openssl md5 for vert and key gives the same check sum.

I am currently running from the cmd line.

[tdonovan]

A couple of things worth checking:

* Generate a new test certificate on the win2003 system (with the new machine name).

* Double-check that your SSLCertificateFile and SSLCertificateKeyFile directives really point to the correct files on the win2003 system.

-tom-

[xyzcoder]

Created a new self signed certificate on the system and tested it with openssl as web server. Works fine. When I started Apache I got another Dr. Watson, this time for LIBEAY32
*----> Stack Back Trace <----*
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00e24030 00e25ad8 00000000 00000000 00030004 LIBEAY32!ASN1_get_object+0x20
00000004 00000000 00000000 00000000 00000000 0xe25ad8

Rebuild the code with the instructions from the apache web site, created a new cert, same problem, slightly different error message:
[Tue Nov 20 16:23:10 2007] [error] SSL Library Error: 218570875 error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long
[Tue Nov 20 16:23:10 2007] [error] SSL Library Error: 218529894 error:0D068066:asn1 encoding routines:ASN1_CHECK_TLEN:bad object header
[Tue Nov 20 16:23:10 2007] [error] SSL Library Error: 218595386 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error

Tested on a few other systems, same problem.The 2.2.3 Apache I built previously works fine.
Thanks for your time.

[xyzcoder]

I downloaded the ApacheLounge 2.2.6 binaries for a test and I get the same errors .

[glsmith]

Curious what a VC6 version would do on this environment.

[xyzcoder]

I downloaded the Apache 2.2.6 binaries from ASF (assuming they are VC6) and used the certificate I created yesterday with the VC8 openssl and it works just fine. No more crashes or certificate loading issues. DEP is not required either.

[glsmith]

This is what I figured. There is something about that VC8 that leaves me with a bad taste in my mouth! It doesn't seem to handle every situation that can be thrown at it. I am going to hate having to move to it eventually .. or actually 2008. Going to hate having to cough up the $$$ even more!

[tdonovan]

xyzcoder: Glad to hear that you are up and running, but it is a puzzle why you need a VC6 build for Win2003/OpenSSL when others don't.
Especially since you need it on several systems, and you didn't need it for Apache 2.2.3/OpenSSL 0.9.8b.

When you built OpenSSL, did you use MASM? NASM?
Are you using VC8 SP1? Doing anything different from the instructions here?

glsmith: There is a free version of VC8 which builds Apache. Ditto for VC9 (haven't used this one yet). No $$$ are necessary.

-tom-

[xyzcoder]

I tried several different sets of build instructions, the result is always the same and I am using MASM with the professional version of VC8. Tried building with and without SP1, with and w/o zlib.
I am excluding all patented ciphers, e.g.: perl Configure no-mdc2 no-rc5 no-idea VC-WIN32.

[xyzcoder]

I tried VC 9, same problem. Went back to VC 7 works fine.