Topic: Apache 2.4.3 - SSL Requests Stop working after a while

[zimmerru]

So I just upgraded Apache 2.2.22 to Apache 2.4.3 and made sure to go through all the options that had changed and update the conf file accordingly. This included adding the cache module for SSL and changing the SSLMutex option over to Mutex default ssl-cache. We also turned off SSLCompression due to the CRIME attack vulnerability.

We use apache strictly as a loadbalancer to 2 tomcat servers via mod_jk. Apache serves no static content at this time.

After being deployed, everything worked fine until later in the day. After 3 hours of heavier load (our site only takes significant traffic during business hours), the SSL requests stopped working/being answered. However if you requested the same page via http instead of https, it worked fine.

Restarting the Apache server fixes this, for a while. Again after a few hours of traffic, the https requests stopped working again. This time I turned the loglevel up to debug and restarted the Apache server.

As traffic slowed down it took another 6 or 7 hours before SSL requests stopped working again. I checked the logs, and nothing notable, the mod_ssl entries just... stopped. (I don't know for sure its ammount of traffic related, it just seems that way)

Any ideas? Let me know if I should post anything specific for review.

I have tried reproducing this in a lab, but have not been able to get it to happen on the lab server.

OS: Windows Server 2008 R2
Apache: 2.4.3 vc9 build with OpenSSL 0.9.8 downloaded from apachelounge.org
Mod_JK Version 1.2.37 vc9 also downloaded from apachelounge.

[admin]

Do you have:

AcceptFilter http none
AcceptFilter https none
EnableSendfile off
EnableMMAP off

?

[zimmerru]

I do not, I was going to add that to address the AcceptEx Failed messages, but didn't want to introduce something new before I figured out what the cause of this issue was.

I am assuming there is a link between the AcceptEx Failed messages and the issue I am having?

[admin]

Yep, that 4 settings solves for many issues.

[zimmerru]

Since I will need to provide documentation, can you explain how those settings are related to the issue I am having? I can't put it back into production and risk an outage without understanding how they are related.

[admin]

Never be sure if this is related. But those settings solves for many stopping issues. No risks to use these settings, they are common and used by most of the users.

Read for explanation of the directives: http://httpd.apache.org/docs/2.4/mod/quickreference.html

[glsmith]

If you are seeing these errors in the error log;

[Fri Oct 05 12:05:37.238437 2012] [mpm_winnt:warn] [pid 3704:tid 940] (OS 64)The specified network name is no longer available. : AH00341: winnt_accept: Asynchronous AcceptEx failed.

It's definitely the AcceptFilter parts of the above suggestion.

[zimmerru]

Yes I do see those errors in the error log, but what I'm trying to figure out, is how those errors are related to the SSL requests failing after a while.


Also, i'm not worried about using the AcceptFilter https none setting, I don't want to risk the setting not fixing my issue, and having another outage in production. Hence why I am trying to understand the link between the AcceptEx errors and the issue I am having with SSL failing.

I am Heading to the link admin provided now to see if it sheds some light on things...

[glsmith]

I do not know how to explain it, but if you ask the dev list at Apache they will tell you it's buggy network drivers. We do not believe that answer but many have the problem, many do not and we here do not know just what triggers it.

But, since you are having the problem with AcceptEx loosing it, then you will need to use the AcceptFilter http/https none. This makes the server use accept() and not AcceptEx() and accept() seems to be immune to any problem.

In 2.0/2.2, it was Win32DisableAcceptEx which covered all protocols (as the same problem exists in it), they've spit it into separate protocols in 2.4.

Anyhow, here is the ASFs explination;

AcceptEx() is a Microsoft WinSock v2 API that provides some performance improvements over the use of the BSD style accept() API in certain circumstances. Some popular Windows products, typically virus scanning or virtual private network packages, have bugs that interfere with the proper operation of AcceptEx(). If you encounter an error condition like:

[error] (730038)An operation was attempted on something that is not a socket.: winnt_accept: AcceptEx failed. Attempting to recover.

you should use this directive to disable the use of AcceptEx().

[zimmerru]

Ok, that makes some sense. I do have the Win32DisableAcceptEX directive in my Apache 2.2 configuration, have for a couple years now, so I had already planned to add it's equivalent to the Apache 2.4.3 configuration but ran into this problem before I got it in place.

We are going to put 2.4.3 into production over the weekend and see if we can re-create the problem with a load testing tool, so we'll see how it goes. Thanks for the information and assitance!

[itest]

zimmerru, we have the same SSL issue on Apache 2.4.3 with the ssl site. I wonder if the issue has been resolved by addding the following directives as recommended in this post:

AcceptFilter http none
AcceptFilter https none
EnableSendfile off
EnableMMAP off