Topic: Websockets: Server restarts eventually freeze whole server

[patchy]

Hi,

Short version:

I use httpd on Windows as a reverse proxy for a microservice system. Some services communicate over websockets (more precicely: SignalR). From time to time I have to restart the server in order to read a new configuration. I observe an increasing number of threads blocked by the SignalR connections. It's a matter of time until the server completely freezes because no threads are available for other requests.

Details:

I reduced my system as much as possible. I end up with two microservices, A and B. A has a SignalR hub. Both, A and B subscribe to the events of this hub. Thus, there should be two connections.

Now the experiment:

1. Start the two microservices: They repeatedly try to connect, but fail. This is expected, because they are configured to connect via the reverse proxy and httpd is not running yet.
2. Start httpd (Windows Service): As expected, both services establish their connection, confirmed by the service logs and mod_status showing 2 connections.
3. Restart httpd: In real-world, I call
httpd.exe -n "ServiceName" -k restart
programmatically. For this experiment, I call it from Powershell. What happens?
3a. The parent starts a new child and hands over 2 sockets, see error.log on Pastebin (link below)
3b. The parent needs to stop the old child. The old child cannot stop because of the open connections. The old child waits a grace period of 30s before, then it terminates the 2 threads. My services log that their connection was disconnected and attempt to reconnect. At this moment, 2 more connections appear in mod_status. However, I don't see any socket handover in error.log.
4. Repeat httpd restart.
4a. The parent starts a new child and hands over 2 sockets, see error.log. It's still 2 sockets, although I saw 4 connections in mod_status in the previous step.
4b. The parent shuts down the old child. This time, there is no grace period, but 18(!) threads that failed to exit are terminated, see error.log. Both services log disconnect and reconnect. However, no additional connections appear in mod_stats, it remains 4.

When I repeat restarting httpd, most of the time it happens the same as described in step 4. Only difference is a changing number of "threads that failed to exit". But sometimes, additional connections appear in mod_status. I can't reproduce this on purpose. I suspect a race condition how fast the old child is shut down, the new one is started and my services trying to reconnect, but I don't know the httpd source code.


To get my job done, I need to know: What can I do to avoid eventually blocking the server?
Out of curiosity, I also would like to know what excatly happens, how the SignalR connectios are handed over to the next child, why the first restart works different than the other restarts, ...

I appreciate any hint, also if it is just about further investigations!


Additional information

Version: 2.4.41
Some config snippets:

[tangent]

A knatty problem indeed, caused no doubt by the fact websockets are stateful.

To me, if you perform a graceful restart of Apache, to reload the configuration for future connections, those updates are not going to be applied to any threads owned by existing child processes. Connections maybe, if the sockets can be duplicated and handed off to threads in a successor child.

So even though a child has been instructed to exit, any free child threads will block until the timeout period has elapsed on any existing child connections.

Your log file initially shows this process, but equally subsequent analysis of the socket handover problem does rather suggest a race condition as you describe, and the unused child threads end up getting orphaned.

However, I note your configuration code snippet shows a mixture of mod_rewrite and specific mod_proxy directives, and if you read the RewriteRule documentation over the proxy [p] flag, https://httpd.apache.org/docs/current/rewrite/flags.html#flag_p, it does say:

Performance warning
Using this flag triggers the use of mod_proxy, without handling of persistent connections. This means the performance of your proxy will be better if you set it up with ProxyPass or ProxyPassMatch

This is because this flag triggers the use of the default worker, which does not handle connection pooling/reuse.

Avoid using this flag and prefer those directives, whenever you can.

This states persistent connections are not handled, so for me, despite the default documentation and other web posts showing the use of mod_rewrite to proxy, I'd try removing the mod_rewrite proxy logic, and try and stick with using mod_proxy/mod_proxy_wstunnel directives for websockets, viz:

[patchy]

The hint about the [P] flag is interesting indeed.

I tried your config snippet, but I had to modify it a bit in order to connect at all. I guess this is because of connection negotiation: When establishing the connection, the client does not know whether the server speaks websockets and starts with plain http. Using the upgrade header, server and client agree on switching protocols to websockets (at least this is the case for SignalR). So I must proxy http and websockets.

I separated proxying for negotiation and actual websocket connection:

[tangent]

Since you're using the WinNT MPM, have you tried disabling accept filters to see if that has any affect on the socket handling?

[James Blond]

From the docs of I see that you don't have to use web sockets.

Also did you read https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html

example

[patch2]