Topic: Apache hitting brick wall serving static content

[jimski]

Problem:
When testing Apache 2.4.16 on windows 8 over 10Gb/s network there is a severe limitation when serving static content. Apache can't serve more than 400 concurrent requests for static content with Keep Alive OFF.

siege -b -c700 -t10s -v http://10.0.0.31/100k.jpg
ERROR
socket: 2039797504 connection refused.: Connection refused

I also tried apache bench and weighttpd with the same results
ab -c 700 -n 10000 http://10.0.0.31/100k.jpg (100KB file)
ERROR:
apr_socket_recv: Connection refused

However when testing with apache bench on the localhost (bypassing the network) apache works fine and can serve 1000 concurrent requests for 100K static file.

Apache has ThreadsPerChild 7000 (increasing it to 14000 didn't make any difference)
MaxConnectionsPerChild 0

My load testing server is Linux on a separate server and sends requests over the network to Apache Windows server.

Conclusions:
There are no errors in the apache log and nothing in the systen logs. The tasklist doesn't show anything unusual.
netstat shows about 1,000 of open TCP connections and then Apache stops responding. However when testing with lower concurrency of 300 then the OS can open tens of thousands of TCP connections and Apache works fine.

At first I thought this is OS stack tuning problem but serving php file with the same concurrency works fine. ab -c 700 -n 10000 http://10.0.0.31/phpinfo.php

Then I tried Nginx for windows on the same machine
and Nginx served this without a problem ab -c 700 -n 10000 http://10.0.0.31/100k.jpg Nginx was able to serve much higher concurrency up to 5000 requests per second (static content) and the OS opened about 40,000 TCP connections which is about the same as the number of ephemeral ports configured on my system.

The most I can get from Apache is concurrency of 400 on 100KB static file.

So this looks to me like a bug or a limitation in the way Apache communicates with the TCP/IP stack on windows.
Other people also reported similar problems. https://www.apachelounge.com/viewtopic.php?t=6021

Can you guys confirm this on your servers.

Just make sure Keep Alive is OFF and test it over the network (not on localhost). Also be aware that some older versions of Apache Bench have a problem with concurrency in excess of 300. So siege, weighttp may be better for this test.

[James Blond]

I had an issue when I tried to set a higher value for ThreadspertChild

What worked was

[James Blond]

However, I wonder why it is so often nginx vs apache instead of nginx plus apache

Why not

nginx<--->apache+php
or
nginx<--->apache+fcgid<--->PHP
or
nginx<--->apache<--->Tomcat


For a bigger client I use

[jimski]

Hi James, thanks for the comment but you are testing on the localhost however the problem only appears over the network.

[jimski]

If you are on 1Gb/s network then try smaller test file of 10KB but with higher concurrency, for example 1000.

[James Blond]

Host win 10 with the 2.5.x-dev version of apache and the client with ab.exe Win7
I'm on the host connected with RDP to the client.

[James Blond]

I tried to increase the value, but then I get

[jimski]

I see the same problem as you on windows 10. First few thousand transactions go through and then major slow down with requests taking up to 10 seconds to be processed. Increasing threads makes no difference. Operating system has plenty of ephemeral ports available and Apache still chokes.

On window 8 I see different symptoms. Either Apache immediately chokes with high concurrency or a few thousand requests go through and then connections are completely refused.

Because this is so severe with static content it is most likely an issue related to the way Apache core communicates with the TCP stack on windows.

[jimski]

THE SOLUTION

Covener replied to my post on stackoverflow and wrote this: