| Author |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
 Posted: Sat 18 Jan '14 17:41 Post subject: 10000 concurrent connections with Apache / Windows (C10K) |
 |
|
Hi guys,
Have anybody build an Apache/Windows server that can handle 10,000 concurrent connections serving dynamic content?
Here is my setup:
Apache 2.4.7 VC10 32bit (from apachelounge)
mod_fcgid 2.3.9 (from apachelounge)
PHP 5.4.24 VC9 Non-thread-safe
Windows 2003 Server Enterprise 32bit
Intel Xeon 5600 server 8 core 2,83Ghz with 72 Gig of RAM
No other applications installed except Apache and php.
My fcgid conf:
# FCGID Configuration Directives
FcgidIOTimeout 128
FcgidBusyTimeout 7200
FcgidConnectTimeout 10
FcgidMaxProcesses 10000
FcgidMaxProcessesPerClass 10000
FcgidOutputBufferSize 65536
FcgidProcessLifeTime 0
FcgidMaxRequestsPerProcess 100000
FcgidFixPathinfo 1
IPCConnectTimeout 3
IPCCommTimeout 360
FcgidSpawnScore 1
FcgidSpawnScoreUpLimit 100
My mpm conf:
ThreadLimit 20000
ThreadsPerChild 10000
MaxConnectionsPerChild 0
Here are sucesses and problems so far  .
The server runs very fast and with no problem when serving large amounts of static content or small amounts of 1000 to 1500 requests of dynamic content with concurrency of 1000 simultaneous requests at a time and when config settings are decreased as described below. But fails with larger amount of dynamic requests.
Here is benchmark serving small static html file.
ab -n 100000 -c 1000 -s60 http://127.0.0.1:80/index.html
Finished 100000 requests
Time taken for tests: 13.047 seconds
Requests per second: 7119.02 [#/sec] (mean)
Transfer rate: 1918.80 [Kbytes/sec] received
1. When mpm ThreadLimit is set to 15001 or more I get the following notice:
Warning: ThreadLimit of 20000 exceeds compile-time limit of 15000 threads, decreasing to 15000
Looks like this is an Apache compilation limit.
Question: Is there a reason that this limit is set at 15000 during compilation?
2. When mpm ThreadsPerChild is set to 10000 I get the following error in the Apache error log:
AH00354: Child: Starting 10000 worker threads.
(OS 8)Not enough storage is available to process this command. : AH00355: Child: CreateThread failed. Unable to create all worker threads.
Created 7809 of the 10000 threads requested with the ThreadsPerChild configuration directive. Parent:Shutting down the server.
When I decrease ThreadsPerChild to 7808 the server starts OK
Even though windows do no show any errors in the error log I tried to search for error reported in Apache log: "Not enough server storage is available to process this command". A suggested resolution on some websites is to increase stack size with registry value IRPStackSize
However when I increase the IRPStackSize several times all the way up to 50 there is no change and the same error still occurs.
HKEY_LOCAL_MACHINE\SYSTEM\CURRENT_CONTROL_SET\SERVICES\ LanmanServer\Parameters => IRPStackSize = 50 (decimal)
Looks like the ThreadsPerChild can't be higher than 7808
Question: Is this error caused by Apache compilation limit or by some windows settings limit? What is a possible resolution to this problem?
3. When running apache bench using phpinfo.php as a test file, Apache fails after about 2000 requests.
ab -n 10000 -c 1000 -s60 http://127.0.0.1:80/phpinfo.php
ab.exe completes about 2000 requests before it reports: failed requests. The following error is logged in the apache error log:
[fcgid:warn] (OS 998)Invalid access to memory location. : [client 127.0.0.1:3269] mod_fcgid: can't read from pipe
[crit] Memory allocation failed, aborting process
I tested the physical memory and is fine. Memory passes a memory test and server can run any application without a problem. Second server with the same configuration also reports the same problem.
Question: What can cause this type of failure.
4. When running apache bench with concurrency -c set to 1025 or more ab.exe reports an error.
ab -n 100000 -c 2000 -s60 http://127.0.0.1:80/phpinfo.php
Error: apr_pollset_create failed: Invalis argument (22)
This looks like another compile time limitation.
When I decrease -c to 1024 Apache Bench starts fine.
Question: Is the apache bench compiled together with the Apachelounge distribution or is it compiled by another party. Why the concurrency limit is set to only 1024?
None of the above problems log anything in the windows error log.
Thanks in advance for any input.
Last edited by jimski on Mon 20 Jan '14 20:02; edited 4 times in total |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
| Posted: Sat 18 Jan '14 20:11 Post subject: Update |
|
|
Update
When examining the windows services after Apache crashes, with C:\tasklist /svc , there are 89 php-cgi.exe processes running and appears that mod_fcgid can't spawn any more and dies even though there is plenty of RAM and CPU power left.
Memory usage in taskmanager is only 1.5 GB.
Also, the FcgidBusyTimeout 7200 is set so high because I want to allow large uploads which will take long time. |
|
| Back to top |
|
CamaroSS
Joined: 24 Jan 2013
Posts: 78
Location: RF, Tver
|
| Posted: Mon 20 Jan '14 9:12 Post subject: |
|
|
Did you consider using IIS, if you're about to put that much load on Windows Server?
Maybe a 64-bit version of the system and Apache will be sufficient. |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
| Posted: Mon 20 Jan '14 18:50 Post subject: |
|
|
Thanks for suggestion CamaroSS, I would prefer to go with Apache because I know it much better but if I don't find a solution then I will have to go with IIS.
Today I've tried the same configuration as described above but using x64 system:
Windows Server 2012 (x64)
Apache 2.4.7 VC11 64bit
mod_fcgid 2.3.9
PHP 5.4.24 VC11 Non-thread-safe
And even though I can establish a few more concurrent connections the performance is 18% slower on x64 than on windows 2003 32 bit.
However, on a 64bit system two problems went away:
1.mpm ThreadsPerChild can now be set to 15000
2. Apache Bench now accepts -c 10000 concurrency
But apache still chokes and dies under heavy load.
Funny thing is that limux/apache have solved c10K problem more than 10 years ago yet apache on windows still can't handle it.
I will do more testing today with different settings of TCP/IP stack and yet another configuration but it looks like fcgid on windows simply can't spawn enough processes to server 10K concurrent requests. |
|
| Back to top |
|
C0nw0nk
Joined: 07 Oct 2013
Posts: 241
Location: United Kingdom, London
|
| Posted: Mon 20 Jan '14 20:41 Post subject: |
|
|
Use my config and also with dynamic content use a extension called wincache enable usercache and opcache.
| Code: | LoadModule fcgid_module modules/mod_fcgid.so
<IfModule fcgid_module>
FcgidInitialEnv PHPRC "c:/server/xampp/php-5.5.7-nts"
FcgidInitialEnv PATH "c:/server/xampp/php-5.5.7-nts;C:/WINDOWS/system32;C:/WINDOWS;C:/WINDOWS/System32/Wbem;"
FcgidInitialEnv SystemRoot "C:/Windows"
FcgidInitialEnv SystemDrive "C:"
FcgidInitialEnv TEMP "C:/WINDOWS/Temp"
FcgidInitialEnv TMP "C:/WINDOWS/Temp"
FcgidInitialEnv windir "C:/WINDOWS"
FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 10000
FcgidInitialEnv PHP_FCGI_CHILDREN 0
FcgidIdleTimeout 3600
FcgidIOTimeout 3699
FcgidMinProcessesPerClass 0
FcgidMaxProcessesPerClass 9999
FcgidMaxRequestsPerProcess 10000
FcgidProcessLifeTime 0
FcgidMaxRequestInMem 1
FcgidWin32PreventOrphans On
FcgidMaxRequestLen 1073741824
<Files ~ "\.php$>"
AddHandler fcgid-script .php
FcgidWrapper "c:/server/xampp/php-5.5.7-nts/php-cgi.exe" .php
</Files>
</IfModule> |
The website on that machine loads very fast Joomla 2.5
And i would put a link but im sure everyone would reject it because it is a pornsite 
Page speed if anyone is intreasted with my config and wincache enabled. On php 64Bit.
http://tools.pingdom.com/fpt/#!/c4jaB1
For anyone intreasted in site stats cloudflare reports.
47,963,473 requests saved by CloudFlare
66,467,273 total requests
So about 19-20million in traffic hits the server a month
Last edited by C0nw0nk on Mon 20 Jan '14 20:57; edited 2 times in total |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
| Posted: Mon 20 Jan '14 20:50 Post subject: |
|
|
Thanks for suggestion C0nw0nk. I noticed that you have #FcgidMaxProcesses 300
Maybe I'm wrong but my understanding is that each php-cgi.php process can handle only one connection at a time.
Last edited by jimski on Mon 20 Jan '14 20:55; edited 3 times in total |
|
| Back to top |
|
C0nw0nk
Joined: 07 Oct 2013
Posts: 241
Location: United Kingdom, London
|
| Posted: Mon 20 Jan '14 20:52 Post subject: |
|
|
| jimski wrote: | | Maybe I'm wrong but my understanding is that each php-cgi.php process can handle only one connection at a time. |
Thats why its hashed out and in my config it uses FcgidMaxProcessesPerClass 9999
No php-cgi process handle 10,000 in my config.
Look at.
FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 10000
FcgidMaxRequestsPerProcess 10000 |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
| Posted: Mon 20 Jan '14 20:54 Post subject: |
|
|
Yes but when is hashed out it defaults to 100 if I remeber well.
Last edited by jimski on Mon 20 Jan '14 21:00; edited 3 times in total |
|
| Back to top |
|
C0nw0nk
Joined: 07 Oct 2013
Posts: 241
Location: United Kingdom, London
|
| Posted: Mon 20 Jan '14 20:56 Post subject: |
|
|
| jimski wrote: | | Yes but when is hashed out it defaults to 100 if I remeber well. |
Do not read the hashed out lines in my config. I am removing them so you dont get confussed.
Look at everything that is not hashed out.
Updated hashed out lines removed from my config. |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
| Posted: Mon 20 Jan '14 20:59 Post subject: |
|
|
Sorry, if #FcgidMaxProcesses is commented out then it defaults to 1000 (not 100). So your setting of FcgidMaxProcessesPerClass 9999 also is limited to 1000.
http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxprocesses
Each of the directives has a default value. And even though some are OS specific, the FcgidMaxProcesses always defaults to 1000 if not specified.
Last edited by jimski on Mon 20 Jan '14 21:06; edited 1 time in total |
|
| Back to top |
|
C0nw0nk
Joined: 07 Oct 2013
Posts: 241
Location: United Kingdom, London
|
| Posted: Mon 20 Jan '14 21:04 Post subject: |
|
|
Basically yes.
http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxprocesses
I set FcgidMaxProcessesPerClass 9999
To a high value because i noticed my server would lock up when my php-cgi process reached 100.
so setting it to a high value prevents the server from locking up.
My site has very high traffic with that config and with wincache my php process i never get more than 20-30 php process running at a time. |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
| Posted: Mon 20 Jan '14 21:11 Post subject: |
|
|
How many concurrent php-cgi.exe processes can you run
| C0nw0nk wrote: | Basically yes.
To a high value because i noticed my server would lock up when my php-cgi process reached 100.
|
Yes, I have the same experience but because php-cgi.exe can't be spawned beyond 100 it effectively limits the number of simultaneous connections to 100.
I'm sure your server can many thousands requests per second but they are not concurrent. Also in my case when 100 people try to upload files all the 100 php-cgi processes are busy and no additional connections can be established until one of those processes finished serving a request.
Last edited by jimski on Mon 20 Jan '14 22:32; edited 2 times in total |
|
| Back to top |
|
C0nw0nk
Joined: 07 Oct 2013
Posts: 241
Location: United Kingdom, London
|
| Posted: Mon 20 Jan '14 21:14 Post subject: |
|
|
| jimski wrote: | | FcgidMaxProcessesPerClass 9999 also is limited to 1000. |
Where did you find out its limited to 1000 ? |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
|
| Back to top |
|
C0nw0nk
Joined: 07 Oct 2013
Posts: 241
Location: United Kingdom, London
|
| Posted: Mon 20 Jan '14 21:19 Post subject: |
|
|
But thats maxprocesses not maxproccessperclass.
Also im pretty sure thats why i use
FcgidInitialEnv PHP_FCGI_CHILDREN 0
I remember i had a issue a long time ago and i added that in and it fixed it. It has been so long since i needed to fix any issues with my config.
We may need Jan-E he understands php-cgi very well i recon since he compiles the builds for us and he should be able to tell us how many concurrent connections php-cgi handles.
Last edited by C0nw0nk on Mon 20 Jan '14 21:23; edited 1 time in total |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
| Posted: Mon 20 Jan '14 21:22 Post subject: |
|
|
| Quote: | Also im pretty sure thats why i use
FcgidInitialEnv PHP_FCGI_CHILDREN 0 |
Thanks C0nw0nk I will try that.
Are you running your apache as service?
if so could you try this command: C:\tasklist /svc
It will list all the php-cgi.exe processes that are spawned.
I'm curious how many can run on your server.
Last edited by jimski on Mon 20 Jan '14 21:25; edited 1 time in total |
|
| Back to top |
|
C0nw0nk
Joined: 07 Oct 2013
Posts: 241
Location: United Kingdom, London
|
| Posted: Mon 20 Jan '14 21:24 Post subject: |
|
|
Yes my apache is a service but all my apache handles is php and html requests all static content gets handled by nginx but that does not matter since it is the same as yours for dynamic content.
It says on the mod_fcgi page.
| Quote: | Special PHP considerations
By default, PHP FastCGI processes exit after handling 500 requests, and they may exit after this module has already connected to the application and sent the next request. When that occurs, an error will be logged and 500 Internal Server Error will be returned to the client. This PHP behavior can be disabled by setting PHP_FCGI_MAX_REQUESTS to 0, but that can be a problem if the PHP application leaks resources. Alternatively, PHP_FCGI_MAX_REQUESTS can be set to a much higher value than the default to reduce the frequency of this problem. FcgidMaxRequestsPerProcess can be set to a value less than or equal to PHP_FCGI_MAX_REQUESTS to resolve the problem.
PHP child process management (PHP_FCGI_CHILDREN) should always be disabled with mod_fcgid, which will only route one request at a time to application processes it has spawned; thus, any child processes created by PHP will not be used effectively. (Additionally, the PHP child processes may not be terminated properly.) By default, and with the environment variable setting PHP_FCGI_CHILDREN=0, PHP child process management is disabled.
The popular APC opcode cache for PHP cannot share a cache between PHP FastCGI processes unless PHP manages the child processes. Thus, the effectiveness of the cache is limited with mod_fcgid; concurrent PHP requests will use different opcode caches.
|
Last edited by C0nw0nk on Mon 20 Jan '14 21:29; edited 1 time in total |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
| Posted: Mon 20 Jan '14 21:29 Post subject: |
|
|
| Quote: | | But thats maxprocesses not maxproccessperclass. |
FcgidMaxProcesses Directive
This directive sets the maximum number of FastCGI application processes which can be active at one time.
This means that no more processes can be active at one time no matter what other directives are set to. |
|
| Back to top |
|
C0nw0nk
Joined: 07 Oct 2013
Posts: 241
Location: United Kingdom, London
|
| Posted: Mon 20 Jan '14 21:31 Post subject: |
|
|
Surely if you have more than 1000 php-cgi processes running thats a bad thing.
I think your cpu would be at 100%
Maybe i got the wrong end of the stick ? I am not sure before i attempt to give any more answers i will wait to see what someone else says i always assumed all connections are concurrent
Last edited by C0nw0nk on Mon 20 Jan '14 21:41; edited 1 time in total |
|
| Back to top |
|
jimski
Joined: 18 Jan 2014
Posts: 196
Location: USSA
|
| Posted: Mon 20 Jan '14 21:41 Post subject: |
|
|
Thanks C0nw0nk
If I could ask you a favor. If you have a minute can you run this command when you have a heavy php load:
C:\tasklist /svc
I'm curious how many php-cgi processes run on your server under heavy php load. On mine 89 seems to be the maximum. |
|
| Back to top |
|
