Topic: High Live Threads on Tomcat

[samv18]

Background:
We had 6 websites hosted on apache(worker MPM_ on single server Unix. Recently we migrated 6 websites to 6 new redhat VM's cluster of 2.

So, 01 and 02 Vm has 3 websites hosted, 03 and 04 has 1 website and 05 and 06 has 1 website.

All six apache servers (Apache/2.2.15, prefork MPM) talks to a single tomcat 6.0.14 (backend with 8 tomcat instances (2 daemon)) which is a unix box.


Issue

After Migration we are seeing that live threads on non dameon tomcat engines are going higher (180 Live Threads) which causing a service outage.

Current Apache/Tomcat Configuration:
# engine 0
worker.0.type=ajp13
worker.0.host=localhost
worker.0.port=9110
worker.0.lbfactor=1
worker.0.cachesize=30
worker.0.cache_timeout=600
worker.0.socket_keepalive=1
worker.0.socket_timeout=300
worker.0.connect_timeout=10000
worker.0.reply_timeout=120000
|
|# same conifg for all rest tomcat engines.


LoadModule jk_module /usr/lib64/httpd/modules/mod_jk.so
JkWorkersFile conf.d/workers.properties
JkLogFile logs/jk.log
JkLogLevel error
JkEnvVar HTTP_PORT 0
JkEnvVar HTTPS_PORT 0

JkMount /0 0
JkMount /0/* 0

#httpd config
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 10000
</IfModule>

#server.xml on tomcat engines.
<Connector port="9110"
protocol="AJP/1.3"
maxThreads="150"
xpoweredBy="true"
backlog="60" />


I have contacted red hat and they gave me below recomendation. But that recomendation is based on Tomcat supplied by redhat. we only have apache httpd from redhat but not tomcat.

redhat recomendation:

<IfModule mpm_prefork_module>
ServerLimit 400
StartServers 5
MinSpareServers 5
MaxSpareServers 20
MaxClients 400
MaxRequestsPerChild 0
</IfModule>

worker.list=loadbalancer,status
worker.template.port=8009
worker.template.type=ajp13
worker.template.ping_mode=A
worker.template.reply_timeout=120000
worker.template.socket_connect_timeout=10000
worker.template.connection_pool_timeout=600

server.xml
<Connector port="8009" protocol="AJP/1.3"
emptySessionPath="true" enableLookups="false" redirectPort="8443"
maxThreads="150" connectionTimeout="600000" ></Connector>

#connection_pool_size
Given the ThreadsPerChild of 400 and theMaxClients (400) / ThreadsPerChild (400) = Processes (1) and with 6 Apache instance(s) the connection_pool_size has been best determined to be 25
Take the maxThreads of 150 / 1 Apache server processes to get the initial connection_pool_size of150 then / this number by the number of Apache instances 6 to arrive at the correct connection_pool_size of 25.



One thing I asked redhat is about connection_pool_size. My understanding was that connection_pool_size should not be greater than 1 for apache.2.x prefork MPM. This is also mentioned in https://tomcat.apache.org/connectors-doc/reference/workers.html. They replied that the recommendation they are making is based on tomcat supplied by Redhat.

Any advise or help in reducing live threads on tomcat will help.