Topic: Apache httpd 2.4.4 many connections in Reading Request state

[forke]

Hi,

i have weird problem with newly configured server.
I have running Apache 2.2.16(prefork) with php(module) 5.2.14 running on CentOS5 for two years on my old server and last week i decided to
move to new server (CentOS6) and upgrade Apache to 2.4.4(prefork) and php to 5.4.13.
On old server everything is working fine but when i route this traffic (A DNS record of my domain) to new server then strange things are happening.

For the first 30 seconds after httpd start everything is normal, and then after 30 seconds about 25% of active connections are in "R" (Reading Request) state. These connections are freezed in R state for
10-25secs and then dissapear.
They are not logged in error_log even in trace6 level.

server-status for such connection looks like:
PID Acc M CPU SS Req Conn Child Slot Client VHost
22182 0/90/90 R 0.73 3 5 0.0 0.03 0.03 81.96.86.224

netstat for this IP shows:
tcp 0 0 ::ffff:95.211.131.XXX:80 ::ffff:81.96.86.224:52168 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80 ::ffff:81.96.86.224:52202 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80 ::ffff:81.96.86.224:52254 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80 ::ffff:81.96.86.224:52201 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80 ::ffff:81.96.86.224:52255 ESTABLISHED 22182/httpd
tcp 0 0 ::ffff:95.211.131.XXX:80 ::ffff:81.96.86.224:52169 TIME_WAIT -
tcp 0 0 ::ffff:95.211.131.XXX:80 ::ffff:81.96.86.224:52203 TIME_WAIT -

Server load is about 0.5 and total CPU usage ~5%. Server is 16core, 32gb ram, 4xSAS hdd so it is not overloaded.
Network is 100mbps and usage of about 4mbps (only html content). I have tested connection to few IPs which freeze in R state with mtr, but no single packet loss were found.

Server is handling about 100requests/sec in peek hours and 50req/sec during night.

Regardless of req/sec problem always occurs after about 30secs after httpd start.
apachectl restart does not change anything. Only manually killing all httpd processes and starting them from scratch can give 30seconds free of R state connections.

I was also trying to recompile httpd with event mpm - effect is the same. Even when i remove all htdocs content and every request is 404, Reading Request connections appears after 30secs. Was trying httpd
2.2.24 and the effect was the same.

I was even trying different kernels (Centos6 2.6.32-358.2.1.el6 and newest one 3.8.4) but with no luck.

These Reading Request connections are comming from different networks all over the world.

I was also trying to do some stress tests with ab -n 50000 -c 20 from different servers but each time these tests completed without any freeze.

Some details:
[root@srv logs]# httpd -V
Server version: Apache/2.4.4 (Unix)
Server built: Mar 24 2013 01:43:25
Server's Module Magic Number: 20120211:11
Server loaded: APR 1.4.6, APR-UTIL 1.5.1
Compiled using: APR 1.4.6, APR-UTIL 1.5.1
Architecture: 64-bit
Server MPM: prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/usr/local/httpd"
-D SUEXEC_BIN="/usr/local/httpd/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
[root@srv logs]# httpd -l
Compiled in modules:
core.c
mod_so.c
http_core.c
prefork.c

Loaded modules:
LoadModule php5_module modules/libphp5.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule mime_module modules/mod_mime.so
LoadModule dir_module modules/mod_dir.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule status_module modules/mod_status.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule auth_basic_module modules/mod_auth_basic.so

Prefork config:
StartServers 256
MinSpareServers 256
MaxSpareServers 512
MaxRequestWorkers 1024
MaxConnectionsPerChild 0



Currently i am out of ideas what else could i check to trace this problem.

Regards,
Marcin