Apache :: httpd processes cpu spike to 100%+ every xx seconds

Krispy

My first post, unfortunately due to an issue on my dedicated (Linux Centos 6) server running some personal and one commercial website.

Server version: Apache/2.4.18 (Unix)
Server built: Mar 7 2016 20:22:35
Cpanel::Easy::Apache v3.32.10 rev9999

root@server213-171-196-40 [~]# free
total used free shared buffers cached
Mem: 16212880 15939912 272968 1939060 65928 13730952
-/+ buffers/cache: 2143032 14069848
Swap: 4194296 1477616 2716680

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2900 nobody 20 0 90636 14m 2408 R 99.1 0.1 0:07.42 httpd
72 root 20 0 0 0 0 R 17.3 0.0 8:08.35 kswapd0
436 root 20 0 0 0 0 S 2.3 0.0 1:54.42 md1_raid1
1 root 20 0 19356 668 452 S 0.0 0.0 0:00.72 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.03 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.37 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.05 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:00.04 migration/1

Above is an example from top when the httpd process is 'thrashing'. It obviously pushes the load on the server right up, and when there's 4 httpd processes doing the same, the websites become very unresponsive as you can imagine. normally the httpd processes are running at 0.1% 0.5% range, ticking over nicely.

The spikes in CPU load only started early hours of Monday morning, and I had not re-built apache / changed any config on Sunday.

i run quite a few wordpress sites - I double checked and see any updates to plugins at that time.

My linux skills a bit limited, I picked a PID for one of the maxed httpd processes, and tried to trace what it was doing using strace -p command

showed this scrolling past at a fast rate.. looks to my untrained eye like DOS attack? flooding the server?

1457649966.654499 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654515 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654531 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654545 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000006>
1457649966.654559 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654573 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654587 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>

I could obviously be wrong - in 10 or so seconds of a spike, my logfile out output was over 2 million rows! so a fantstic amount of short running 'reads'.

I don't actually understand what is happening there - my guess is something is somehow pushing a huge amount of noise/empty data at my httpd processes, and doing so every 20 seconds/every couple of minutes.

Or it's something else completely and I'm well off the mark.

I'm trying to put some of my sites under cloudflare to a) speed up the graphics intensive sites using their CDN, but also to try and add an extra layer of defense. hasn't done anything I can see yet. I can't even work out which url / website is the cause, if any.

The server does have mod_security, with the standard set of rules - I did turn off an ip check as read somewhere it could slowdown serving (no effect though). The server has plenty of memory (16 gig), 4 cpus (an 8 core intel machine) and the apache processes which spike show a tiny memory usage, it's all cpu.

PHP version 5.6.18 (or .19, need to double check).

Netstat at times of 'spike'don't show any single ip having more than 20 connections - and the server isn't overloaded (I've turned off my busy website which was getting over 15,000 visitors a day to remove traffic as the cause).

root@server213-171-196-40 [~]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

1 108.162.221.138
1 108.162.246.252
1 141.101.98.176
1 173.245.50.109
1 199.16.156.125
1 213.128.67.90
1 213.205.194.66
1 222.186.34.163
1 79.70.61.9
1 86.128.207.3
1 94.14.114.56
1 95.151.139.42
1 Address
1 servers)
2 108.162.222.88
2 173.245.56.127
4 86.4.247.37
5 136.243.48.85
6 78.147.41.131
7 90.220.251.88

The server was working fine up to Monday morning - under heavier load than it is now.

From httpd.conf:
# These can be set in WHM under 'Apache Global Configuration'
Timeout 300
TraceEnable Off
ServerSignature Off
ServerTokens ProductOnly
FileETag All
StartServers 5
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 10
</IfModule>
ServerLimit 256
MaxRequestWorkers 150
MaxConnectionsPerChild 15000
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 100

Current using the prefork mpm, but I was running using the event mpm happily, I rebuilt apache under the 'standard' pre-fork set-up as per WHM/CPANAL easyapache3 to see if re-building and changing
the config for apache would work, it didn't.

If any of you have seen similar output of a trace, or the same sort of http thrashing, I'd love to hear your advice.

cheers, Krispy