Author |
|
Megunticook
Joined: 03 Feb 2018 Posts: 12 Location: Maine
|
Posted: Thu 16 Aug '18 18:09 Post subject: need a little advice on tuning Apache 2.4 |
|
|
I manage a WordPress site on an AWS Linux instance running Apache 2.4. Just had an issue today where the server became unresponsive, so I rebooted and it came back. Then I looked at the log and it appears Apache ran out of memory, although the AWS metrics indicate that the server never used more than 33% of available memory (4GB).
I've pasted some of the httpd_error_log file in below from the point where the "out of memory" warnings began. But from my research online I'm thinking I need to specify ServerLimit and MaxRequestWorkers values in my httpd.conf file. Right now there is nothing specified at all there, so I'm thinking Apache must be using default values.
Is this a good place to start? Based on a blog post I read ( https://aws.amazon.com/premiumsupport/knowledge-center/ec2-apache-memory-tuning/ ), I'm thinking I would add this to my httpd.conf file. Please tell me if I'm on the right track or should look at other options:
Code: | <IfModule mpm_prefork_module>
ServerLimit 40
MaxRequestWorkers 40
</IfModule> |
Thanks for the help.
Code: | 14:06:35
mmap() failed: [12] Cannot allocate memory
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:34.992686 2018] [php7:error] [pid 17353] [client 66.31.16.41:54460] PHP Fatal error: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in /var/www/html/wp-content/plugins/woocommerce-abandoned-cart/woocommerce-ac.php on line 2927
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.001614 2018] [php7:error] [pid 17182] [client 66.31.16.41:54374] PHP Fatal error: Out of memory (allocated 65011712) (tried to allocate 32768 bytes) in /var/www/html/wp-content/plugins/paypal-for-woocommerce/angelleye-includes/express-checkout/class-wc-gateway-paypal-express-helper-angelleye.php on line 43
14:06:35
mmap() failed: [12] Cannot allocate memory
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.004601 2018] [php7:error] [pid 17501] [client 66.31.16.41:54704] PHP Fatal error: Out of memory (allocated 10485760) (tried to allocate 32768 bytes) in /var/www/html/wp-includes/comment.php on line 1873
14:06:35
mmap() failed: [12] Cannot allocate memory
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.007878 2018] [php7:error] [pid 17361] [client 66.31.16.41:54464] PHP Fatal error: Out of memory (allocated 37748736) (tried to allocate 36864 bytes) in /var/www/html/wp-content/plugins/popup-maker/includes/pum-sdk/freemius/includes/class-freemius.php on line 18549
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.011104 2018] [php7:error] [pid 17914] [client 66.31.16.41:54674] PHP Fatal error: Out of memory (allocated 12582912) (tried to allocate 32768 bytes) in /var/www/html/wp-includes/general-template.php on line 2551
14:06:35
mmap() failed: [12] Cannot allocate memory
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.018423 2018] [php7:error] [pid 17740] [client 66.31.16.41:54722] PHP Fatal error: Out of memory (allocated 8388608) (tried to allocate 36864 bytes) in /var/www/html/wp-includes/formatting.php on line 5322
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.032744 2018] [php7:error] [pid 17541] [client 66.31.16.41:54716] PHP Fatal error: Out of memory (allocated 8388608) (tried to allocate 32768 bytes) in /var/www/html/wp-includes/general-template.php on line 2551
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.038810 2018] [php7:error] [pid 17794] [client 66.31.16.41:54724] PHP Fatal error: Out of memory (allocated 10485760) (tried to allocate 32768 bytes) in /var/www/html/wp-includes/general-template.php on line 2551
14:06:35
[Thu Aug 16 14:06:34.946451 2018] [php7:error] [pid 17260] [client 66.31.16.41:54414] PHP Fatal error: Out of memory (allocated 41943040) (tried to allocate 32768 bytes) in /var/www/html/wp-content/plugins/woocommerce/includes/class-wc-datetime.php on line 1
14:06:35
mmap() failed: [12] Cannot allocate memory
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.068733 2018] [php7:error] [pid 17182] [client 66.31.16.41:54374] PHP Fatal error: Out of memory (allocated 65011712) (tried to allocate 32768 bytes) in /var/www/html/wp-content/plugins/woocommerce/includes/log-handlers/class-wc-log-handler-file.php on line 1
14:06:35
mmap() failed: [12] Cannot allocate memory
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.112216 2018] [php7:error] [pid 17206] [client 66.31.16.41:54402] PHP Fatal error: Out of memory (allocated 52428800) (tried to allocate 32768 bytes) in /var/www/html/wp-content/plugins/wordfence/lib/wordfenceClass.php on line 251
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.112535 2018] [php7:error] [pid 17206] [client 66.31.16.41:54402] PHP Fatal error: Out of memory (allocated 52428800) (tried to allocate 32768 bytes) in /var/www/html/wp-content/plugins/woocommerce/includes/log-handlers/class-wc-log-handler-file.php on line 1
14:06:35
mmap() failed: [12] Cannot allocate memory
14:06:35
mmap() failed: [12] Cannot allocate memory
14:06:35
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:35.148430 2018] [php7:error] [pid 17734] [client 66.31.16.41:54788] PHP Fatal error: Out of memory (allocated 2097152) (tried to allocate 1032192 bytes) in /var/www/html/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/utils.php on line 437
14:12:16
mmap() failed: [12] Cannot allocate memory
14:12:16
mmap() failed: [12] Cannot allocate memory
14:12:16
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:36.714877 2018] [php7:error] [pid 17535] [client 66.31.16.41:54780] PHP Fatal error: Out of memory (allocated 8388608) (tried to allocate 32768 bytes) in /var/www/html/wp-includes/general-template.php on line 2551
14:12:16
[Thu Aug 16 14:06:36.714932 2018] [php7:error] [pid 17460] [client 66.31.16.41:54472] PHP Fatal error: Out of memory (allocated 41943040) (tried to allocate 32768 bytes) in /var/www/html/wp-content/plugins/woocommerce/includes/class-wc-datetime.php on line 1
14:12:16
mmap() failed: [12] Cannot allocate memory
14:12:16
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:36.719059 2018] [php7:error] [pid 17360] [client 66.31.16.41:54446] PHP Fatal error: Out of memory (allocated 41943040) (tried to allocate 32768 bytes) in /var/www/html/wp-content/plugins/woocommerce/includes/class-wc-datetime.php on line 1
14:12:16
mmap() failed: [12] Cannot allocate memory
14:12:16
mmap() failed: [12] Cannot allocate memory[Thu Aug 16 14:06:36.720712 2018] [php7:error] [pid 17736] [client 66.31.16.41:54714] PHP Fatal error: Out of memory (allocated 14680064) (tried to allocate 131072 bytes) in /var/www/html/wp-includes/script-loader.php on line 267
....(lots of repeating out of memory events recorded)
14:12:16
mmap() failed: [12] Cannot allocate memory
14:12:16
mmap() failed: [12] Cannot allocate memory
14:12:16
[Thu Aug 16 14:11:41.720182 2018] [mpm_prefork:notice] [pid 6729] AH00169: caught SIGTERM, shutting down
14:12:19
[Thu Aug 16 14:12:19.850580 2018] [suexec:notice] [pid 2973] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
14:12:19
[Thu Aug 16 14:12:19.874096 2018] [http2:warn] [pid 2974] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
14:12:19
[Thu Aug 16 14:12:19.874579 2018] [lbmethod_heartbeat:notice] [pid 2974] AH02282: No slotmem from mod_heartmonitor
14:12:20
[Thu Aug 16 14:12:19.981498 2018] [mpm_prefork:notice] [pid 2974] AH00163: Apache/2.4.33 (Amazon) OpenSSL/1.0.2k-fips PHP/7.1.17 configured -- resuming normal operations
14:12:20
[Thu Aug 16 14:12:19.981522 2018] [core:notice] [pid 2974] AH00094: Command line: '/usr/sbin/httpd'
14:13:20
[Thu Aug 16 14:13:20.127359 2018] [mpm_prefork:notice] [pid 2974] AH00169: caught SIGTERM, shutting down
14:13:20
[Thu Aug 16 14:13:20.210406 2018] [suexec:notice] [pid 3246] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
14:13:20
[Thu Aug 16 14:13:20.232589 2018] [http2:warn] [pid 3247] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
14:13:20
[Thu Aug 16 14:13:20.233099 2018] [lbmethod_heartbeat:notice] [pid 3247] AH02282: No slotmem from mod_heartmonitor
14:13:20
[Thu Aug 16 14:13:20.254773 2018] [mpm_prefork:notice] [pid 3247] AH00163: Apache/2.4.33 (Amazon) OpenSSL/1.0.2k-fips PHP/7.1.17 configured -- resuming normal operations
14:13:20
[Thu Aug 16 14:13:20.254791 2018] [core:notice] [pid 3247] AH00094: Command line: '/usr/sbin/httpd' |
|
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Fri 17 Aug '18 9:20 Post subject: |
|
|
The first error PHP Fatal error: Out of memory is PHP related, not apache. See memory_limit in your php.ini
That HTTP/2 is not working is because of prefork mpm. You would have to use worker or event mpm to use that. However than you can use PHP as module but must run it over mod_fcgid or FPM. |
|
Back to top |
|
Megunticook
Joined: 03 Feb 2018 Posts: 12 Location: Maine
|
Posted: Fri 17 Aug '18 14:49 Post subject: |
|
|
Thanks for your reply.
PHP memory limit is set at 128M in php.ini. I changed it to 256M.
Couldn't those PHP errors be a result of Apache hogging all the available memory? Or is the log telling me that the source of the problem is PHP?
I think if WordPress was running out of memory I would get an error message like this:
Code: | Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 2348617 bytes) in /home4/xxx/public_html/wp-includes/plugin.php on line xxx |
Last night I set up a new server using the Event MPM instead of Prefork. From what I understand this uses less overhead, plus supports http2. |
|
Back to top |
|
Megunticook
Joined: 03 Feb 2018 Posts: 12 Location: Maine
|
Posted: Fri 17 Aug '18 19:20 Post subject: |
|
|
I'm monitoring the new server, which is now using the event MPM instead of prefork.
After running the top command, it's telling me that it's using 3.2G our of 4G total memory, which seems awfully high given that the site is only seeing light to moderate use. It says there are 133 total tasks, 1 running, 90 sleeping.
Do I need to make some adjustments here? Seems odd that it would be consuming 80% of available memory when there's such a light load on the server.
What are all those sleeping procecess? Are they consuming a bunch of memory? Can I reduce them? Something doesn't seem right here...
I just added this to my httpd.conf and restarted Apache:
Code: | <IfModule mpm_event_module>
MaxRequestWorkers 50
ServerLimit 50
</IfModule> |
I got those numbers following the forumula laid out in that AWS blog post I referred to above.
After restarting Apache, I ran top again and it looks the same. Running well over 80% memory in use and 92 sleeping tasks.
Any advice or help would be much appreciated--thanks. |
|
Back to top |
|
Megunticook
Joined: 03 Feb 2018 Posts: 12 Location: Maine
|
Posted: Sat 18 Aug '18 3:03 Post subject: |
|
|
Update: had another brief server downtime, but it came back up by itself after 3 minutes. I upgraded my AWS instance to 8GB memory and revised my httpd.conf settings to set MaxRequestWorkers and ServerLimit to 75. Everything seems to be fine now, 6.7GB memory free, 111 total tasks, 1 running, 69 sleeping.
I think the MaxRequestWorkers and ServerLimit were too low. Here's the last few log entries from httpd_error:
Code: | [Fri Aug 17 21:15:07.118715 2018] [mpm_event:notice] [pid 8639:tid 140447802792128] AH00492: caught SIGWINCH, shutting down gracefully
[Fri Aug 17 21:16:41.151683 2018] [mpm_event:warn] [pid 3174:tid 140605688780992] AH00514: MaxRequestWorkers of 40 is not an integer multiple of ThreadsPerChild of 25, decreasing to nearest multiple 25
[Fri Aug 17 21:16:41.153332 2018] [lbmethod_heartbeat:notice] [pid 3174:tid 140605688780992] AH02282: No slotmem from mod_heartmonitor
[Fri Aug 17 21:16:41.158418 2018] [mpm_event:notice] [pid 3174:tid 140605688780992] AH00489: Apache/2.4.33 () OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Fri Aug 17 21:16:41.158442 2018] [core:notice] [pid 3174:tid 140605688780992] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Fri Aug 17 21:16:42.159535 2018] [mpm_event:error] [pid 3174:tid 140605688780992] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Fri Aug 17 21:20:34.925530 2018] [mpm_event:notice] [pid 3174:tid 140605688780992] AH00492: caught SIGWINCH, shutting down gracefully
[Fri Aug 17 21:20:36.010339 2018] [lbmethod_heartbeat:notice] [pid 3750:tid 140148336576704] AH02282: No slotmem from mod_heartmonitor
[Fri Aug 17 21:20:36.013267 2018] [mpm_event:notice] [pid 3750:tid 140148336576704] AH00489: Apache/2.4.33 () OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Fri Aug 17 21:20:36.013284 2018] [core:notice] [pid 3750:tid 140148336576704] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Fri Aug 17 21:21:30.073716 2018] [mpm_event:error] [pid 3750:tid 140148336576704] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting |
|
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Sun 19 Aug '18 15:34 Post subject: |
|
|
That are very high settings. Unlike prefork event and worker mpm start only a few processes and those start threads. The threads then handle the requests.
Here are the stettings I work with.
Code: |
# event MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxConnectionsPerChild: maximum number of connections a server process serves
# before terminating
<IfModule mpm_event_module>
StartServers 3
MaxClients 320
MinSpareThreads 20
MaxSpareThreads 40
ThreadsPerChild 20
MaxConnectionsPerChild 0
</IfModule>
|
|
|
Back to top |
|
Megunticook
Joined: 03 Feb 2018 Posts: 12 Location: Maine
|
Posted: Sun 19 Aug '18 17:54 Post subject: |
|
|
Thanks for the clarification. I'll try lowering those--was just following the formula from that AWS blogpost. Yesterday I removed the directive on MaxWorkerRequests and MaxServers from my httpd.conf just so Apache would use the default values. I'll try using lower values next.
Today I saw another head-scratcher in my log file. Did some Googling but if you have any insights to what is happening here I'd really appreciate it. Here's the most recent entries in this morning's log:
Code: | [Sun Aug 19 14:04:24.353987 2018] [mpm_event:notice] [pid 4054:tid 140033308842176] AH00492: caught SIGWINCH, shutting down gracefully
[Sun Aug 19 14:04:25.437266 2018] [lbmethod_heartbeat:notice] [pid 10910:tid 139984931231936] AH02282: No slotmem from mod_heartmonitor
[Sun Aug 19 14:04:25.440798 2018] [mpm_event:notice] [pid 10910:tid 139984931231936] AH00489: Apache/2.4.33 () OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Sun Aug 19 14:04:25.440819 2018] [core:notice] [pid 10910:tid 139984931231936] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' |
|
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Mon 20 Aug '18 14:21 Post subject: |
|
|
Megunticook wrote: |
[Sun Aug 19 14:04:25.437266 2018] [lbmethod_heartbeat:notice] [pid 10910:tid 139984931231936] AH02282: No slotmem from mod_heartmonitor
|
sually, you can suppress that message by commenting out the heartbeat load-balancer method module from httpd.conf (unless, of course, you are actually using it). It has been suggested[1] that this be disabled in the default configuration, but evidently it is not yet on your distro.
Code: |
#LoadModule lbmethod_heartbeat_module modules/mod_lbmethod_heartbeat.so
|
[1] https://mail-archives.apache.org/mod_mbox/httpd-dev/201211.mbox/%3C1162516276.8579887.1353080461144.JavaMail.root@consol.de%3E |
|
Back to top |
|
Megunticook
Joined: 03 Feb 2018 Posts: 12 Location: Maine
|
Posted: Mon 20 Aug '18 15:51 Post subject: |
|
|
That's odd...I just checked my configuration saw heartbeat was already commented out by default:
Code: | #LoadModule heartbeat_module modules/mod_heartbeat.so
#LoadModule heartmonitor_module modules/mod_heartmonitor.so |
Strange. I looked through all the configuration files in /etc/httpd/conf.modules.d/ to make sure it wasn't enabled somewhere else.
What is that "caught SIGWICH, shutting down gracefully" all about? That was the first log entry after about a 2-hour gap of no log activity, so I don't understand what triggered it.
Thanks again for your help. |
|
Back to top |
|