Author |
|
pidfile
Joined: 25 Dec 2013 Posts: 4
|
Posted: Wed 25 Dec '13 16:31 Post subject: Apache 2.4.3 mod_proxy not stable in Production |
|
|
Merry Xmas
Have run through a bunch of threads on mod_proxy race conditions, possible causes, and potential solutions suggested by Apache core devs.
What is the status? Is there a fix for this, maybe in 2.4.7?
On 2.4.3 in production we literally have 10K errors in the past 2 months along the lines of:
Code: | AH00898: Error reading from remote server returned by... |
Application server (JVM webapp) issues no errors at the time of mod_proxy error occurring. It seems then that mod_proxy is sporadically broken, which is horrible for end users accessing an otherwise rock solid production application.
Would love to stay on Apache but we're going to have to switch to a different solution if no fix has been made since 2.4.3
Thanks |
|
Back to top |
|
jraute
Joined: 13 Sep 2013 Posts: 188 Location: Rheinland, Germany
|
Posted: Wed 25 Dec '13 16:59 Post subject: |
|
|
What is the complete error line and what configuration you are using? |
|
Back to top |
|
Steffen Moderator
Joined: 15 Oct 2005 Posts: 3092 Location: Hilversum, NL, EU
|
Posted: Wed 25 Dec '13 17:00 Post subject: |
|
|
When does it happen ?
Assuming that you use ProxyPass directive.
What about the keepalive timeout, front and back the same ? Good habbit is to make different, 1-2 seconds different (not recall front or back)
Also the default for Timeout has been reduced in 2.4 from 300 to 60. The developers said that if
there are complaints from users, the new value may be too low and we should maybe reconsider the new value.
So check and play with the settings:
TimeOut 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeOut 15
Running Windows ?
You can try:
AcceptFilter http none
AcceptFilter https none
EnableSendfile off
EnableMMAP off |
|
Back to top |
|
pidfile
Joined: 25 Dec 2013 Posts: 4
|
Posted: Wed 25 Dec '13 21:37 Post subject: |
|
|
Thanks guys.
Linux box (CentOS 6.3)
*******************
# conf.d/foo.conf
<Location />
ProxyPass http://172.16.xx.xx:9001/ max=60 retry=10
ProxyPassReverse http://172.16.xx.xx:9001/
SetOutputFilter DEFLATE
SetEnv proxy-sendchunks 1
</Location>
# httpd.conf
KeepAlive On
KeepAliveTimeout 2500ms
EnableSendfile On
********************
Timeout shouldn't come into play, page requests typically served in < 400ms, not the 60s default.
Linux default KeepAlive timeout is 2 hours.
Again, apache dev lists do indicate that there is an as yet unresolved (as of 2.4.3 at any rate) issue with mod_proxy and keepalive race conditions.
Not sure what the underlying cause is, but getting on average at least 150 proxy errors per day _in production_ not good. |
|
Back to top |
|
jraute
Joined: 13 Sep 2013 Posts: 188 Location: Rheinland, Germany
|
Posted: Wed 25 Dec '13 23:05 Post subject: |
|
|
Although you didn't post the complete error log line i am quite sure that the problem still exists.
Reading the bug blog there seems to be only one solution which is based on deactivating keep alives and i remember such errors even with 2.4.7 in our logs too although the clients do not have any problems.
Have you tried the settings Steffen mentioned? |
|
Back to top |
|
pidfile
Joined: 25 Dec 2013 Posts: 4
|
Posted: Thu 26 Dec '13 0:43 Post subject: |
|
|
Thanks, settings are the same as Stefan's, just shorter TTL.
Could turn off KeepAlive, but prefer performance gain, not loss.
Will dig around some more, perhaps there's something specific to my setup if clients are affected by the errors while in your case apparently they are not. |
|
Back to top |
|
jraute
Joined: 13 Sep 2013 Posts: 188 Location: Rheinland, Germany
|
Posted: Thu 26 Dec '13 11:26 Post subject: |
|
|
There's one advice in the blog to use
Code: | SetEnv proxy-initial-not-pooled |
Have you tried that?
We have this in use for many configurations with positive effects.
Btw proxy-sendchunks is only working reliably with backend servers supporting HTTP 1.1 |
|
Back to top |
|
pidfile
Joined: 25 Dec 2013 Posts: 4
|
Posted: Thu 26 Dec '13 12:44 Post subject: |
|
|
See http://www.gossamer-threads.com/lists/apache/dev/428261 for all the gory details. From the thread:
Quote: | As Covener pointed out to me in
IRC, using mod_proxy_http's env variable "proxy-initial-not-pooled" does
offer a solution to the problem albeit at the cost of performance. |
Thread has been running for over 2 years, last updated a couple of weeks ago with a bunch of dev activity; there is as yet no solution that does not impact performance.
Our JVM app speaks HTTP 1.1, no problem there. |
|
Back to top |
|
jraute
Joined: 13 Sep 2013 Posts: 188 Location: Rheinland, Germany
|
Posted: Fri 27 Dec '13 12:44 Post subject: |
|
|
Ok, no solution so far.
(Honestly spoken in our setting the "performance problem" is minimal - up to 30000 page requests per day and the performance and especially the response time is ok. But you are right, it depends on the things you want to do.)
Greets |
|
Back to top |
|