logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: Apache 2.4.3 mod_proxy not stable in Production
Author
pidfile



Joined: 25 Dec 2013
Posts: 4

PostPosted: Wed 25 Dec '13 16:31    Post subject: Apache 2.4.3 mod_proxy not stable in Production Reply with quote

Merry Xmas Wink

Have run through a bunch of threads on mod_proxy race conditions, possible causes, and potential solutions suggested by Apache core devs.

What is the status? Is there a fix for this, maybe in 2.4.7?

On 2.4.3 in production we literally have 10K errors in the past 2 months along the lines of:
Code:
AH00898: Error reading from remote server returned by...


Application server (JVM webapp) issues no errors at the time of mod_proxy error occurring. It seems then that mod_proxy is sporadically broken, which is horrible for end users accessing an otherwise rock solid production application.

Would love to stay on Apache but we're going to have to switch to a different solution if no fix has been made since 2.4.3

Thanks
Back to top
jraute



Joined: 13 Sep 2013
Posts: 188
Location: Rheinland, Germany

PostPosted: Wed 25 Dec '13 16:59    Post subject: Reply with quote

What is the complete error line and what configuration you are using?
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU

PostPosted: Wed 25 Dec '13 17:00    Post subject: Reply with quote

When does it happen ?

Assuming that you use ProxyPass directive.

What about the keepalive timeout, front and back the same ? Good habbit is to make different, 1-2 seconds different (not recall front or back)

Also the default for Timeout has been reduced in 2.4 from 300 to 60. The developers said that if
there are complaints from users, the new value may be too low and we should maybe reconsider the new value.

So check and play with the settings:

TimeOut 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeOut 15

Running Windows ?
You can try:

AcceptFilter http none
AcceptFilter https none
EnableSendfile off
EnableMMAP off
Back to top
pidfile



Joined: 25 Dec 2013
Posts: 4

PostPosted: Wed 25 Dec '13 21:37    Post subject: Reply with quote

Thanks guys.

Linux box (CentOS 6.3)

*******************
# conf.d/foo.conf
<Location />
ProxyPass http://172.16.xx.xx:9001/ max=60 retry=10
ProxyPassReverse http://172.16.xx.xx:9001/
SetOutputFilter DEFLATE
SetEnv proxy-sendchunks 1
</Location>

# httpd.conf
KeepAlive On
KeepAliveTimeout 2500ms
EnableSendfile On
********************

Timeout shouldn't come into play, page requests typically served in < 400ms, not the 60s default.

Linux default KeepAlive timeout is 2 hours.

Again, apache dev lists do indicate that there is an as yet unresolved (as of 2.4.3 at any rate) issue with mod_proxy and keepalive race conditions.

Not sure what the underlying cause is, but getting on average at least 150 proxy errors per day _in production_ not good.
Back to top
jraute



Joined: 13 Sep 2013
Posts: 188
Location: Rheinland, Germany

PostPosted: Wed 25 Dec '13 23:05    Post subject: Reply with quote

Although you didn't post the complete error log line Wink i am quite sure that the problem still exists.
Reading the bug blog there seems to be only one solution which is based on deactivating keep alives and i remember such errors even with 2.4.7 in our logs too although the clients do not have any problems.

Have you tried the settings Steffen mentioned?
Back to top
pidfile



Joined: 25 Dec 2013
Posts: 4

PostPosted: Thu 26 Dec '13 0:43    Post subject: Reply with quote

Thanks, settings are the same as Stefan's, just shorter TTL.

Could turn off KeepAlive, but prefer performance gain, not loss.

Will dig around some more, perhaps there's something specific to my setup if clients are affected by the errors while in your case apparently they are not.
Back to top
jraute



Joined: 13 Sep 2013
Posts: 188
Location: Rheinland, Germany

PostPosted: Thu 26 Dec '13 11:26    Post subject: Reply with quote

There's one advice in the blog to use

Code:
  SetEnv proxy-initial-not-pooled 


Have you tried that?
We have this in use for many configurations with positive effects.

Btw proxy-sendchunks is only working reliably with backend servers supporting HTTP 1.1
Back to top
pidfile



Joined: 25 Dec 2013
Posts: 4

PostPosted: Thu 26 Dec '13 12:44    Post subject: Reply with quote

See http://www.gossamer-threads.com/lists/apache/dev/428261 for all the gory details. From the thread:

Quote:
As Covener pointed out to me in
IRC, using mod_proxy_http's env variable "proxy-initial-not-pooled" does
offer a solution to the problem albeit at the cost of performance.


Thread has been running for over 2 years, last updated a couple of weeks ago with a bunch of dev activity; there is as yet no solution that does not impact performance.

Our JVM app speaks HTTP 1.1, no problem there.
Back to top
jraute



Joined: 13 Sep 2013
Posts: 188
Location: Rheinland, Germany

PostPosted: Fri 27 Dec '13 12:44    Post subject: Reply with quote

Ok, no solution so far.

(Honestly spoken in our setting the "performance problem" is minimal - up to 30000 page requests per day and the performance and especially the response time is ok. But you are right, it depends on the things you want to do.)

Greets
Back to top


Reply to topic   Topic: Apache 2.4.3 mod_proxy not stable in Production View previous topic :: View next topic
Post new topic   Forum Index -> Apache