Topic: Apache 2.2.26 with backend server - http 503

[Maycon Oliveira]

Hi there.

I have and apache server in front a wildfly server running for 2 years now. Recently I deployed on wildfly a java web start program (a desktop program that has auto-update on startup) and when e start the program, my apache server start to respond http 503 for everything. Then i need to wait for a while and sometimes i need to restart my backend server.

If I bypass apache and connect directly to wildfly, everything runs ok.

My suspicious is that apache is not handling http 304 from wildfly. Or maybe apache is running out of resources? the files has 3mb em average.

Apache log:

[Sun Feb 28 16:51:52 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:51:52 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:52:04 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:52:04 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:54:51 2016] [error] (105)No buffer space available: proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[Sun Feb 28 16:54:51 2016] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Sun Feb 28 16:54:51 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:54:51 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:54:51 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:54:52 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:54:52 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:54:52 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:54:52 2016] [error] proxy: HTTP: disabled connection for (localhost)
[Sun Feb 28 16:54:52 2016] [error] proxy: HTTP: disabled connection for (localhost)


Wildfly responses log:


FIRST REQUEST:
----------------------------REQUEST---------------------------
URI=/xxxpdvweb/lib/liquibase.jar
characterEncoding=null
contentLength=-1
contentType=null
header=Connection=Keep-Alive
header=X-Forwarded-For=189.114.143.103
header=X-Forwarded-Server=_wildcard_.xxx.com.br
header=User-Agent=Apache-HttpClient/4.2.6 (java 1.5)
header=Host=demo.xxx.com.br
header=X-Forwarded-Host=demo.xxx.com.br
locale=[]
method=GET
protocol=HTTP/1.1
queryString=
remoteAddr=/127.0.0.1:21991
remoteHost=localhost
scheme=http
host=demo.xxx.com.br
serverPort=8080

--------------------------RESPONSE--------------------------
contentLength=2183684
contentType=application/octet-stream
header=Connection=keep-alive
header=Last-Modified=Fri, 26 Feb 2016 01:07:10 GMT
header=X-Powered-By=Undertow/1
header=Server=WildFly/8
header=Content-Type=application/octet-stream
header=Content-Length=2183684
header=Date=Sun, 28 Feb 2016 20:36:52 GMT
status=200

SECOND REQUEST:
----------------------------REQUEST---------------------------
URI=/xxxpdvweb/lib/liquibase.jar
characterEncoding=null
contentLength=-1
contentType=null
header=Connection=Keep-Alive
header=X-Forwarded-For=189.114.143.103
header=X-Forwarded-Server=_wildcard_.xxx.com.br
header=User-Agent=Apache-HttpClient/4.2.6 (java 1.5)
header=If-Modified-Since=Fri, 26 Feb 2016 01:07:10 GMT
header=Host=demo.xxx.com.br
header=X-Forwarded-Host=demo.xxx.com.br
locale=[]
method=GET
protocol=HTTP/1.1
queryString=
remoteAddr=localhost/127.0.0.1:21989
remoteHost=localhost
scheme=http
host=demo.xxx.com.br
serverPort=8080
--------------------------RESPONSE--------------------------
contentLength=0
contentType=null
header=Connection=keep-alive
header=X-Powered-By=Undertow/1
header=Server=WildFly/8
header=Content-Length=0
header=Date=Sun, 28 Feb 2016 20:37:01 GMT
status=304



Anybody has a hint?

Thanks

[Maycon Oliveira]

Almost giving up and connecting directly with the backend server.

Sometimes seems the problem is with OS (centos) but Im really in the dark.

[James Blond]

Do you use mod_jk or mod_proxy_ajp ?

[Maycon Oliveira]

Hi there!

Sorry for the delay. I was waiting to see if a change that i made solved the problem (Increasing prefork values). But i didnt suceeded.


I use this configuration to redirect the requests

ProxyPreserveHost On
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/



The problem persists. I´m almost giving up apache and connecting directly to my aplication server.


Thanks!

[James Blond]

I had simular problems with my jenkins server. Now I use mod_proxy_ajp and it works fine.

[Maycon Oliveira]

I changed to ajp and now the problem is happening later on the tests (the apache seems to holding better the connections). But connecting directly with the backend server the problem doesn´t happend at all!

No error logs on the backend server.

Looks like that the problem is some network configuration on the machine.

The error nows, changed a little bit:


[Sun Mar 27 15:46:40 2016] [error] (105)No buffer space available: proxy: AJP: attempt to connect to 127.0.0.1:8009 (localhost) failed
[Sun Mar 27 15:46:40 2016] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Sun Mar 27 15:46:40 2016] [error] proxy: AJP: failed to make connection to backend: localhost

[James Blond]

OK then your backend server doesn't speak AJP.
However the problem is that the backend server fails.

[Maycon Oliveira]

In fact, the backend speak ajp. And this configuration work while i dont request a lot a large files using some test tool like jmeter.

Then the apache start to respond 503 and these logs apear.

[James Blond]

Can you run several intances of your backend server? Cause with mod_proxy_ajp and mod_balancer you can put the load on several instances. Another good way is to exclude the static files from the reverse proxy and put them in the front end / apache server. Then the java backend does only the work for the non static requests where logic is needed.