Topic: httpclient.ConnectTimeoutException to Apache Httpd

[martinr]

Hello,

Our application makes SOAP requests (using Apache commons htttclient) which are proxied through apache web server to the endpoint:
Application -> Apache HTTPD -> Service endpoint.

Lately we are quite frequently getting the following errors when the requests are made:
Caused By: org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 3000 ms
at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:155)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)

The initial timeout was 1000ms, which we now changed to 3000ms, but the problem seems to persist with the same frequency. Most of the requests are going through, but some are getting the error above.

Some remarks and things I have tried to narrow down the problem:
* I tried using curl to make regular HTTP GET requests and measure the connect time using the example provided here:
https://blog.josephscott.org/2011/10/14/timing-details-with-curl/

This test did not show such high connect times we are experiencing in our application

* For one of our application`s subcomponents I removed apache httpd from the connection chain and configured it to connect directly to the endpoint. So instead of "Application -> Apache httpd -> Endpoint" we now have "Application -> Endpoint."

Currently it seems that there have been no problems that way. This seems to indicate that the problem only appears when the requests are proxied through apache server.

* The problems appear quite frequently during off-hours as well when the application and apache work load is significantly lower than during work hours.

----------------

Does anybody have ideas how to further narrow down the problem we are having.

As I understand the problem appears before any actual application data is sent (during application -> apache httpd:80/443 socket creation). What could be the reason for it? Any hints and pointers are greatly appreciated.