Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: Requests stuck in queue due to termination of I/O Reactor |
|
| Author |
|
kbandi
Joined: 20 Sep 2023
Posts: 2
Location: India
|
 Posted: Thu 21 Sep '23 10:45 Post subject: Requests stuck in queue due to termination of I/O Reactor |
 |
|
Library & version: Apache httpasyncclient - 4.1.4
Operating System: Windows
Runtime Environment: Java 1.8
Issue description:
Requests are stuck in the requestQueue of IO Reactor (DefaultConnectingIOReactor) but not processed due to abnormal termination of I/O Reactor thread that processes the requests (started as part of calling the start() in CloseableHttpAsyncClientBase).
I've enabled http logging and collected the following error logs: | Quote: |
DEBUG 2023-09-18 10:16:38,706Z [parallel_; tid=11876601] [ o.a.http.impl.nio.client.MainClientExec] : [exchange: 47975] start execution
DEBUG 2023-09-18 10:16:38,706Z [parallel_; tid=11876601] [ o.a.h.client.protocol.RequestAddCookies] : CookieSpec selected: standard
DEBUG 2023-09-18 10:16:38,706Z [parallel_; tid=11876601] [ o.a.h.client.protocol.RequestAuthCache] : Auth cache not set in the context
DEBUG 2023-09-18 10:16:38,706Z [parallel_; tid=11876601] [o.a.h.i.n.client.InternalHttpAsyncClient] : [exchange: 47975] Request connection for {s}-> xxxxxxxxxxxxxxxxxxxxx
DEBUG 2023-09-18 10:16:38,706Z [parallel_; tid=11876601] [.n.c.PoolingNHttpClientConnectionManager] : Connection request: [route: {s}->xxxxxxxxxxxxxxxxxxxxxxxxx][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
ERROR 2023-09-18 10:16:38,757Z [ pool-48041-thread-1] [o.a.h.i.n.client.InternalHttpAsyncClient] : I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: Failure opening selector
at org.apache.http.impl.nio.reactor.AbstractIOReactor.<init>(AbstractIOReactor.java:103)
at org.apache.http.impl.nio.reactor.BaseIOReactor.<init>(BaseIOReactor.java:85)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:318)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: Unable to establish loopback connection
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:171)
at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
at java.nio.channels.Pipe.open(Pipe.java:155)
at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:142)
at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
at java.nio.channels.Selector.open(Selector.java:227)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.<init>(AbstractIOReactor.java:101)
... 5 common frames omitted
Caused by: java.net.SocketException: Socket operation on nonsocket: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:461)
at sun.nio.ch.Net.bind(Net.java:453)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78)
at sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(PipeImpl.java:121)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:76)
... 14 common frames omitted
|
I would like to know what is the cause for underlying exception:
| Quote: | Caused by: java.net.SocketException: Socket operation on nonsocket: bind
|
Is it environment issue that can be fixed with any configuration changes on Windows system?
If it is non-recoverable exception, why would the HttpAsyncClient doesn't shutdown the connection manager and all relevant components and respond back to the caller by calling the callback methods?
Finally, Can I expect a fix for apache team on this issue? thanks. |
|
| Back to top |
|
kbandi
Joined: 20 Sep 2023
Posts: 2
Location: India
|
| Posted: Thu 21 Sep '23 12:45 Post subject: |
|
|
I believe it is a bug where there is no exception handling when creating IO dispatchers.
Class: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor
| Code: |
public void execute(
final IOEventDispatch eventDispatch) throws InterruptedIOException, IOReactorException {
Args.notNull(eventDispatch, "Event dispatcher");
synchronized (this.statusLock) {
if (this.status.compareTo(IOReactorStatus.SHUTDOWN_REQUEST) >= 0) {
this.status = IOReactorStatus.SHUT_DOWN;
this.statusLock.notifyAll();
return;
}
Asserts.check(this.status.compareTo(IOReactorStatus.INACTIVE) == 0,
"Illegal state %s", this.status);
this.status = IOReactorStatus.ACTIVE;
// Start I/O dispatchers
for (int i = 0; i < this.dispatchers.length; i++) {
final BaseIOReactor dispatcher = new BaseIOReactor(this.selectTimeout, this.interestOpsQueueing);
dispatcher.setExceptionHandler(exceptionHandler);
this.dispatchers[i] = dispatcher;
}
for (int i = 0; i < this.workerCount; i++) {
final BaseIOReactor dispatcher = this.dispatchers[i];
this.workers[i] = new Worker(dispatcher, eventDispatch);
this.threads[i] = this.threadFactory.newThread(this.workers[i]);
}
}
try {
for (int i = 0; i < this.workerCount; i++) {
if (this.status != IOReactorStatus.ACTIVE) {
return;
}
this.threads[i].start();
}
for (;;) {
final int readyCount;
try {
readyCount = this.selector.select(this.selectTimeout);
} catch (final InterruptedIOException ex) {
throw ex;
} catch (final IOException ex) {
throw new IOReactorException("Unexpected selector failure", ex);
}
if (this.status.compareTo(IOReactorStatus.ACTIVE) == 0) {
processEvents(readyCount);
}
// Verify I/O dispatchers
for (int i = 0; i < this.workerCount; i++) {
final Worker worker = this.workers[i];
final Throwable ex = worker.getThrowable();
if (ex != null) {
throw new IOReactorException(
"I/O dispatch worker terminated abnormally", ex);
}
}
if (this.status.compareTo(IOReactorStatus.ACTIVE) > 0) {
break;
}
}
} catch (final ClosedSelectorException ex) {
addExceptionEvent(ex);
} catch (final IOReactorException ex) {
if (ex.getCause() != null) {
addExceptionEvent(ex.getCause());
}
throw ex;
} finally {
doShutdown();
synchronized (this.statusLock) {
this.status = IOReactorStatus.SHUT_DOWN;
this.statusLock.notifyAll();
}
}
}
|
From the above method, following block should go into try catch block,
| Code: |
// Start I/O dispatchers
for (int i = 0; i < this.dispatchers.length; i++) {
final BaseIOReactor dispatcher = new BaseIOReactor(this.selectTimeout, this.interestOpsQueueing);
dispatcher.setExceptionHandler(exceptionHandler);
this.dispatchers[i] = dispatcher;
}
for (int i = 0; i < this.workerCount; i++) {
final BaseIOReactor dispatcher = this.dispatchers[i];
this.workers[i] = new Worker(dispatcher, eventDispatch);
this.threads[i] = this.threadFactory.newThread(this.workers[i]);
}
|
so that if there are any exceptions while creating IO dispatchers, shutdown() will be called and all the components down the line are shutdown and cancel() will be called for future callbacks references, thus the caller won't wait indefinitely for the response from Apache Http client. |
|
| Back to top |
|
|
|
|
|
|
