-
Notifications
You must be signed in to change notification settings - Fork 351
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Carrier Thread pinning on stack CommittingOutputStream.flushBuffer() #5623
Comments
At first glance, |
Yes, BufferedOutputStream.toByteArray() is synchronised , but it does not lead to the specific issue of carrier thread pinning, as there is no IO happening from with this sync block. public synchronized byte[] toByteArray() { yes, It does make Array copy . Looking further at the. stack snippet. in this scenario, the concrete type of "adaptedOutput" is of. type. "TracingObserver$TracingStreamOutputDelegate" . Further, The BufferedOutputStream.write() is synchronized only when it is subclassed and not used directly. Stack Snippet
|
Interesting test results, for buffer=ByteArrayOutputStream, adaptedOutput=FileOutputStream, 32MB written:
It looks like the proposed method is about 10% slowlier, no virtual threads. So we should have this just for virtual threads. |
Perhaps a better solution is to have a ByteArrayOutputStream without synchronized blocks, with reentrant locks instead to avoid data copying. |
I have copied the ByteArrayOutputStream into a new file and replaced the synchronized with the ReentrantLock. The synchronized blocks are 10x faster. |
#5629 Is for 2.x and 3.0, we should check for Thread#isVirtual in 3.1 and forward. |
Keep open for 3.1 |
#5632 fixes the issue in 2.4.3. Helidon 4.x. uses jersey 3.1.5 Which version of 3.1 would have this fix for Helidon to uptake? |
The next one, 3.1.7 |
Has this change has been delivered into in 3.1.7? |
Yes |
Thank you @jansupol |
We have verified this fix, we no longer see pinning on this stack , post fix |
@vasanth-bhat |
We are running tests for sample application with Helidon 4.x based on Loom virtual threads with JDK 21 in the Kubernetes containers. When running load test with concurrent users we see that application hangs and JVM becomes unresponsive
we do not see any resource issues for CPU, Memory, Heap etc.. The usage are much below the configured capacity.
In JFR, we do seeing large number of carrier thread pinning on the below stack. The pinning is happening when the. org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer() is invoking ByteArrayOutputStream.writeTo().
stack snippet below from tracePinnedThreads output:
io.helidon.webserver.observe.tracing.TracingObserver$TracingStreamOutputDelegate.write(TracingObserver.java:525)
java.base/java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:163) <== monitors:1
org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:278) org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232) org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247) org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:865) org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:403)
The recommendation from teh JDK team was to modify the implementation of "org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer()" to avoid calling the synchronised method. ByteArrayOutputStream.writeTo()
Based on the suggestion, the below code change was tried out, in the implementation of CommittingOutputStream.flushBuffer()
private void flushBuffer(boolean endOfStream) throws IOException {
.........
............
if (buffer != null && adaptedOutput != null) {
//buffer.writeTo(adaptedOutput);
adaptedOutput.write(buffer.toByteArray());
}
}
}
With this change, we no longer see the carrier thread pinning on this stack.
The ask is to see if this code change can be implemented in Jersey.
The text was updated successfully, but these errors were encountered: