-
Notifications
You must be signed in to change notification settings - Fork 38k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallel graceful shutdown for ThreadPoolTaskExecutor
and ThreadPoolTaskScheduler
#27090
Comments
Implementation considerationsProbably the implementation might make In my opinion, the implementation of In the After, the |
Hello people. |
@rstoyanchev , I do not want to contribute to this vast amount of issues in the spring project. As it seems, the community creates an amount of issues greater than the capacity of yours to analyze them. Feel free to just close the issue. |
Spring team, Here, the lenghty application shutdown remains. Thanks. |
If I work on the code and submit a PR for this, it would be considered? |
In the context of Project CRaC support, we are revisiting our own lifecycle adapters, and we ran into the executor/scheduler implementations as well. It is somewhat more complicated with stop/restart having to be possible, so a shutdown on We'll try to sort this out for our 6.1. |
The challenge with the Up until this point, we treated For Project CRaC, it is not actually necessary to stop/restart thread pools since the active threads are a part of the checkpoint image anyway. So we currently only stop @jgslima what kind of tasks are typically left to await termination for in your scenario, and how did those tasks get submitted? Is this about late-submitted one-off tasks during the shutdown process, or rather about periodic tasks kicking in once more? |
In terms of an early shutdown signal for executors and schedulers, it does not have to come from |
Experimenting with this a bit, we seem to be able to implement the general This combines nicely with a |
Hello.
Good point. I had not thought about it.
Ok, that seems fine. It also solves another point, that is, if we really would implement
I understand.
It is not about late-submitted tasks. Some examples of thread pools used in some applications I work with:
My little "obsession" to have a graceful shutdown with the most possible efficiency, is to avoid loose database locks when some task is aborted abruptly. These create some headaches (MySQL by default only kills inactive sessions after 8 hours). It is true that really long running tasks (like a full batch process) should not be run with any of the means above. For instance, in Kubernetes, a long running task should be submitted as a Job, which is not asked to be stopped before its conclusion. However in some cases above, the task may involve doing something potentially slow, like making a remote call. When this is the case, ideally we would like to configure I would love to help with with this change, if you want, please tell me. Thanks |
@jhoeller , since you will change Something like We miss this in order to be able to provide a I think it is better to expose just the size and not the Queue as whole, to avoid direct interactions with it. Thanks |
ThreadPoolTaskExecutor
and ThreadPoolTaskScheduler
@jgslima it would be great if you could give the upcoming |
You bet, I will give a try and get back to you. |
@jhoeller, sorry for taking so long. I tested it and at first I did not really get it, then I saw the new code and understood. I have 2 comments:
Please tell me if I can help somehow. |
My mistake. The containers, like you said, use executors which are not context beans. But this is not the case for |
Applications that need to perform graceful shutdown of tasks submitted to
ThreadPoolTaskExecutor
and/orThreadPoolTaskScheduler
, may make use of the settingawaitTerminationMillis
(and possiblywaitForTasksToCompleteOnShutdown
).However, application may take a very long time to actually finish if these conditions apply:
ThreadPoolTaskExecutor
andThreadPoolTaskScheduler
(and possibly multipleThreadPoolTaskExecutor
s).SmartLifecycle
beans which implements lenghty stopping.Real examples are web applications that use such components and make use of the web container "graceful shutdown" feature of Spring Boot. The overall termination sequence of an application is:
SmartLifecycle
asynchronous stopping triggers the web container graceful shutdown.SmartLifecycle
asynchronous stopping blocks and waits for the web container shutdown.DisposableBean
/@PreDestroy
methods, so, say:ThreadPoolTaskExecutor
'sdestroy()
is called, blocking and awaiting for the tasks to finish. If the application uses multipleThreadPoolTaskExecutor
s, for each one of them an awaiting occurs.ThreadPoolTaskScheduler
'sdestroy()
is called, blocking and awaiting for the tasks to finish.The proposal here is to create the possibility to have some form to make all the pools to finish their tasks in parallel. Ideally, in parallel with the stopping of other
SmartLifecycle
beans.In Kubernetes spring web applications that fits in the scenario above, the total time took by a Pod to finish ends up being too high (which is aggravated due to the need to configure a preStop hook to give time to kubeproxy to note the pod deletion). This has real effects: as, rigthly, in a rollout Kubernetes does not wait for old Pods to actually finish in order to create new ones, applications with a large number of pods and with a large termination time, end up having a large number of Pods actually running in the rollout (new Pods and many still being terminated). We have seen this triggering a cluster auto scale to make the cluster able to handle this large number of Pods that occur during the rollout.
The text was updated successfully, but these errors were encountered: