Modfiy to work with variable timestepping #339
Comments
This was referenced Oct 24, 2023
odunbar
changed the title
bors bot
added a commit
that referenced
this issue
Oct 30, 2023
341: modified accelerator for variable timestep r=odunbar a=odunbar <!--- THESE LINES ARE COMMENTED --> ## Purpose <!--- One sentence to describe the purpose of this PR, refer to any linked issues: #14 -- this will link to issue 14 Closes #2 -- this will automatically close issue 2 on PR merge --> - closes #339 ## Content <!--- specific tasks that are currently complete - Solution implemented --> - changes the momentum parameter from $\beta_k = \frac{2}{k+2}$ into $\beta_k = \theta_k(\theta_{k-1}^{-1} - 1)$ with $\theta_{0} = 1$. The new general formulation always satisfies a required quadratic inequality needed for convergence acceleration proofs. Explicitly, one picks $\theta_k$ to satisfy $\frac{1-\theta_k}{\theta_k^2}\Delta t_k \leq \frac{\Delta t_{k-1}}{\theta_{k-1}^2}$. - small UKI bug-fix (just stops some warnings on compile) - moved the original `NesterovAccelerator` to `ConstantStepNesterovAccelerator` ## Results from Unit tests Runs additional tests with 20 steps of the `DataMisfitController()` learning rate scheduler. Preliminary findings - It seems even for constant timesteps this formulation is better (in the early stages) than the original formulation - It seems that the update seems to give benefit to all of the methods using DMC timestepper/ EKS Stable timestepper. ### For Constant timestep problems ``` Accelerator: DefaultAccelerator Process: Inversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.507262042724763 Accelerator: ConstantStepNesterovAccelerator Process: Inversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 17.64047643331899 Accelerator: NesterovAccelerator Process: Inversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.059129584415419 Accelerator: DefaultAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.414729659424668 Accelerator: ConstantStepNesterovAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.394047688287434 Accelerator: NesterovAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.39590620316884 Accelerator: DefaultAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.537821536383912 Accelerator: ConstantStepNesterovAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 11.155144156229342 Accelerator: NesterovAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.574272962450679 ``` ### For variable timestep problems ``` Accelerator: DefaultAccelerator Process: Inversion [ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration. ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.883305568165869 Accelerator: NesterovAccelerator Process: Inversion [ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration. ┌ Warning: Termination condition of timestepping scheme `DataMisfitController` has been exceeded. Preventing futher updates │ Set on_terminate="continue" in `DataMisfitController` to ignore termination └ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/LearningRateSchedulers.jl:278 ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.4114237694466025 Accelerator: DefaultAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.414729659424668 Accelerator: NesterovAccelerator Process: TransformInversion [ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration. ┌ Warning: Termination condition of timestepping scheme `DataMisfitController` has been exceeded. Preventing futher updates │ Set on_terminate="continue" in `DataMisfitController` to ignore termination └ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/LearningRateSchedulers.jl:278 ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.400576672334215 Accelerator: DefaultAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.765752171697719 Accelerator: NesterovAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.2125091342967265 Accelerator: DefaultAccelerator Process: Sampler ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 6.239456914512827 Accelerator: NesterovAccelerator Process: Sampler ┌ Warning: Acceleration is experimental for Sampler processes and may affect convergence. └ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcess.jl:220 ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.896253011160538 ``` <!--- Review checklist I have: - followed the codebase contribution guide: https://clima.github.io/ClimateMachine.jl/latest/Contributing/ - followed the style guide: https://clima.github.io/ClimateMachine.jl/latest/DevDocs/CodeStyle/ - followed the documentation policy: https://github.com/CliMA/policies/wiki/Documentation-Policy - checked that this PR does not duplicate an open PR. In the Content, I have included - relevant unit tests, and integration tests, - appropriate docstrings on all functions, structs, and modules, and included relevant documentation. --> ---- - [x] I have read and checked the items on the review checklist. Co-authored-by: odunbar <[email protected]>
bors bot
added a commit
that referenced
this issue
Oct 30, 2023
341: modified accelerator for variable timestep r=odunbar a=odunbar <!--- THESE LINES ARE COMMENTED --> ## Purpose <!--- One sentence to describe the purpose of this PR, refer to any linked issues: #14 -- this will link to issue 14 Closes #2 -- this will automatically close issue 2 on PR merge --> - closes #339 ## Content <!--- specific tasks that are currently complete - Solution implemented --> - changes the momentum parameter from $\beta_k = \frac{2}{k+2}$ into $\beta_k = \theta_k(\theta_{k-1}^{-1} - 1)$ with $\theta_{0} = 1$. The new general formulation always satisfies a required quadratic inequality needed for convergence acceleration proofs. Explicitly, one picks $\theta_k$ to satisfy $\frac{1-\theta_k}{\theta_k^2}\Delta t_k \leq \frac{\Delta t_{k-1}}{\theta_{k-1}^2}$. - small UKI bug-fix (just stops some warnings on compile) - moved the original `NesterovAccelerator` to `ConstantStepNesterovAccelerator` ## Results from Unit tests Runs additional tests with 20 steps of the `DataMisfitController()` learning rate scheduler. Preliminary findings - It seems even for constant timesteps this formulation is better (in the early stages) than the original formulation - It seems that the update seems to give benefit to all of the methods using DMC timestepper/ EKS Stable timestepper. ### For Constant timestep problems ``` Accelerator: DefaultAccelerator Process: Inversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.507262042724763 Accelerator: ConstantStepNesterovAccelerator Process: Inversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 17.64047643331899 Accelerator: NesterovAccelerator Process: Inversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.059129584415419 Accelerator: DefaultAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.414729659424668 Accelerator: ConstantStepNesterovAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.394047688287434 Accelerator: NesterovAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.39590620316884 Accelerator: DefaultAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.537821536383912 Accelerator: ConstantStepNesterovAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 11.155144156229342 Accelerator: NesterovAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.574272962450679 ``` ### For variable timestep problems ``` Accelerator: DefaultAccelerator Process: Inversion [ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration. ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.883305568165869 Accelerator: NesterovAccelerator Process: Inversion [ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration. ┌ Warning: Termination condition of timestepping scheme `DataMisfitController` has been exceeded. Preventing futher updates │ Set on_terminate="continue" in `DataMisfitController` to ignore termination └ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/LearningRateSchedulers.jl:278 ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.4114237694466025 Accelerator: DefaultAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.414729659424668 Accelerator: NesterovAccelerator Process: TransformInversion [ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration. ┌ Warning: Termination condition of timestepping scheme `DataMisfitController` has been exceeded. Preventing futher updates │ Set on_terminate="continue" in `DataMisfitController` to ignore termination └ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/LearningRateSchedulers.jl:278 ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.400576672334215 Accelerator: DefaultAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.765752171697719 Accelerator: NesterovAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.2125091342967265 Accelerator: DefaultAccelerator Process: Sampler ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 6.239456914512827 Accelerator: NesterovAccelerator Process: Sampler ┌ Warning: Acceleration is experimental for Sampler processes and may affect convergence. └ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcess.jl:220 ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.896253011160538 ``` <!--- Review checklist I have: - followed the codebase contribution guide: https://clima.github.io/ClimateMachine.jl/latest/Contributing/ - followed the style guide: https://clima.github.io/ClimateMachine.jl/latest/DevDocs/CodeStyle/ - followed the documentation policy: https://github.com/CliMA/policies/wiki/Documentation-Policy - checked that this PR does not duplicate an open PR. In the Content, I have included - relevant unit tests, and integration tests, - appropriate docstrings on all functions, structs, and modules, and included relevant documentation. --> ---- - [x] I have read and checked the items on the review checklist. Co-authored-by: odunbar <[email protected]>
bors bot
added a commit
that referenced
this issue
Oct 30, 2023
341: modified accelerator for variable timestep r=odunbar a=odunbar <!--- THESE LINES ARE COMMENTED --> ## Purpose <!--- One sentence to describe the purpose of this PR, refer to any linked issues: #14 -- this will link to issue 14 Closes #2 -- this will automatically close issue 2 on PR merge --> - closes #339 ## Content <!--- specific tasks that are currently complete - Solution implemented --> - changes the momentum parameter from $\beta_k = \frac{2}{k+2}$ into $\beta_k = \theta_k(\theta_{k-1}^{-1} - 1)$ with $\theta_{0} = 1$. The new general formulation always satisfies a required quadratic inequality needed for convergence acceleration proofs. Explicitly, one picks $\theta_k$ to satisfy $\frac{1-\theta_k}{\theta_k^2}\Delta t_k \leq \frac{\Delta t_{k-1}}{\theta_{k-1}^2}$. - small UKI bug-fix (just stops some warnings on compile) - moved the original `NesterovAccelerator` to `ConstantStepNesterovAccelerator` - added 1 hr to bors timer, mac OS tests are being 10x slower today than yesterday (with no changes on our end) ## Results from Unit tests Runs additional tests with 20 steps of the `DataMisfitController()` learning rate scheduler. Preliminary findings - It seems even for constant timesteps this formulation is better (in the early stages) than the original formulation - It seems that the update seems to give benefit to all of the methods using DMC timestepper/ EKS Stable timestepper. ### For Constant timestep problems ``` Accelerator: DefaultAccelerator Process: Inversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.507262042724763 Accelerator: ConstantStepNesterovAccelerator Process: Inversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 17.64047643331899 Accelerator: NesterovAccelerator Process: Inversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.059129584415419 Accelerator: DefaultAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.414729659424668 Accelerator: ConstantStepNesterovAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.394047688287434 Accelerator: NesterovAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.39590620316884 Accelerator: DefaultAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.537821536383912 Accelerator: ConstantStepNesterovAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 11.155144156229342 Accelerator: NesterovAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.574272962450679 ``` ### For variable timestep problems ``` Accelerator: DefaultAccelerator Process: Inversion [ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration. ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.883305568165869 Accelerator: NesterovAccelerator Process: Inversion [ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration. ┌ Warning: Termination condition of timestepping scheme `DataMisfitController` has been exceeded. Preventing futher updates │ Set on_terminate="continue" in `DataMisfitController` to ignore termination └ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/LearningRateSchedulers.jl:278 ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.4114237694466025 Accelerator: DefaultAccelerator Process: TransformInversion ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.414729659424668 Accelerator: NesterovAccelerator Process: TransformInversion [ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration. ┌ Warning: Termination condition of timestepping scheme `DataMisfitController` has been exceeded. Preventing futher updates │ Set on_terminate="continue" in `DataMisfitController` to ignore termination └ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/LearningRateSchedulers.jl:278 ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.400576672334215 Accelerator: DefaultAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.765752171697719 Accelerator: NesterovAccelerator Process: Unscented ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 5.2125091342967265 Accelerator: DefaultAccelerator Process: Sampler ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 6.239456914512827 Accelerator: NesterovAccelerator Process: Sampler ┌ Warning: Acceleration is experimental for Sampler processes and may affect convergence. └ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcess.jl:220 ┌ Info: Convergence: │ cost_initial = 123.31914542296597 └ cost_final = 4.896253011160538 ``` <!--- Review checklist I have: - followed the codebase contribution guide: https://clima.github.io/ClimateMachine.jl/latest/Contributing/ - followed the style guide: https://clima.github.io/ClimateMachine.jl/latest/DevDocs/CodeStyle/ - followed the documentation policy: https://github.com/CliMA/policies/wiki/Documentation-Policy - checked that this PR does not duplicate an open PR. In the Content, I have included - relevant unit tests, and integration tests, - appropriate docstrings on all functions, structs, and modules, and included relevant documentation. --> ---- - [x] I have read and checked the items on the review checklist. Co-authored-by: odunbar <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: