modified accelerator for variable timestep

[odunbar]

Purpose

• closes Modfiy to work with variable timestepping #339

Content

• changes the momentum parameter from $\beta_k = \frac{2}{k+2}$ into $\beta_k = \theta_k(\theta_{k-1}^{-1} - 1)$ with $\theta_{0} = 1$. The new general formulation always satisfies a required quadratic inequality needed for convergence acceleration proofs. Explicitly, one picks $\theta_k$ to satisfy $\frac{1-\theta_k}{\theta_k^2}\Delta t_k \leq \frac{\Delta t_{k-1}}{\theta_{k-1}^2}$.
• small UKI bug-fix (just stops some warnings on compile)
• moved the original NesterovAccelerator to ConstantStepNesterovAccelerator
• added 1 hr to bors timer, mac OS tests are being 10x slower today than yesterday (with no changes on our end)

Results from Unit tests

Runs additional tests with 20 steps of the DataMisfitController() learning rate scheduler. Preliminary findings

• It seems even for constant timesteps this formulation is better (in the early stages) than the original formulation
• It seems that the update seems to give benefit to all of the methods using DMC timestepper/ EKS Stable timestepper.

For Constant timestep problems

Accelerator: DefaultAccelerator Process: Inversion
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 4.507262042724763
Accelerator: ConstantStepNesterovAccelerator Process: Inversion
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 17.64047643331899
Accelerator: NesterovAccelerator Process: Inversion
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 5.059129584415419
Accelerator: DefaultAccelerator Process: TransformInversion
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 4.414729659424668
Accelerator: ConstantStepNesterovAccelerator Process: TransformInversion
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 4.394047688287434
Accelerator: NesterovAccelerator Process: TransformInversion
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 4.39590620316884
Accelerator: DefaultAccelerator Process: Unscented
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 5.537821536383912
Accelerator: ConstantStepNesterovAccelerator Process: Unscented
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 11.155144156229342
Accelerator: NesterovAccelerator Process: Unscented
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 5.574272962450679

For variable timestep problems

Accelerator: DefaultAccelerator Process: Inversion
[ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration.
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 4.883305568165869
Accelerator: NesterovAccelerator Process: Inversion
[ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration.
┌ Warning: Termination condition of timestepping scheme `DataMisfitController` has been exceeded. Preventing futher updates
│  Set on_terminate="continue" in `DataMisfitController` to ignore termination
└ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/LearningRateSchedulers.jl:278
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 4.4114237694466025
Accelerator: DefaultAccelerator Process: TransformInversion
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 4.414729659424668
Accelerator: NesterovAccelerator Process: TransformInversion
[ Info: Termination condition of timestepping scheme `DataMisfitController` will be exceeded during the next iteration.
┌ Warning: Termination condition of timestepping scheme `DataMisfitController` has been exceeded. Preventing futher updates
│  Set on_terminate="continue" in `DataMisfitController` to ignore termination
└ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/LearningRateSchedulers.jl:278
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 4.400576672334215
Accelerator: DefaultAccelerator Process: Unscented
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 5.765752171697719
Accelerator: NesterovAccelerator Process: Unscented
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 5.2125091342967265
Accelerator: DefaultAccelerator Process: Sampler
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 6.239456914512827
Accelerator: NesterovAccelerator Process: Sampler
┌ Warning: Acceleration is experimental for Sampler processes and may affect convergence.
└ @ EnsembleKalmanProcesses ~/Dropbox/Caltech/CESjl/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcess.jl:220
┌ Info: Convergence:
│   cost_initial = 123.31914542296597
└   cost_final = 4.896253011160538

---

• I have read and checked the items on the review checklist.

[eviatarbach]

Looks great! Just two things:

1. Were you able to find a reference for this? I remember you found some lecture notes; I think that would also work. It would be good to have as a comment in the code for future reference.
2. Did you verify that the EKP time-step is consistent with the notion of time-step in gradient descent?

[odunbar]

Hi Eviatar,

Great points

• I'll add a reference for this in a docstring.
• It's an interesting Q, because the growth of stable timestep is counterintuitive,

From the calculations leading to Eqn 4.41 in https://arxiv.org/pdf/2209.11371.pdf we have that the ensemble approximation is exactly addressing the gradient of the misfit $\Phi(u)$. Thus the ensemble approximation is disconnected from the time discretization.

However if we follow through the discretization, we can bridge from 4.41 to our code via
$\frac{du}{dt} = C^{uG}\Gamma^{-1}(y-G(u))$
do forward Euler
$u_{n+1} = u_n + \Delta t C^{uG}\Gamma^{-1}(y-G(u_n))$
Add in the $\Delta tC^{GG}$ scaled quantity
$u_{n+1} = u_n + \Delta t C^{uG}(\Delta t C^{GG}+ \Gamma)^{-1}(y-G(u_n))$
Add in a scaled stochastic perturbation (this is now Alg 6.3 in the paper)
$u_{n+1} = u_n + \Delta t C^{uG}(\Delta t C^{GG}+ \Gamma)^{-1}(y-G(u_n)+\eta), \ \ \eta \sim N(0, \frac{1}{\Delta t}\Gamma)$
Moving the $\Delta t$ onto the obs noise (this is our code)
$u_{n+1} = u_n + C^{uG}( C^{GG}+ \frac{1}{\Delta t}\Gamma)^{-1}(y-G(u_n)+\eta),\ \ \eta \sim N(0, \frac{1}{\Delta t} \Gamma)$

[eviatarbach]

LGTM after you run the autoformatter!

[odunbar]

bors r+

[bors]

Timed out.

[odunbar]

bors r+

[bors]

Timed out.

[odunbar]

bors r+

[bors]

Timed out.

[odunbar]

bors r+

[bors]

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• docs-build
• format
• test_success