Timesteppers for EKP, from ParameterEstimocean #285

odunbar · 2023-05-09T23:48:00Z

Coauthored with @eviatarbach

Purpose

Closes #253
Closes #154

Content

Implemented informative warning message for building
EnsembleKalmanProcess(...; Deltat = x, ...)
In favour of the new constructor
EnsembleKalmanProcess(...; scheduler = DefaultScheduler(x), ...)
and updated all unit tests to the latter.
New LearningRateScheduler type and Objects in backwards compatible way, by adding the keyword scheduler = ...
1. DefaultScheduler(k) (always does step k unless user overrides at a given step),
2. MutableScheduler() (initially does step 1, until user overrides with k, then does k thereafter until user next overrides)
3. EKSStableScheduler() - performs the stable EKS step calculation externally, in place of the original internal one.
4. DataMisfitController() - based on Iglesias Yan 2021's Bayesian tempering scheme.
New Defaults are DefaultScheduler(1) usually, and EKSStableScheduler for EKS.
Current examples no longer have Deltat= x expressions in constructors of EKP object.
calculate_timestep! can output a flag if a termination condition is met, this will prevent further EKP updates (e.g. for DataMisfitController
Further options for DMC when it reaches the termination condition. on_terminate = stop continue or continue_fixed to either terminate, continue, or continue with latest timestep fixed.
Example comparing different schedulers (see results below)
Docstrings & docs page, EKI and UKI example: SEE DOCS HERE

Also Changed

A test coding pattern, where the forward map G transformed parameters internally, rather than using the parameter distributions to to return "constrained parameters" which is the proper way these should be used
The Nonlinear forward map that had a multiplicative noise internal variability, now has a no-noise and additive noise version used in testing (a more appropriate problem for EKP).
UKI initialization (/bugfix?) for timestep = 0.0 not \Delta t. Allows for constructor cleanup and clean interface with new schedulers.
UKI now is consistent with other tools in building Unscented{FT,IT} types, not unparameterized Unscented types.

Example experiment - explained more clearly in docs

Learning (A,B) from f = exp(Asin(t) + B), from observing only max(f) and mean(f)

Final model solutions, as well as the max and mean of the solution
left: black = true model (not the observed data)
right: black = observed truth, grey = mean/2*sd of true model observations

Individual experiment behaviour

mean ensemble member error to truth, (error), and mean ensemble member error to ensemble mean (spread).
Avoiding ensemble collapse => dashed and solid lines are similar.

Green - constant step 0.02 for 50 steps (finishes at T=1)
Orange - constant step 0.5 for 50 steps (finishes at T=25)
Red - adaptive step, usually used for EKS (finishes at T=10^13?)
Purple - adaptive DMC step terminates at 12 steps (finishes at T=1)
Brown - adaptive DMC step with on_terminate = continue (finishes at T = 10^15)

Average convergence over 100 experiments

50 ensemble members and 50 iterations (DMC typically terminates ~14-20 iterations)
Method : DefaultScheduler{Float64}(0.5)
Final misfit: 0.011919044450605893
Final error : 0.027449304379020753
Final spread: 0.0031725385439674517

Method : DefaultScheduler{Float64}(0.02)
Final misfit: 1.0919040792305175
Final error : 0.09912866440799409
Final spread: 0.07396181868428042

Method : EKSStableScheduler{Float64}(1.0, 2.220446049250313e-16)
Final misfit: 5.648615687716918e-5
Final error : 0.03018464453288676
Final spread: 1.9601185362422715e-5

Method : DataMisfitController{...}(..., "stop")
Final misfit: 1.1677896425071728
Final error : 0.09817906335167692
Final spread: 0.08739534124484036

Method : DataMisfitController{...}(..., "continue")
Final misfit: 8.52533406253277e-17
Final error : 0.030283499635998162
Final spread: 5.974713527239014e-17

Conclusion?

These results imply that

DMC behaves similarly to fixed small-step EKI on 0->1, but is far more efficient with regards to step size
DMC and fixed small-step EKI on 0->1 both retain better spread in the ensemble than other options,
Termination at T=1 (even averaged over 100's of runs) seems to give only conservative estimates of the mean, to an order of magnitude or more. (Likely to protect the spread of the ensemble?)
Continuing DMC causes it to mimic EKSStableScheduler behaviour but with less steps. (Brown is similar to red)

I have read and checked the items on the review checklist.

odunbar · 2023-05-16T22:10:57Z

Note tests failing due to upgrades to Julia v1.9.0 - they do pass in v1.8.5. This will be addressed shortly

odunbar · 2023-05-17T17:32:22Z

#286

eviatarbach

Great work on this! I made a few comments in the code. I think the main issues are lack of tests for UKI and sparse EKI.

docs/src/learning_rate_scheduler.md

eviatarbach · 2023-05-25T20:08:43Z

test/EnsembleKalmanProcess/runtests.jl

+
+
+
+    # build EKP and eki objects


I think we need tests for UKI and sparse EKI as well.

I have added both sets of tests. The Sparse inversion tests are in the SparseInversion/ test directory.

Currently UKI tests with the DataMisfitController do not appear to converge, I will investigate later what is going on.

I've added UKI example in the docs too now, and here it seems to perform ok. I wonder if there is some quirk in the test setup that's giving it problems

src/LearningRateSchedulers.jl

docs/src/learning_rate_scheduler.md

odunbar · 2023-05-31T23:37:01Z

Leaving UKI alone, as I have made some easier tests and it still seems to have poor performance when solving nonlinear problems. I suspect a bug rather than anything to do with schedulers (causes issues on fixed timestep testing too). Commenting out UKI test, to be uncommented when performance has been thoroughly assessed.

I would like to merge this PR to introduce the new features, and have created a new issue #287 to investigate the UKI

[The example + docs are still working and so I have left them. I have only removed the unit tests for schedulers+UKI for now.]

eviatarbach · 2023-06-01T20:21:43Z

Looks great to me! Thank you for all your work on this. Not sure why a test is failing on Windows but I think this should be ready to merge after that passes.

odunbar · 2023-06-01T22:29:46Z

bors try

odunbar · 2023-06-01T22:30:35Z

The windows test passed on rerun - I'll give it a couple more times to see if it is recurring

bors · 2023-06-01T22:39:05Z

try

Build failed:

test_success

loop branch for mutable timestepper added parameters to EKSStabilityTimestepper added DMC timestepper timestepper interface bugfixes format externalize dim-check error, bugfix stableEKS timestep Unscented consistent, and tests pass update Delt =... to timestepper = ... in unit tests` update example unit test for constructors format consistency wiith 1,9 (not-yet-refined) timestepper example with various plots Timestepper termination condition upheld docstrings unit tests loosen tol typo improved output example loosen tolerance cleaned up plots and experiment repeated for many runs format removed nothing # hide, and added more strings rename timestepper to scheduler added continue options for DMC run test longer preliminary docs for scheduler comparison example improve plot legend and docs typos etc examples/LearningRateSchedulers/compare_schedulers.jl move docs page out of examples subfolder move docs in contents modify conclusions API docstrings docs and example improvements added SEKI and UKI tests for schedulers (UKI fails), fixed dangerous pattern where G applied to untransformed coordinates format uki example and docs consistency with seeding for plots separate posterior vs optimize, add uki descriptions format typo easier tests for UKI, but it still fails extend num steps robust test UKI comments return nothing explicitly

odunbar · 2023-06-02T00:49:06Z

bors try

odunbar · 2023-06-02T01:02:25Z

bors r+

bors · 2023-06-02T01:03:12Z

try

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

bors · 2023-06-02T01:17:54Z

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

odunbar force-pushed the orad/timesteppers branch from 016c9f2 to 82a2827 Compare May 17, 2023 16:52

odunbar force-pushed the orad/timesteppers branch 7 times, most recently from cb6e42f to 9958b4c Compare May 24, 2023 19:51

odunbar force-pushed the orad/timesteppers branch 2 times, most recently from 401a171 to b6d32b9 Compare May 29, 2023 15:29

eviatarbach requested changes May 30, 2023

View reviewed changes

odunbar force-pushed the orad/timesteppers branch 2 times, most recently from 12e2973 to 0c5a2e5 Compare May 31, 2023 23:44

odunbar requested a review from eviatarbach May 31, 2023 23:46

odunbar force-pushed the orad/timesteppers branch from 9d2615a to 50aaff9 Compare June 1, 2023 10:36

odunbar force-pushed the orad/timesteppers branch from b8f779c to 66397a5 Compare June 1, 2023 22:29

bors bot added a commit that referenced this pull request Jun 1, 2023

Try #285:

bc490c1

odunbar force-pushed the orad/timesteppers branch from 4ea2d81 to 4ea03c6 Compare June 2, 2023 00:48

bors bot added a commit that referenced this pull request Jun 2, 2023

Try #285:

94d6367

bors bot merged commit 03c50e2 into main Jun 2, 2023
11 checks passed

bors bot deleted the orad/timesteppers branch June 2, 2023 01:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Timesteppers for EKP, from ParameterEstimocean #285

Timesteppers for EKP, from ParameterEstimocean #285

odunbar commented May 9, 2023 •

edited

Loading

odunbar commented May 16, 2023

odunbar commented May 17, 2023

eviatarbach left a comment

eviatarbach May 25, 2023

odunbar May 31, 2023

odunbar May 31, 2023

odunbar commented May 31, 2023 •

edited

Loading

eviatarbach commented Jun 1, 2023

odunbar commented Jun 1, 2023

odunbar commented Jun 1, 2023

bors bot commented Jun 1, 2023

odunbar commented Jun 2, 2023

odunbar commented Jun 2, 2023

bors bot commented Jun 2, 2023

bors bot commented Jun 2, 2023

Timesteppers for EKP, from ParameterEstimocean #285

Timesteppers for EKP, from ParameterEstimocean #285

Conversation

odunbar commented May 9, 2023 • edited Loading

Purpose

Content

Also Changed

Example experiment - explained more clearly in docs

Individual experiment behaviour

Average convergence over 100 experiments

Conclusion?

odunbar commented May 16, 2023

odunbar commented May 17, 2023

eviatarbach left a comment

Choose a reason for hiding this comment

eviatarbach May 25, 2023

Choose a reason for hiding this comment

odunbar May 31, 2023

Choose a reason for hiding this comment

odunbar May 31, 2023

Choose a reason for hiding this comment

odunbar commented May 31, 2023 • edited Loading

eviatarbach commented Jun 1, 2023

odunbar commented Jun 1, 2023

odunbar commented Jun 1, 2023

bors bot commented Jun 1, 2023

try

odunbar commented Jun 2, 2023

odunbar commented Jun 2, 2023

bors bot commented Jun 2, 2023

try

bors bot commented Jun 2, 2023

odunbar commented May 9, 2023 •

edited

Loading

odunbar commented May 31, 2023 •

edited

Loading