Add failsafe EKI with sample_succ_gauss handler.

[ilopezgp]

• Adds a failure handler struct to all EnsembleKalmanProcess objects.
• Implements the IgnoreFailures failure handler for all EnsembleKalmanProcesses, which maintains the former behavior of the algorithms (i.e, do nothing upon failure).
• Implements the SampleSuccGauss failure handler for the Inversion process, which advances the failed ensemble by sampling from an empirical gaussian formed by the successful ensemble.
• Failure may be defined by the user by passing the indices of failed particles to update_ensemble!. If nothing is passed, failures are taken to be those forward model evaluations that contain NaNs.
• Adds tests of new failure handlers.

In addition:

• Update equations are encapsulated outside of update_ensemble!, and into functions ..._update.
• Transposes in the Sampler are encapsulated to the failsafe_update call. This allows to work with data as columns in the Sampler's version of update_ensemble!.

[ilopezgp]

This works for me locally, any idea what the error means? @charleskawczynski @jakebolewski

[charleskawczynski]

I'm getting a seg fault, I can try taking a closer look tomorrow

[ilopezgp]

I'm getting a seg fault, I can try taking a closer look tomorrow

Thanks! That would be helpful. Are you getting seg fault locally? All the tests run for me locally (MacOS), but fail here

[charleskawczynski]

Are you getting seg fault locally?

Yep

[charleskawczynski]

I think this is an upstream issue. I'm still not sure from where, but it might be productive to try using more recent versions to confirm if that's the case.

[ilopezgp]

I think this is an upstream issue. I'm still not sure from where, but it might be productive to try using more recent versions to confirm if that's the case.

Upstream from what packages? We should be up to date with all packages since the CompatHelper has not recommended any new PR.

[charleskawczynski]

Upstream from what packages? We should be up to date with all packages since the CompatHelper has not recommended any new PR.

Our compat entries allow for the latest version of SCS.jl, but we're not using the latest version of SCS.jl (the resolver gives us 0.9.something).

[ilopezgp]

Upstream from what packages? We should be up to date with all packages since the CompatHelper has not recommended any new PR.

Our compat entries allow for the latest version of SCS.jl, but we're not using the latest version of SCS.jl (the resolver gives us 0.9.something).

Thanks @charleskawczynski !! I think we merged a compat change that did not test the changes from SCS. I tested SCS 0.9 and it leads to segfaults. SCS 1.0 cannot be used yet because of compat issues with Convex.jl, although I think they will be updating that package to v0.15 very soon. Version 0.8 works.

[odunbar]

Thanks this looks great! i think all my comments are now addressed.

For the sake of codecov - could you add something to test the warnings that are thrown when 50% fail, or singular covariance? I think you could maybe see if the following are nonempty:
@test_logs(:warn,) update_ensemble(...) or
@test_logs(:info,) update_ensemble(...)

Then I'm happy to merge

[ilopezgp]

Thanks this looks great! i think all my comments are now addressed.

For the sake of codecov - could you add something to test the warnings that are thrown when 50% fail, or singular covariance? I think you could maybe see if the following are nonempty: @test_logs(:warn,) update_ensemble(...) or @test_logs(:info,) update_ensemble(...)

Then I'm happy to merge

Sounds good. I will add these as unit tests of the respective functions to be a bit cleaner.

[ilopezgp]

bors r+

[bors]

Build succeeded:

• docs-build
• format
• test_success