Test on latest Julia version (~1.8)

[ilopezgp]

PULL REQUEST

Purpose and Content

Update package to be compatible with Julia v1.8. Compatibility does not seem to be automatic, since our tests are not passing.

[odunbar]

Note - the test Test / Julia 1 - ubuntu-latest - x64 failed previously with a SingularException.

On re-running tests (with no change) this now passes, along with macOS-latest,

but now windows-latest fails with the same SingularException.

So there may be some instability in the linear solvers from LinearAlgebra, as I don't think the problem is really singular. I believe someone has recommended linearsolvers.jl in the past, maybe this is a way forward if these tests still give us issues

[ilopezgp]

Note - the test Test / Julia 1 - ubuntu-latest - x64 failed previously with a SingularException.

On re-running tests (with no change) this now passes, along with macOS-latest,

but now windows-latest fails with the same SingularException.

So there may be some instability in the linear solvers from LinearAlgebra, as I don't think the problem is really singular. I believe someone has recommended linearsolvers.jl in the past, maybe this is a way forward if these tests still give us issues

Thoughts on this issue, @charleskawczynski ? We are getting SingularExceptions in linear systems that are not singular by Julia <1.8 standards.

[charleskawczynski]

Thoughts on this issue, @charleskawczynski ? We are getting SingularExceptions in linear systems that are not singular by Julia <1.8 standards.

I'm not sure there's enough information to debug. What's the stack trace? Is this an error in TC or in CEDMF? We haven't started testing TC with 1.8 yet, so it might make sense to do that first before upgrading this repo.

I think we're seeing similar periodic windows errors in TC-- which, IIRC, is some sort of memory mismanagement, and I'm not exactly sure what's going on there. I've not been able to reproduce locally

[ilopezgp]

Thoughts on this issue, @charleskawczynski ? We are getting SingularExceptions in linear systems that are not singular by Julia <1.8 standards.

I'm not sure there's enough information to debug. What's the stack trace? Is this an error in TC or in CEDMF? We haven't started testing TC with 1.8 yet, so it might make sense to do that first before upgrading this repo.

I think we're seeing similar periodic windows errors in TC-- which, IIRC, is some sort of memory mismanagement, and I'm not exactly sure what's going on there. I've not been able to reproduce locally

This error is localized to EKP.jl, it comes up in a unit test independent of TC or CEDMF. Basically, the error comes from the operation:

result = (cov_mat) \ (vector)

which throws a SingularException (associated with cov_mat) that we did not see in Julia 1.7.3.

[charleskawczynski]

Can you make a reproducer of the values used?

[ilopezgp]

bors try

[bors]

try

Build failed:

• test_success

[odunbar]

update:

Looking at the dumped matrices we find that \ works on 1.7.2 and not 1.8.0 because the matrices are near ill-conditioned. Operations cond(A), and det(A) give different answers too.

One option

After suggestion of @charleskawczynski , replacing

EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl

Line 104 in 6e59c64

tmp = FT.((cov_gg + obs_noise_cov) \ (y - g))

with

LHS = Matrix{BigFloat}(cov_gg + obs_noise_cov)
RHS = Matrix{BigFloat}(y - g)
tmp = FT.(LHS \ RHS )

leads to tests passing in 1.8.0

Of course this is not ideal, as it is likely a far less optimized solve method when using arbitrary precision.

[ilopezgp]

We could implement that in a try catch. As you said, I think that will be less efficient in general settings.

[ilopezgp]

We could implement that in a try catch. As you said, I think that will be less efficient in general settings.

Done and solved! I will merge this.

[ilopezgp]

bors r+

[bors]

Build succeeded:

• docs-build
• format
• test_success