Support file split in NetCDFOutputWriter when max_filesize is exceeded

[josuemtzmo]

This PR aims to allow the user to split netCDF files into smaller files when the filesize exceeds the max_filesize. This addresses #2967

[josuemtzmo]

Although the new test_netcdf_file_splitting is working, I'm currently having issues in with the test_netcdf_time_averaging, where this test fails.

Oceananigans.jl/test/test_netcdf_output_writer.jl

Line 683 in 3bb62a6

@test all(isapprox.(ds["c2"][:, n+1], c̄2(averaging_times), rtol=rtol))

I've tested the PR & the main upstream branch in my computer and I have the same error:

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined]
 [2] test_netcdf_time_averaging(arch::CPU)
   @ Main ~/github/Oceananigans.jl/test/test_netcdf_output_writer.jl:727
 [3] macro expansion
   @ ~/github/Oceananigans.jl/test/test_netcdf_output_writer.jl:889 [inlined]
 [4] macro expansion
   @ ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [5] top-level scope
   @ ~/github/Oceananigans.jl/test/test_netcdf_output_writer.jl:880
NetCDF output writer [CPU]: Test Failed at /Users/jmtzmo/github/Oceananigans.jl/test/test_netcdf_output_writer.jl:727
  Expression: all(isapprox.((ds["c2"])[:, n + 1], c̄2(averaging_times), rtol = rtol))

[navidcy]

Although the new test_netcdf_file_splitting is working, I'm currently having issues in with the test_netcdf_time_averaging, where this test fails.

Oceananigans.jl/test/test_netcdf_output_writer.jl

Line 683 in 3bb62a6

@test all(isapprox.(ds["c2"][:, n+1], c̄2(averaging_times), rtol=rtol))

I've tested the PR & the main upstream branch in my computer and I have the same error:

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined]
 [2] test_netcdf_time_averaging(arch::CPU)
   @ Main ~/github/Oceananigans.jl/test/test_netcdf_output_writer.jl:727
 [3] macro expansion
   @ ~/github/Oceananigans.jl/test/test_netcdf_output_writer.jl:889 [inlined]
 [4] macro expansion
   @ ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [5] top-level scope
   @ ~/github/Oceananigans.jl/test/test_netcdf_output_writer.jl:880
NetCDF output writer [CPU]: Test Failed at /Users/jmtzmo/github/Oceananigans.jl/test/test_netcdf_output_writer.jl:727
  Expression: all(isapprox.((ds["c2"])[:, n + 1], c̄2(averaging_times), rtol = rtol))

me and @josuemtzmo sorted this out; things should be OK now!

[glwagner]

Cool! Very nice!

[navidcy]

@Sbozzolo, why is the distributed CI not starting here? Because this PR is from a fork?

[Sbozzolo]

The job is not showing up in the buildkite pipeline at all...

It should build PRs, can you try commiting again to see if it picked up?

[glwagner]

We also had this problem on Thermodynamics and it was fixed by re-opening the PR from the repo rather than a fork. There is a security issue because you may not want to run buildkite jobs from random people (despite that we definitely do want to run buildkite for PRs from @josuemtzmo <3).

[glwagner]

@Sbozzolo is this enabled? (This is in the pipeline settings on buildkite)

[Sbozzolo]

Now it is.

(But I can turn it off if you prefer)

[navidcy]

hm... OK -- thanks all.

Not sure whether we want that setting or not... if we keep it disable it implies that every PR from a fork will never be merged?)

In the meantime, @josuemtzmo any chance you can re-open this as a branch of the main repo? we have the convention that branches are named with initials/my-branch-name (dashes), e.g., ncc/the-best-new-feature. I invited you as a collaborator in the repo.

[josuemtzmo]

Ok, I will push to the upstream, I'm closing this PR in preference to that other one.

[josuemtzmo]

This PR was replaced by #3512

[glwagner]

Now it is.

(But I can turn it off if you prefer)

Well it is enabled for Oceananigans too. So I guess there is a different reason that forks are problematic.