Parallel and Serial Split Explicit Free Surface #2888
Conversation
simone-silvestri
changed the title
…into ss/split_explicit
…into ss/split_explicit
|
I should have addressed all the changes except the |
sounds good |
|
|
||
| # Check that the size of a bottom field are | ||
| # consistent with the size of the field | ||
| if any(size(ib.mask) .!= mask_size) |
There was a problem hiding this comment.
This needs a comment at the least
| Nx, Ny, Nz = size(grid) | ||
| Hx, Hy, Nz = halo_size(grid) | ||
|
|
||
| mask_size = (Nx, Ny, Nz) .+ 2 .* (Hx, Hy, Hz) |
There was a problem hiding this comment.
Is this the same as total_size(grid)?
There was a problem hiding this comment.
I think it's important to use the functions here. Remember you want to change the size of a Center, Bounded field. When you do that, the above code is wrong.
There was a problem hiding this comment.
I think you want total_size((Center, Center, Nothing), grid) via
Oceananigans.jl/src/Grids/grid_utils.jl
Line 84 in e394bf7
This PR improves the currently implemented split explicit surface solver in serial mode
(partially using the algorithm in Shchepetkin & McWilliams, Ocean Modelling 9, 2005 albeit with a linear free surface) and implements single-node parallel (
MultiRegion), and multi-node parallel (Distributed) split explicit free surfaceIn addition, quite some improvements have been made to the distributed module to allow a distributed
LatitudeLongituteGridandIBGand various comments in theMultiRegionmoduleEdit: MPI does not exploit CUDA-aware message passing for CuArray views, so this PR also implements buffered Halo communication for distributed models, unifying a bit of the code for Distributed and MultiRegion. This is a fundamental step to achieve the goals of the next PR which will deal with heterogenous distributed - shared models (i.e. a MultiRegionGrid of a DistributedGrid)
Edit Edit: apparently CUDA-aware MPI allows passing views of CuArrays. Buffers are still implemented for those architectures where CUDA-aware MPI is not available
Edit Edit Edit: apparently for Oceananigans that sends strided memory, buffers are crucial because
(from openmpi-link)
Edit: This PR adds the possibility of distributing along a
Boundeddirection, and the possibility of having correct 2D parallelization with Coriolis. Previously it was not possible because corners were not communicated correctlyDoesn't have to be merged now.
(Validation cases are not necessary and will be deleted prior to merging)