Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default partition in Distributed #3496

Merged
merged 18 commits into from
Mar 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/AbstractOperations/conditional_operations.jl
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
using Oceananigans.Fields: OneField
using Oceananigans.Grids: architecture
using Oceananigans.Architectures: on_architecture
import Oceananigans.Architectures: on_architecture
import Oceananigans.Fields: condition_operand, conditional_length, set!, compute_at!, indices

# For conditional reductions such as mean(u * v, condition = u .> 0))
Expand Down
7 changes: 5 additions & 2 deletions src/DistributedComputations/distributed_architectures.jl
Original file line number Diff line number Diff line change
Expand Up @@ -204,16 +204,19 @@ Keyword arguments
if not for testing or developing. Change at your own risk!
"""
function Distributed(child_architecture = CPU();
communicator = MPI.COMM_WORLD,
communicator = nothing,
devices = nothing,
synchronized_communication = false,
partition = Partition(MPI.Comm_size(communicator)))
partition = nothing)

if !(MPI.Initialized())
@info "MPI has not been initialized, so we are calling MPI.Init()."
MPI.Init()
end

communicator = isnothing(communicator) ? MPI.COMM_WORLD : communicator
partition = isnothing(partition) ? Partition(MPI.Comm_size(communicator)) : partition

ranks = size(partition)
Rx, Ry, Rz = ranks
total_ranks = Rx * Ry * Rz
Expand Down
1 change: 1 addition & 0 deletions src/DistributedComputations/distributed_on_architecture.jl
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
using CUDA: CuArray
using OffsetArrays
import Oceananigans.Architectures: on_architecture

# We do not support switching from distributed and serial through `on_architecture`.
# We only support moving a type from CPU to GPU and the other way around
Expand Down
2 changes: 1 addition & 1 deletion src/DistributedComputations/partition_assemble.jl
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
using Oceananigans.Architectures: on_architecture
import Oceananigans.Architectures: on_architecture

all_reduce(op, val, arch::Distributed) =
MPI.Allreduce(val, op, arch.communicator)
Expand Down
1 change: 0 additions & 1 deletion src/ImmersedBoundaries/grid_fitted_bottom.jl
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ using OffsetArrays: OffsetArray
using Oceananigans.Utils: getnamewrapper
using Oceananigans.Grids: total_size
using Oceananigans.Fields: fill_halo_regions!
using Oceananigans.Architectures: on_architecture
using Oceananigans.BoundaryConditions: FBC
using Printf

Expand Down
1 change: 0 additions & 1 deletion src/ImmersedBoundaries/mask_immersed_field.jl
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
using KernelAbstractions: @kernel, @index
using Statistics
using Oceananigans.Architectures: architecture
using Oceananigans.Fields: location, ZReducedField, Field

instantiate(T::Type) = T()
Expand Down
1 change: 0 additions & 1 deletion src/ImmersedBoundaries/partial_cell_bottom.jl
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
using Oceananigans.Utils: prettysummary
using Oceananigans.Fields: fill_halo_regions!
using Oceananigans.Architectures: on_architecture
using Printf

#####
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
using Oceananigans.Utils: on_architecture
using Oceananigans.Grids: AbstractGrid

abstract type AbstractTimeDiscretization end
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
using Oceananigans.Architectures: architecture, on_architecture
using Oceananigans.Architectures: architecture
using Oceananigans.AbstractOperations: KernelFunctionOperation
using Oceananigans.BuoyancyModels: ∂z_b
using Oceananigans.Operators: ℑzᵃᵃᶜ
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
using Oceananigans.Architectures: architecture, on_architecture
using Oceananigans.Architectures: architecture
using Oceananigans.BuoyancyModels: ∂z_b
using Oceananigans.Operators
using Oceananigans.Grids: inactive_node
Expand Down