Forward Modeling Issues

[tjk9501]

Hello!
I run a 3D forward modeling program with Julia 1.6.6 and JUDI, when it began modeling of the last transducer, the following error appeared:

ERROR: LoadError: TaskFailedException

    nested task error: TaskFailedException
    
        nested task error: On worker 3:
        peer 10 didn't connect to 3 within 59.99996590614319 seconds
        Stacktrace:
          [1] error
            @ ./error.jl:33
          [2] wait_for_conn
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:194
          [3] check_worker_state
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:168
          [4] send_msg_
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/messages.jl:165
          [5] send_msg
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/messages.jl:122 [inlined]
          [6] #remotecall_fetch#143
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:392 [inlined]
          [7] remotecall_fetch
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:389
          [8] #remotecall_fetch#146
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:424 [inlined]
          [9] remotecall_fetch
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:424 [inlined]
         [10] local_reduce!
            @ ~/.julia/packages/JUDI/kyqM8/src/TimeModeling/Modeling/distributed.jl:192
         [11] #106
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278
         [12] run_work_thunk
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
         [13] macro expansion
            @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278 [inlined]
         [14] #105
            @ ./task.jl:411
        Stacktrace:
         [1] remotecall_fetch(::Function, ::Distributed.Worker, ::Future, ::Vararg{Future, N} where N; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
           @ Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:397
         [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Future, ::Vararg{Future, N} where N)
           @ Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:389
         [3] remotecall_fetch(::Function, ::Int64, ::Future, ::Vararg{Future, N} where N; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
           @ Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:424
         [4] remotecall_fetch
           @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:424 [inlined]
         [5] (::JUDI.var"#189#190"{Vector{Future}, Int64})()
           @ JUDI ./task.jl:411
    
    ...and 31 more exceptions.
    
    Stacktrace:
     [1] sync_end(c::Channel{Any})
       @ Base ./task.jl:369
     [2] macro expansion
       @ ./task.jl:388 [inlined]
     [3] reduce_level!(futures::Vector{Future}, nleaf::Int64)
       @ JUDI ~/.julia/packages/JUDI/kyqM8/src/TimeModeling/Modeling/distributed.jl:208
     [4] reduce!(futures::Vector{Future})
       @ JUDI ~/.julia/packages/JUDI/kyqM8/src/TimeModeling/Modeling/distributed.jl:242
     [5] (::JUDI.var"#186#188"{Vector{Any}, Int64, Vector{Future}})()
       @ JUDI ./task.jl:411
Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:369
 [2] macro expansion
   @ ./task.jl:388 [inlined]
 [3] task_distributed(::Function, ::WorkerPool, ::Model, ::Vararg{Any, N} where N; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ JUDI ~/.julia/packages/JUDI/kyqM8/src/TimeModeling/Modeling/distributed.jl:132
 [4] task_distributed(::Function, ::WorkerPool, ::Model, ::Vararg{Any, N} where N)
   @ JUDI ~/.julia/packages/JUDI/kyqM8/src/TimeModeling/Modeling/distributed.jl:125
 [5] time_modeling(::Model, ::Vararg{Any, N} where N; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ JUDI ~/.julia/packages/JUDI/kyqM8/src/TimeModeling/Modeling/distributed.jl:4
 [6] time_modeling
   @ ~/.julia/packages/JUDI/kyqM8/src/TimeModeling/Modeling/distributed.jl:4 [inlined]
 [7] (::JUDI.var"#220#222"{Options, Info, Model, GeometryIC{Float32}, GeometryIC{Float32}, UnitRange{Int64}})(src::judiVector{Float32, Matrix{Float32}})
   @ JUDI ~/.julia/packages/JUDI/kyqM8/src/TimeModeling/LinearOperators/judiPDEfull.jl:52
 [8] *(A::judiPDEfull{Float32, Float32}, v::judiVector{Float32, Matrix{Float32}})
   @ JUDI ~/.julia/packages/JUDI/kyqM8/src/TimeModeling/LinearOperators/judiPDEfull.jl:104
 [9] top-level scope
   @ /WORK/Labdata/DEVITO_DATA/forward_FWI.jl:83
in expression starting at /WORK/Labdata/DEVITO_DATA/forward_FWI.jl:23

It worthwhile to note that when I run the 2D code, no error was found, and I ran the same 3D code on a workstation with julia 1.5 and JUDI, no error was found.
PS: in the 2D cases, when I finished the forward modeling, Julia will say remove process 8 failed sometimes. In that case, I use 8 process to run forward modeling in parallel with the following code:

using Distributed; addprocs(16)

I don't know if this is related with the above issue.
Hope anyone can help to point out the problems, thanks!

[mloubout]

Can you try using Distributed; addprocs(16;lazy=false) ? Since julia 1.6, Distributed handles workers initialization a bit differently and can lead to weird behavior.

[tjk9501]

I have tried your method, and the problem is solved! Thank you very much!