You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
How to solve the following problems when I train? thanks!
(unept) lhj@zhouwei-SYS-4028GR-TR2:~/unified-ept$ CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" python -m torch.distributed.launch --nproc_per_node=8 --master_port=29500 train.py --launcher pytorch --config /home/lhj/unified-ept/configs/res50_unept_ade20k.py
/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects --local_rank argument to be set, please
change it to read from os.environ['LOCAL_RANK'] instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions
FutureWarning,
WARNING:torch.distributed.run:
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 3704) of binary: /home/lhj/.conda/envs/unept/bin/python
Traceback (most recent call last):
File "/home/lhj/.conda/envs/unept/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/lhj/.conda/envs/unept/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/run.py", line 713, in run
)(*cmd_args)
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
How to solve the following problems when I train? thanks!
(unept) lhj@zhouwei-SYS-4028GR-TR2:~/unified-ept$ CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" python -m torch.distributed.launch --nproc_per_node=8 --master_port=29500 train.py --launcher pytorch --config /home/lhj/unified-ept/configs/res50_unept_ade20k.py
/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects
--local_rank
argument to be set, pleasechange it to read from
os.environ['LOCAL_RANK']
instead. Seehttps://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions
FutureWarning,
WARNING:torch.distributed.run:
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
Traceback (most recent call last):
File "train.py", line 21, in
from modified_mmseg.apis import set_random_seed, train_segmentor
File "/home/lhj/unified-ept/modified_mmseg/init.py", line 27, in
f'MMCV=={mmcv.version} is used but incompatible. '
AssertionError: MMCV==1.3.18 is used but incompatible. Please install mmcv>=[1, 1, 4], <=[1, 3, 0].
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 3704) of binary: /home/lhj/.conda/envs/unept/bin/python
Traceback (most recent call last):
File "/home/lhj/.conda/envs/unept/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/lhj/.conda/envs/unept/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/run.py", line 713, in run
)(*cmd_args)
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/lhj/.local/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train.py FAILED
Failures:
[1]:
time : 2021-12-17_17:25:25
host : zhouwei-SYS-4028GR-TR2
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 3705)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2021-12-17_17:25:25
host : zhouwei-SYS-4028GR-TR2
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 3706)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2021-12-17_17:25:25
host : zhouwei-SYS-4028GR-TR2
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 3707)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
time : 2021-12-17_17:25:25
host : zhouwei-SYS-4028GR-TR2
rank : 4 (local_rank: 4)
exitcode : 1 (pid: 3708)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
time : 2021-12-17_17:25:25
host : zhouwei-SYS-4028GR-TR2
rank : 5 (local_rank: 5)
exitcode : 1 (pid: 3709)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
time : 2021-12-17_17:25:25
host : zhouwei-SYS-4028GR-TR2
rank : 6 (local_rank: 6)
exitcode : 1 (pid: 3710)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
time : 2021-12-17_17:25:25
host : zhouwei-SYS-4028GR-TR2
rank : 7 (local_rank: 7)
exitcode : 1 (pid: 3711)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
Root Cause (first observed failure):
[0]:
time : 2021-12-17_17:25:25
host : zhouwei-SYS-4028GR-TR2
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 3704)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
The text was updated successfully, but these errors were encountered: