python -m eagle.ge_data.allocation --outdir [path of data]

[zpz915]

when i generate traindata python -m eagle.ge_data.allocation --outdir [path of data],but i meet the error

[zpz915]

Traceback (most recent call last):
File "/root/autodl-tmp/EAGLE/eagle/ge_data/ge_data_all_vicuna.py", line 148, in
ds = build_dataset_rank(bigtokenizer)
File "/root/autodl-tmp/EAGLE/eagle/ge_data/ge_data_all_vicuna.py", line 130, in build_dataset_rank
ds1 = ds1.map(
File "/root/miniconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 602, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 567, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3253, in map
for rank, done, content in iflatmap_unordered(
File "/root/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
[async_result.get(timeout=0.05) for async_result in async_results]
File "/root/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 718, in
[async_result.get(timeout=0.05) for async_result in async_results]
File "/root/miniconda3/lib/python3.8/site-packages/multiprocess/pool.py", line 771, in get
raise self._value
IndexError: list index out of range

[Liyuhui-12]

There might be an issue with parallel processing. You can try using num_proc=1 to see if it helps.