Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation error when tring to reproduce wmt22 results #136

Open
SefaZeng opened this issue May 22, 2023 · 1 comment
Open

Segmentation error when tring to reproduce wmt22 results #136

SefaZeng opened this issue May 22, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@SefaZeng
Copy link

🐛 Bug

To Reproduce

pip install unbabel-comet
comet-score -s ../mt-metrics-eval-v2/wmt22/sources/en-zh.txt -t ../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt -r ../mt-metrics-eval-v2/wmt22/references/en-zh.refA.txt > log.comet
Global seed set to 1
Fetching 5 files: 100%|█████████████████████████| 5/5 [00:00<00:00, 88487.43it/s]
Lightning automatically upgraded your loaded checkpoint from v1.8.3.post1 to v1.9.5. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint --file ../../../../../root/.cache/huggingface/hub/models--Unbabel--wmt22-comet-da/snapshots/371e9839ca4e213dde891b066cf3080f75ec7e72/checkpoints/model.ckpt`
Encoder model frozen.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Predicting DataLoader 0: 100%|████████████████████| 128/128 [01:00<00:00,  2.10it/s]
[1]    13312 segmentation fault  comet-score -s ../mt-metrics-eval-v2/wmt22/sources/en-zh.txt -t  -r  > 

And the result in log.comet is like:

../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 0   score: 0.8275                                                                                                                                                                    
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 1   score: 0.8833
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2   score: 0.7753
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 3   score: 0.9103
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 4   score: 0.8103
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 5   score: 0.9792
...
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2033    score: 0.9494
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2034    score: 0.9332
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2035    score: 0.9397
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2036    score: 0.9048
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  score: 0.8622 

Expected behaviour

Output the system score for candidates.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

OS: [e.g. iOS, Linux, Win]
Packaging [e.g. pip, conda] pip/py39
Version [e.g. 0.5.2.1] 2.0.1

Additional context

@SefaZeng SefaZeng added the bug Something isn't working label May 22, 2023
@SefaZeng
Copy link
Author

And the results for wm22 en-zh HuaweiTSC.txt do not match that in the wmt22 released COMET-22-refA.sys.score.
The score in the file is 0.47647532625047895.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant