You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, I tried to train eagle on 4x A6000 and found that the training process could crash down sometimes. The error seems to be about multi-gpu connection timeout (>30min).
I suspected it's because of crashing of one gpu, however all gpus works properly until the whole program is down (according to the temperature), it's strange to throw a timeout.
Although I think it's my own problem, I would appreciate it if you have any experience or advice to address this problem. Thank you!
The text was updated successfully, but these errors were encountered:
Hello, I tried to train eagle on 4x A6000 and found that the training process could crash down sometimes. The error seems to be about multi-gpu connection timeout (>30min).
![image](https://private-user-images.githubusercontent.com/56115919/348482432-3783b6a3-f273-4e53-bd88-5b336b3dbba1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU3NTU2MjIsIm5iZiI6MTcyNTc1NTMyMiwicGF0aCI6Ii81NjExNTkxOS8zNDg0ODI0MzItMzc4M2I2YTMtZjI3My00ZTUzLWJkODgtNWIzMzZiM2RiYmExLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA4VDAwMjg0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQ3ZWZmMTgwOTM0MGVmMjkzMjQzMjk0M2M3YTE1YWU4NjczNWEwMmE2NGI5ZWYwZTllYWI4Y2M3NzBkZjVhNTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.9dQr58VXQxjwA7y5jkt6iAsTvsQ6opUU9A24Wp5uaV0)
I suspected it's because of crashing of one gpu, however all gpus works properly until the whole program is down (according to the temperature), it's strange to throw a timeout.
Although I think it's my own problem, I would appreciate it if you have any experience or advice to address this problem. Thank you!
The text was updated successfully, but these errors were encountered: