FSDP: Mistral(mathstral) sbatch file - MISTRAL MODEL SUPPORT #385
Conversation
…g-mixtral.sbatch renaming to mixtral because the args here/ referenced in model _type flag are specific to mixtral models
contains updated sbatch for mistral models. Before these changes, only mixtral was supported. This PR is part of other PRs that Armando and I will incorporate. This has been tested on hyperpod already.
| --val_batch_size=1 \ | ||
| --max_steps=5000 \ | ||
| --seed=42 \ | ||
| # torch.get_default_dtype() returns fp32 |
There was a problem hiding this comment.
hey Pierre, would you like us to just remove just the comment with fp32 or the other lines you've highlighted as well?
There was a problem hiding this comment.
comment only :) The thing is that either you keep it and document why it's there and commented or remove. typically these are left-overs of coding or left for a good reason but in which case it needs a bit more info
There was a problem hiding this comment.
Understood! Thanks for the quick response
There was a problem hiding this comment.
Have removed the comment
Hey @perifaws, have updated the readme to include mathstral and the output when launching training, please let me know if there are any other changes required. Thanks! |
Adding support for Mistral models when using FSDP on Hyperpod. Previously only mixtral was supported. This sbatch file is part of a hyperpod blog on training mathstral. This PR is part of other PRs that Armando and I will incorporate as part of the mathstral sbatch job already tested on Hyperpod.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.