Make finetune actually fine tune the model
#25
Conversation
| pub struct LearningRateSchedules { | ||
| pub classifier: PlateauLearningRate<ExponentialDecay>, | ||
| pub encoder: PlateauLearningRate<ExponentialDecay>, | ||
| } |
There was a problem hiding this comment.
Why is it not possible to use any other schedules?
There was a problem hiding this comment.
Good question! We had the same issue in sticker1. The problem is that we have not found a clean way yet with clap to have/show options dependent on some other option. All these LR schedules have different hyperparameters and we'd want to be able to say something like --lr-schedule expdecay and only have the options relevant to exponential decay available. You can hack your way around it by not permitting other options, but the UI gets pretty ugly, because you cannot show options dependent on another option in groups.
There was a problem hiding this comment.
💡 Thanks for clarifying!
There was a problem hiding this comment.
Maybe it could move to a config file at some point. OTOH, patience + exp decay is reasonable and one can hack around the patience by using 99999 as decay patience.
relevant issue: stickeritis/sticker#147
There was a problem hiding this comment.
Indeed, it seems like a reasonable default and you can even disable one of the two by setting the right hyper parameters ;).
No description provided.