You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The detailed performance evaluation / comparison plots of the different model architectures in the readme make it seem like the results can be used to select the best model architecture.
Especially beginners in the field of data science might be fooled into selecting their preferred architecture based on these results, perhaps even in a completely different use case solely based on this information.
In the current version the training script does include only training with one selection of hyperparameters per architecture.
The performance especially of deep ML architectures are known to be heavily reliant on the selection of hyperparameters.
I want to point out, that without extensive(!) hyperparameter optimsation one should not draw any conclusions about the potential performance of an ML architecture. Hence the best performing model architecture cannot be selected by training with only one or few hyperparameter settings.
I suggest pointing this out in the readme and ideally referencing to hyperparameter tuning packages like https://optuna.org/ or similar.
The text was updated successfully, but these errors were encountered:
jannikmi
changed the title
performance comparison without hyperparameter tuning
misleading performance comparison without hyperparameter tuning
Feb 1, 2024
The detailed performance evaluation / comparison plots of the different model architectures in the readme make it seem like the results can be used to select the best model architecture.
Especially beginners in the field of data science might be fooled into selecting their preferred architecture based on these results, perhaps even in a completely different use case solely based on this information.
In the current version the training script does include only training with one selection of hyperparameters per architecture.
The performance especially of deep ML architectures are known to be heavily reliant on the selection of hyperparameters.
I want to point out, that without extensive(!) hyperparameter optimsation one should not draw any conclusions about the potential performance of an ML architecture. Hence the best performing model architecture cannot be selected by training with only one or few hyperparameter settings.
I suggest pointing this out in the readme and ideally referencing to hyperparameter tuning packages like https://optuna.org/ or similar.
The text was updated successfully, but these errors were encountered: