Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

misleading performance comparison without hyperparameter tuning #10

Open
jannikmi opened this issue Feb 1, 2024 · 0 comments
Open

misleading performance comparison without hyperparameter tuning #10

jannikmi opened this issue Feb 1, 2024 · 0 comments

Comments

@jannikmi
Copy link

jannikmi commented Feb 1, 2024

The detailed performance evaluation / comparison plots of the different model architectures in the readme make it seem like the results can be used to select the best model architecture.
Especially beginners in the field of data science might be fooled into selecting their preferred architecture based on these results, perhaps even in a completely different use case solely based on this information.

In the current version the training script does include only training with one selection of hyperparameters per architecture.

The performance especially of deep ML architectures are known to be heavily reliant on the selection of hyperparameters.
I want to point out, that without extensive(!) hyperparameter optimsation one should not draw any conclusions about the potential performance of an ML architecture. Hence the best performing model architecture cannot be selected by training with only one or few hyperparameter settings.

I suggest pointing this out in the readme and ideally referencing to hyperparameter tuning packages like https://optuna.org/ or similar.

@jannikmi jannikmi changed the title performance comparison without hyperparameter tuning misleading performance comparison without hyperparameter tuning Feb 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant