347 refactor validation category

[nmaarnio]

New tools added and old ones modified. Here is a brief list of the changes:

• Removed AUC tool and related tests and docs (the reason is that AUC is calculated directly from Sklearn when a curve is calculated)
• Moved get_pa_intersection to plot_prediction_area_curves.py module, since it is needed only there
• Added module classifier_probability_evaluation.py that has the following tools:
  • summarize_probability_metrics to generate a report dictionary from probability array
  • plot_roc_curve
  • plot_det_curve
  • plot_precision_recall_curve
  • plot_calibration_curve
  • plot_predicted_probability_distribution
• Added module classifier_label_evaluation.py and the following tools
  • summarize_label_metrics_binary
  • confusion_matrix (just direct import from Sklearn, no wrapper)
• Added notebook called testing_classifier_evaluation.ipynb to demonstrate the two new modules above
• Separated scoring predictions into own module called scoring.py
• Small modification to plot_confusion_matrix
• Separated predicting with ML model functions into their own module called machine_learning_predicti.py
• Separated predict tool for classifier and regressor models, so two new functions:
  • predict_classifier, this tool now can return the probability array based on parameter. NOTE: Testing for Keras models still not adequate enough
  • predict_regressor

TODO:

• Add and update affected CLI functions
• Rename the validation folder to evaluation as the last thing

[nmaarnio]

@msmiyels , do you have time to review this PR in the next couple of days? Ended up doing quite a lot of modeling related stuff, so thought it could be good to involve you. If not, I can ask someone else

[msmiyels]

Hey Nico,

nice work overall! 🐱‍👤 Left a view comments here and there, but nothing criticial.

I'm a bit torned whether it is better to have one classification_evaluation rather than separate modules ...label_evaluation and ...probability_evaluation., but it will not make any difference at least for the GUI.

If you have any questions, let me know 📯

[msmiyels]

import sys
sys.path.append('..')

is missing to import the toolkit from the notebooks folder. Please also check for the other provided notebooks, I did not investigate all of them 😉

[nmaarnio]

I think these lines are only needed if files need to be opened in the notebook, which is currently not the case with testing_classifier_evaluation.ipynb. Also, the other notebooks should be pretty much as they were before, just modified some cells slightly according to the general ML tool modifications.

[nmaarnio]

Hey @msmiyels and thanks for the review!

I made some changes according to your review and carried out the renaming

[nmaarnio]

Do you think this could me merged @msmiyels ?

[msmiyels]

Do you think this could me merged @msmiyels ?

Hi Niko,

yep, think this PR can be merged 👍