Class to create metrics from predict

[ediboc]

Make a Class to create metrics like f1, confusion matrix, ROC AUC from predict results

[pablet]

Finally, instead of a class of metrics, create an evaluate method. With the VisionClassifier instantiated, it is possible to obtain the metric indicated by the user. Example:

vc = VisionClassifier(
   model_name="vit_huge_patch14_224_in21k",
   num_classes=2,
   task="single_label",
)

vc.evaluate(data, "accuracy")

[jorloplaz]

Which metrics would you support for .evaluate()? Note that as in sklearn, for a classifier we could have metrics that would return a numeric array based on pure predictions from predict() (e.g., accuracy), but we could also have probabilistic metrics that would require a predict_proba() beforehand.

And things become even trickier for multiclass problems...

[pablet]

Right now we are going to standardise to metrics that accept y_true and y_pred as pure predictions 1D array. However, we are open to suggestions, what metrics that require probabilities would you like to see?