scikit-fallback is a scikit-learn-compatible Python package for machine learning with a reject option.

👩‍💻 Usage

To allow your probabilistic pipeline to fallback—i.e., abstain from predictions—you can wrap it with a skfb rejector. Training a rejector means both fitting your model and learning to accept or reject predictions. Evaluation of a rejector depends on fallback mode (inference with or without fallback labels) and measures the ability of the rejector to both accept correct predictions and reject ambiguous ones.

For example, skfb.estimators.ThresholdFallbackClassifierCV fits the base estimator and then finds the best confidence threshold via cross-validation. If fallback_mode == "store", then the rejector returns skfb.core.array.FBNDArray of predictions and a sparse fallback-mask property, which lets us summarize the accuracy of both predictions and rejections.

from skfb.estimators import ThresholdFallbackClassifierCV
from sklearn.linear_model import LogisticRegressionCV

rejector = ThresholdFallbackClassifierCV(
    LogisticRegressionCV(cv=4, random_state=0),
    thresholds=10,
    ambiguity_threshold=0.05,
    cv=5,
    fallback_label=-1,
    fallback_mode="store",
)
rejector.fit(X_train, y_train)  # Train base estimator and learn best threshold
rejector.score(X_test, y_test)  # Compute acceptance-correctness accuracy score

And for more motivation and information on usage, please visit our documentation and refer to the Medium series on machine learning with a reject option.

🏗 Installation

scikit-fallback requires:

• Python (>=3.9,<3.13)
• scikit-learn (>=1.0)
• numpy
• scipy
• matplotlib (>=3.0) (optional)

and along with the requirements can be installed via pip :

pip install -U scikit-fallback

📚 Examples

See the examples/ directory for various applications of fallback estimators and scorers to scikit-learn-compatible pipelines.

🔗 References

1. Hendrickx, K., Perini, L., Van der Plas, D. et al. Machine learning with a reject option: a survey. Mach Learn 113, 3073–3110 (2024). https://doi.org/10.1007/s10994-024-06534-x