-
Notifications
You must be signed in to change notification settings - Fork 178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds classifier chains as a generic multi-label classifier #149
Conversation
…exception when used with HashingTrainer as it's implemented in an incompatible way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I ran code coverage analysis for the unit tests and the covereage is really good overall with 76.7% with no concerning gaps
ClassifierChainTrainer lines 222-227 are nearly identical to ClassifierChainModel lines 99-106. It looks like you could consolidate them into a single(static) method?
It seems like the method MultiLabelVotingCombiner.combine could delegate to the other combine method with equal weights.
ClassifierChainModel has just one inner trainer. Seems like it might be useful to have different hyperparameters and/or different trainers for each label.
I agree there's a bit of duplication there in the building of new features, but I'd still need the if statement in WRT to WRT different inner trainers, yes that might be interesting to look at. I was also considering doing bootstrap samples for the datasets which would increase variability and thus improve performance. This PR is a straight implementation of the referenced paper, we can look at doing different extensions later. |
Description
Adds
ClassifierChainTrainer/Model
, andCCEnsembleTrainer
which train classifier chains and an ensemble of classifier chains respectively. The ensemble required a multi-label voting combiner, which allows the use of bagging and RF in multi-label problems in Tribuo in addition to supporting classifier chain ensembles.It also adds the jaccard score as a multi-label evaluation metric as it was useful while testing out this implementation.
As part of the work there is a small performance improvement to
IndependentMultiLabelTrainer
so it doesn't recreate the dataset for each label, and that trainer now correctly throws an exception if it is used insideHashingTrainer
as the hashing mechanism interferes with the multi-label conversion mechanism. It is future work to make the two compatible again (as well asClassifierChainTrainer
which is also incompatible withHashingTrainer
).Finally this adds two methods to
MutableDataset
which allow the regeneration of the feature and output domains on demand. We'll look at using this mechanism to fix issues withDatasetView
which also needs the regeneration functionality later.Motivation
Classifier chains allow the incorporation of label dependence and correlation into multi-label predictions without inducing much more overhead than the current independent prediction model used as a baseline and as implemented in the
LinearSGDTrainer
. This should improve performance where the labels are not independent of each other (which is probably true in most multi-label problems).Paper reference
Classifier chains and ensembles thereof are introduced in:
Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2011). Classifier chains for multi-label classification. Machine learning, 85(3), 333-359.
There's a review on the topic published this year, and we may investigate some of the extensions described herein if the technique proves useful.
Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2021). Classifier chains: a review and perspectives. Journal of Artificial Intelligence Research, 70, 683-718.