[FR] FiftyOne Support for Multi-Label Classifications with Label Studio Integration

[lkaneda]

Proposal Summary

Help us improve the Label Studio integration!

With the Label Studio integration in FiftyOne, add support for multi-label classification by adding support for the Classifications label type.

Motivation

Label Studio supports multi-label classifications on their platform. While FiftyOne can support multi-label classification through the Classifications type, the Label Studio integration does not currently support it. Adding this would provide a richer integration experience between FiftyOne and Label Studio.

What areas of FiftyOne does this feature affect?

• App: FiftyOne application
• Core: Core fiftyone Python library
• Server: FiftyOne server

Details

Currently, the Label Studio integration supports the Classification type. In order to support multi-label classifications, the integration will need to support the Classifications type.

You can learn more about the Label Studio integration in FiftyOne here.

Looking for Help

FiftyOne is an app centered around our community. To continuously improve the open source solution, we're searching for contributors to help improve (and contribute new!) features. This is one of those that we hope to get community support on. If you'd like to help us out, feel free to pick this up and ask us questions here or in the slack community!

In exchange for your help, you'll get some pretty sweet FiftyOne swag!

[tataganesh]

Hey @lkaneda,
Are you still looking for contributors for this issue?

[brimoor]

Hi @tataganesh 👋

Yes, a contribution for this would be amazing!

[tataganesh]

Great, thanks @brimoor ! Would you have any pointers to get started with the support for the Classifications type?

[brimoor]

@tataganesh absolutely!

1. The labelstudio integration is implemented here: https://github.com/voxel51/fiftyone/blob/develop/fiftyone/utils/labelstudio.py
2. Documentation for using the integration: https://docs.voxel51.com/integrations/labelstudio.html
3. Documentation for multilabel classifications: https://docs.voxel51.com/user_guide/using_datasets.html#multilabel-classification

[tataganesh]

Thank for the clarifications! I had a follow-up question (sorry if I am misunderstanding something) -

I am going through labelstudio.py and I noticed that in the export_label_to_label_studio function, both Classification and Classifications types are checked:

fiftyone/fiftyone/utils/labelstudio.py

Lines 710 to 711 in c6c4abb

	elif _check_type(label, fol.Classification, fol.Classifications):
	result_value, ls_type, ids = _to_classification(label)

Looking at the _to_classification function:

fiftyone/fiftyone/utils/labelstudio.py

Lines 763 to 768 in c6c4abb

	if isinstance(label, fol.Classifications):
	return (
	{ls_type: [l.label for l in label.classifications]},
	ls_type,
	[l.id for l in label.classifications],
	)

Could you clarify if this means some support for exporting multi-label classifications already exists?

[brimoor]

Hmm that's interesting, looks like there indeed may already be some partial support.

Upon a brief glance, it looks like there's at least a couple other changes needed:

1. Add "classifications" here to formally declare support for Classifications-type labels:
   

   fiftyone/fiftyone/utils/labelstudio.py

   Lines 99 to 115 in c6c4abb

   	@property
   	def supported_label_types(self):
   	return [
   	"classification",
   	"detection",
   	"detections",
   	"instance",
   	"instances",
   	"polyline",
   	"polylines",
   	"polygon",
   	"polygons",
   	"keypoint",
   	"keypoints",
   	"segmentation",
   	"scalar",
   	]

2. Update this line:
   

   fiftyone/fiftyone/utils/labelstudio.py

   Line 894 in c6c4abb

   return [fol.Classification(label=l) for l in label_values]

   
   to return a Classifications label rather than a raw list:

    return fol.Classifications(
        classifications=[fol.Classification(label=l) for l in label_values]
    )

3. Update the docs (documenting for myself; I can handle this part 😄):

• fiftyone/docs/source/integrations/labelstudio.rst

  Line 29 in c6c4abb

  - :ref:`Classification <classification>`

• fiftyone/docs/source/integrations/labelstudio.rst

  Line 418 in c6c4abb

  - ``"classification"``: a single classification stored in

Ultimately the way to test that everything is working as expected is to run the basic recipe on a dataset that contains some Classifications labels and checking that everything works as expected.

[tataganesh]

This is great @brimoor, thank you for the breakdown! I will try to submit a PR soon!

[tataganesh]

@brimoor
I have started working on this issue and I am unable to resolve an error. I have made the changes mentioned in your comment, and when I try to load the label studio annotations using dataset.load_annotations(anno_key) , I get an error on this line:

fiftyone/fiftyone/utils/labelstudio.py

Line 419 in c6c4abb

results[sample_id] = {l.id: (ln, l) for (ln, l) in labels}

Error:

    results[sample_id] = {l.id: (ln, l) for (ln, l) in labels}
AttributeError: 'Classifications' object has no attribute 'id'

I noticed that the Classifications object has no id attribute (only the Classification object does). So I made the following change to the import_label_studio_annotation function.

Changes made to lines:

fiftyone/fiftyone/utils/labelstudio.py

Line 677 in c6c4abb

label.id = label_id

Changed lines:

        if isinstance(label, fol.Classifications):
            for l in label.classifications:
                l.id = label_id
        else:
            label.id = label_id

And in _import_annotations changed

 results[sample_id] = {l.id: (ln, l) for (ln, l) in labels} 

to

results[sample_id] = {}
for ln, l in labels:
    if isinstance(l, fol.Classifications):
        label_id = l.classifications[0].id
        results[sample_id][label_id] = (ln, l)
    else:
        results[sample_id][l.id] = (ln, l)

But even if with this I get some other error. Any help would be appreciated to resolve this error, thank you!

[brimoor]

@tataganesh looping in @ehofesmann who can help advise on how to proceed here (when he has some bandwidth!)

[ehofesmann]

Hey @tataganesh! I believe the issue is that for a label list like Classifications, the results object is expecting the label of each classification to be unique. So when you do this:

        if isinstance(label, fol.Classifications):
            for l in label.classifications:
                l.id = label_id

Then all of the individual fo.Classification labels have the same ID which would cause issues downstream, likely the error that you're seeing. I believe the final_annotations dictionary from download_annotations would then have a structure like this {label_field: {return_type: {sample_id: {label_id: fo.Classification}}}}. Notably that the end label id would correspond to a single fo.Classification, of which there could be multiple for a single sample