Supervised Identity Prediction (#460)

[roomrys]

Description

Implements supervised identity prediction models that predict instance identity.

Supplants the temporal association-based trackers since classification is performed with single images from visual appearance alone. This implements both bottom-up and top-down versions of the classification models.

Types of changes

• Bugfix
• New feature
• Refactor / Code style update (no logical changes)
• Build / CI changes
• Documentation Update
• Other (explain)

Does this address any currently open issues?

Supervised identity prediction

Outside contributors checklist

• Review the guidelines for contributing to this repository
• Read and sign the CLA and add yourself to the authors list
• Make sure you are making a pull request against the develop branch (not main). Also you should start your branch off develop
• Add tests that prove your fix is effective or that your feature works
• Add necessary documentation (if appropriate)

Thank you for contributing to SLEAP!

❤️

Full changelog

• Add test dataset with tracks

• Add track indices to labels provider

• Add identity class map generator

• Update docstring

• Add class map model trainer, head and config

• Add inference

• Docs and tests for identity module

  • Slightly modified matching to greedy-like behavior

• Fix inference

  • Add imports to evals inference
  • Move the common Predictor.predict() method to base class
  • Fix docstrings for new inference classes
  • Add test model and integration test

• Generate tracks from config metadata if not provided

• Force typecasting in identity functions

• Force boolean masking op

• Clean up inference module

  • Move common Predictor methods to base class
  • Switch to model.predict_on_batch() for massive performance increase
    with predictor.predict().
  • Enable prediction directly on arrays (slow)

• Enable Qt5Agg backend only when necessary during training

• Top-down supervised identity prediction (Top-down supervised identity prediction #476)

• Add sizematcher to new training pipelines

• Fix topdown ID visualization during training

• Add LabeledFrame.tracked_instances property for filtering

  • Greedy checking in has_* properties

• Add Labels.copy() method for creating deep copies

  • Works by serializing and deserializing to JSON (inefficient, but
    guaranteed to work since we have lots of coverage on I/O)

• Extract labels with tracked instances

  • Add copy kwarg to extract to return deep copies
  • Remove user and/or untracked instances in with_user_labels_only().
    Previously this functionality was blocked since we couldn't remove the
    instances from labeled frames without affecting the source labels.
  • Add remove_untracked_instances() utility for filtering out instances
    from the labels.

• Add track filtering in LabelsReader provider

  • This is slightly redundant with
    Labels.with_user_labels_only(..., with_track_only=True) but serves as
    an extra guarantee that we don't train on instances without tracks
    accidentally, regardless of how the data is preprocessed. Can still
    emit "empty" frames if no instances have tracks set, however.

• Add track filtering in DataReaders during training

  • Auto-enabled when training from ID models
  • Filters out instances without tracks BEFORE train/val splitting
  • Split is now done on copy of labels
  • Fix DataReaders arg typing
  • Tests for DataReaders

• Add crop size detection to topdown ID models

  • Add training integration test for topdown ID

• Add removal of untracked instances for labeled instances

  • previously used LabeledFrame.tracked_instances() which only returns predicted instances with tracking
  • created LabeledFrame.remove_untracked which returns both user labeled and predicted instances with tracking

• Add tests

  • test Labels.remove_untracked_instances() for both cases of remove_empty_frames: bool
  • test LabeledFrames.remove_untracked() for both user-labeled and predicted frames

[codecov]

Codecov Report

Merging #679 (6be61d0) into develop (1c57287) will increase coverage by 0.50%.
The diff coverage is 75.82%.

@@             Coverage Diff             @@
##           develop     #679      +/-   ##
===========================================
+ Coverage    64.89%   65.40%   +0.50%     
===========================================
  Files          124      126       +2     
  Lines        20202    21002     +800     
===========================================
+ Hits         13111    13737     +626     
- Misses        7091     7265     +174     

Impacted Files	Coverage Δ
sleap/nn/config/__init__.py	100.00% <ø> (ø)
sleap/nn/evals.py	13.04% <0.00%> (-0.24%)	⬇️
sleap/nn/data/instance_cropping.py	65.69% <50.00%> (-0.48%)	⬇️
sleap/nn/data/providers.py	64.88% <60.00%> (+1.89%)	⬆️
sleap/nn/training.py	60.02% <65.00%> (+0.68%)	⬆️
sleap/nn/inference.py	65.01% <65.56%> (+1.67%)	⬆️
sleap/nn/identity.py	80.00% <80.00%> (ø)
sleap/nn/data/identity.py	83.92% <83.92%> (ø)
sleap/nn/model.py	91.66% <84.78%> (-3.39%)	⬇️
sleap/nn/data/pipelines.py	89.56% <90.72%> (+0.37%)	⬆️
... and 8 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c57287...6be61d0. Read the comment docs.

[roomrys]

Need to add import sleap.nn.identity to sleap/nn/__init__.py.

Upon running pytest tests in main sleap directory, currently get following error (which is resolved when add change mentioned above):

E                 AttributeError: Exception encountered when calling layer "top_down_multi_class_find_peaks" (type TopDownMultiClassFindPeaks).
E
E                 in user code:
E
E                     File "D:\social-leap-estimates-animal-poses\random\sleap2\sleap\sleap\nn\inference.py", line 3373, in call  *
E                         points, point_vals, class_probs = sleap.nn.identity.classify_peaks_from_vectors(
E
E                     AttributeError: module 'sleap.nn' has no attribute 'identity'