Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

treat torch models as feature extractors when feature_extractor=True #3517

Merged
merged 2 commits into from
Sep 27, 2023

Conversation

Rusteam
Copy link
Contributor

@Rusteam Rusteam commented Sep 13, 2023

What changes are proposed in this pull request?

New parameter feature_extractor=False added to torch models.
If set to True, then the model output is treated as features without
forward hooks.

When using dino model with default forward hooks, its output
becomes 3-demensional and can't be used as embeddings. With
this fix the output is expected 2-dimensional.

Also, if embeddings_layer=xxxx is provided, then the model
is replaced with a feature extractor for that layer using torchvision.

Finally, added a check on embeddings_layer making sure it's
not an Identity() layer is it does not return anything with
forward hooks.

How is this patch tested? If it is not, please explain why.

Added a test with dino model from torch hub.

Release Notes

Is this a user-facing change that should be mentioned in the release notes?

  • No. You can skip the rest of this section.
  • Yes. Give a description of this change to be included in the release
    notes for FiftyOne users.

(Details in 1-2 sentences. You can just refer to another PR with a description
if this PR is part of a larger change.)

What areas of FiftyOne does this PR affect?

  • App: FiftyOne application changes
  • Build: Build and test infrastructure changes
  • Core: Core fiftyone Python library changes
  • Documentation: FiftyOne documentation changes
  • Other

@codecov
Copy link

codecov bot commented Sep 25, 2023

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (957e87b) 15.98% compared to head (ec66f7e) 15.88%.
Report is 254 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3517      +/-   ##
===========================================
- Coverage    15.98%   15.88%   -0.11%     
===========================================
  Files          573      594      +21     
  Lines        71345    72838    +1493     
  Branches       800      871      +71     
===========================================
+ Hits         11406    11567     +161     
- Misses       59939    61271    +1332     
Flag Coverage Δ
app 15.88% <0.00%> (-0.11%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
app/packages/map/src/MapTabIndicator.tsx 0.00% <0.00%> (ø)

... and 70 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fiftyone/utils/torch.py Outdated Show resolved Hide resolved
fiftyone/utils/torch.py Show resolved Hide resolved
Copy link
Contributor

@brimoor brimoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, makes sense 👍

@brimoor brimoor merged commit f1e2537 into voxel51:develop Sep 27, 2023
8 of 11 checks passed
benjaminpkane pushed a commit that referenced this pull request Sep 28, 2023
treat torch models as feature extractors when feature_extractor=True
@brimoor brimoor mentioned this pull request Oct 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants