Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release time distributed model trained on ~250k videos #226

Merged
merged 5 commits into from
Sep 14, 2022
Merged

Conversation

ejm714
Copy link
Collaborator

@ejm714 ejm714 commented Sep 13, 2022

Replaces time distributed model trained on 15k videos with one trained on 250k videos. The same (improved) frame selection is used for both).

Key decision

This removes the 15k model so there is only one "time distributed" model available as an official model.

Rationale

As seen in the val_metrics.json, the new model does not have improved performance on every species judged by the F1 score. However, the new model has been evaluated on 30k instead of 3k videos and includes a greater diversity of videos given the inclusion of the Chimp & See data. From exploration into calibration and gains curves, it appears that there is still signal for the species with low or zero F1 score, it is just that probabilities are of lower confidence (e.g. highest probabilities still contain an animal, but those probabilities are less than 0.5). The old model was also trained on a much more balanced dataset (blanks at 15%) compared to the new model (blanks account for around 40%). The new model does significantly better on blank performance (f1 of 0.83 vs. 0.54) while also maintaining high performance on other species.

Closes https://github.com/drivendataorg/pjmf-zamba/issues/129

@ejm714 ejm714 requested a review from pjbull September 13, 2022 18:01
@netlify
Copy link

netlify bot commented Sep 13, 2022

Deploy Preview for silly-keller-664934 ready!

Name Link
🔨 Latest commit e33df8d
🔍 Latest deploy log https://app.netlify.com/sites/silly-keller-664934/deploys/6321fe290a10c700091c887b
😎 Deploy Preview https://deploy-preview-226--silly-keller-664934.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 13, 2022

@codecov-commenter
Copy link

Codecov Report

Merging #226 (e33df8d) into master (570f9cc) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master    #226   +/-   ##
======================================
  Coverage    87.0%   87.0%           
======================================
  Files          29      29           
  Lines        1937    1937           
======================================
  Hits         1686    1686           
  Misses        251     251           
Impacted Files Coverage Δ
zamba/models/config.py 96.7% <ø> (ø)

@pjbull
Copy link
Member

pjbull commented Sep 14, 2022

🚀 !!!

@pjbull pjbull merged commit 2a4e9fc into master Sep 14, 2022
@pjbull pjbull deleted the td-full-set branch September 14, 2022 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants