Initial commit -- Adding calibration loss specific to segmentation #7819

Bala93 · 2024-06-02T02:12:49Z

Description

Model calibration has helped in developing reliable deep learning models. In this pull request, I have added a new loss function NACL (https://arxiv.org/abs/2303.06268, https://arxiv.org/abs/2401.14487) which has shown promising results for both discriminative and calibration in segmentation.

Future Plans: Currently, MONAI has some of the alternative loss functions (Label Smoothing, and Focal Loss), but it doesn't have the calibration specific loss functions (https://arxiv.org/abs/2111.15430, https://arxiv.org/abs/2209.09641). Besides, these methods are better evaluated with calibration metrics, Expected Calibration Error (https://lightning.ai/docs/torchmetrics/stable/classification/calibration_error.html).

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

for more information, see https://pre-commit.ci

Fix the order of loss function in import statement

Fixed the formatting and minor issues.

Reorder import statements

Reorder system imports.

isort inferred one

Updated class name according to standard

Fixed the inline comment

Add datatype in functions and classes

ericspod

Thanks for the contribution! I had a few comments, and the CICD errors are easy fixes I think. The test fail is from accuracy it seems which can be fixed by changing the error.

monai/losses/segcalib.py

Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

for more information, see https://pre-commit.ci

Add model description.

for more information, see https://pre-commit.ci

Fixed the initialization issues with return types

for more information, see https://pre-commit.ci

Use Any instead of tensor

Import typing

Fix return type formatting

DCO Remediation Commit for bala93 <balamuralim.1993@gmail.com> I, Balamurali <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: c4f8283 I, bala93 <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: 8fbec82 I, bala93 <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: 7c121a0 I, bala93 <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: dccde47 Signed-off-by: bala93 <balamuralim.1993@gmail.com>

DCO Remediation Commit for bala93 <balamuralim.1993@gmail.com> I, bala93 <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: 8fbec82 I, bala93 <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: 7c121a0 I, bala93 <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: dccde47 Signed-off-by: bala93 <balamuralim.1993@gmail.com>

DCO Remediation Commit for Balamurali <balamuralim.1993@gmail.com> I, Balamurali <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: bc6b995 Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

I, bala93 <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: 3a00aec Signed-off-by: bala93 <balamuralim.1993@gmail.com>

for more information, see https://pre-commit.ci

Bala93 · 2024-08-05T23:48:32Z

Hi @Bala93 I have a few further comments, the fixes for the mypy issues should be minor and the DCO page tells you how to address that. Thanks!

Hi @ericspod, Thanks for your feedback. I have addressed all the issues. I will work on using the monai functions (for the gaussian, and mean) in some time. Until then, is it possible to merge it with the dev branch and make the loss function available ? Thank you.

docs/source/losses.rst

Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

KumoLiu

Thanks for the contribution! Overall looks good to me. Several minor inline comments.

monai/losses/nacl_loss.py

tests/test_nacl_loss.py

* Modify the code with MeanFilter, and GaussianFilter of MONAI layers * Add Doc string explaining the mask preparation Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

for more information, see https://pre-commit.ci

Bala93 · 2024-08-07T05:32:37Z

Hi @KumoLiu @ericspod. I have addressed the comments with the latest commit. Please feel free to resolve the comments if deemed solved. Thanks for your prompt response.

KumoLiu

Thanks for the quick update, few further comments inline.

monai/losses/nacl_loss.py

Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

ericspod · 2024-08-07T16:13:07Z

Looks good now to me with the last minor comments addressed.

Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

…nto model-calibration

DCO Remediation Commit for Balamurali <balamuralim.1993@gmail.com> I, Balamurali <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: 7deb2cc Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

KumoLiu

Thanks for the quick update, looks good to me now. Will trigger more tests and try to merge this one.

KumoLiu · 2024-08-08T06:45:20Z

/build

Bala93 and others added 10 commits June 1, 2024 22:11

Initial commit -- Adding calibration loss specific to segmentation

8fbec82

[pre-commit.ci] auto fixes from pre-commit.com hooks

23b897b

for more information, see https://pre-commit.ci

Update __init__.py

b2ec62b

Fix the order of loss function in import statement

Update segcalib.py

93ee114

Fixed the formatting and minor issues.

Update segcalib.py

42e732b

Reorder import statements

Update segcalib.py

187053d

Reorder system imports.

Update segcalib.py

1d27ec5

isort inferred one

Update segcalib.py

d499134

Updated class name according to standard

Update segcalib.py

1e3f755

Fixed the inline comment

Update segcalib.py

9dedfba

Add datatype in functions and classes

ericspod reviewed Jun 13, 2024

View reviewed changes

ericspod requested a review from KumoLiu June 13, 2024 14:34

Bala93 and others added 18 commits June 14, 2024 06:32

Update monai/losses/segcalib.py

59959ce

Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

Update monai/losses/segcalib.py

cf1d044

Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

0926851

for more information, see https://pre-commit.ci

Update segcalib.py

5317706

Add model description.

[pre-commit.ci] auto fixes from pre-commit.com hooks

3155433

for more information, see https://pre-commit.ci

Add specific to gaussian for both 2d and 3d

7c121a0

[pre-commit.ci] auto fixes from pre-commit.com hooks

24efd85

for more information, see https://pre-commit.ci

Merge branch 'Project-MONAI:dev' into model-calibration

0067953

Add mean loss and resolve formatting

dccde47

[pre-commit.ci] auto fixes from pre-commit.com hooks

44e8065

for more information, see https://pre-commit.ci

Merge branch 'dev' into model-calibration

57686d7

Update segcalib.py

5cd9a33

Fixed the initialization issues with return types

[pre-commit.ci] auto fixes from pre-commit.com hooks

b547c4e

for more information, see https://pre-commit.ci

Update segcalib.py

42a0215

Use Any instead of tensor

Update segcalib.py

7e36ca1

Import typing

Update segcalib.py

6dbd53d

Update segcalib.py

354056c

Update segcalib.py

7eb911f

Fix return type formatting

Bala93 and others added 7 commits August 5, 2024 14:25

Formatting check with monai

3a00aec

DCO Remediation Commit for Balamurali <balamuralim.1993@gmail.com> I, Balamurali <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: bc6b995 Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

Update test_nacl_loss.py

818b42b

DCO Remediation Commit for Balamurali <balamuralim.1993@gmail.com> I, Balamurali <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: bc6b995 Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

Added mypy fixes

6647708

Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

DCO Remediation Commit for bala93 <balamuralim.1993@gmail.com>

7e579dd

I, bala93 <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: 3a00aec Signed-off-by: bala93 <balamuralim.1993@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

4f8abf1

for more information, see https://pre-commit.ci

Bala93 requested a review from ericspod August 5, 2024 23:50

KumoLiu reviewed Aug 6, 2024

View reviewed changes

docs/source/losses.rst Outdated Show resolved Hide resolved

Update docs/source/losses.rst

b72e478

Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

KumoLiu reviewed Aug 6, 2024

View reviewed changes

Bala93 and others added 2 commits August 7, 2024 01:24

* Include test cases covering more cases

747681d

* Modify the code with MeanFilter, and GaussianFilter of MONAI layers * Add Doc string explaining the mask preparation Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

3b15554

for more information, see https://pre-commit.ci

KumoLiu reviewed Aug 7, 2024

View reviewed changes

monai/losses/nacl_loss.py Outdated Show resolved Hide resolved

monai/losses/nacl_loss.py Outdated Show resolved Hide resolved

monai/losses/nacl_loss.py Show resolved Hide resolved

Bala93 and others added 2 commits August 7, 2024 06:27

Update monai/losses/nacl_loss.py

877139c

Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

Update monai/losses/nacl_loss.py

4679456

Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

Bala93 added 6 commits August 7, 2024 10:32

* Add docstring with better explanations

7c5217e

Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

* Maintain the dimension consistency.

d33f435

Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

Update nacl_loss.py

7deb2cc

Update nacl_loss.py

91ce50b

Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

Merge branch 'model-calibration' of https://github.com/Bala93/MONAI i…

7f87e0c

…nto model-calibration

Modify docstring

0e880a8

DCO Remediation Commit for Balamurali <balamuralim.1993@gmail.com> I, Balamurali <balamuralim.1993@gmail.com>, hereby add my Signed-off-by to this commit: 7deb2cc Signed-off-by: Balamurali <balamuralim.1993@gmail.com>

Bala93 requested a review from KumoLiu August 8, 2024 06:32

Merge branch 'dev' into model-calibration

db9daeb

KumoLiu approved these changes Aug 8, 2024

View reviewed changes

KumoLiu merged commit 660891f into Project-MONAI:dev Aug 8, 2024
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial commit -- Adding calibration loss specific to segmentation #7819

Initial commit -- Adding calibration loss specific to segmentation #7819

Bala93 commented Jun 2, 2024 •

edited

Loading

ericspod left a comment

Bala93 commented Aug 5, 2024 •

edited

Loading

KumoLiu left a comment

Bala93 commented Aug 7, 2024 •

edited

Loading

KumoLiu left a comment

ericspod commented Aug 7, 2024

KumoLiu left a comment

KumoLiu commented Aug 8, 2024

Initial commit -- Adding calibration loss specific to segmentation #7819

Initial commit -- Adding calibration loss specific to segmentation #7819

Conversation

Bala93 commented Jun 2, 2024 • edited Loading

Description

Types of changes

ericspod left a comment

Choose a reason for hiding this comment

Bala93 commented Aug 5, 2024 • edited Loading

KumoLiu left a comment

Choose a reason for hiding this comment

Bala93 commented Aug 7, 2024 • edited Loading

KumoLiu left a comment

Choose a reason for hiding this comment

ericspod commented Aug 7, 2024

KumoLiu left a comment

Choose a reason for hiding this comment

KumoLiu commented Aug 8, 2024

Bala93 commented Jun 2, 2024 •

edited

Loading

Bala93 commented Aug 5, 2024 •

edited

Loading

Bala93 commented Aug 7, 2024 •

edited

Loading