add softmax version to focal_loss

[qingpeng9802]

Fixes #6510 .

Description

Add softmax version to Focal loss

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[qingpeng9802]

For further reference, the implementation reference (pytorch/detectron) has been visualized above.
It is worth to notice that current MONAI's sigmoid focal loss is kind of different from detectron's. Detectron's sigmoid implementation does not include background class by default. See, https://github.com/pytorch/pytorch/blob/6e0c741105b4022fc7c3f8c2919aad9f971beb62/modules/detectron/sigmoid_focal_loss_op.cc#L35-L36
Other implementations also include background class by default like MONAI so this might not be a big issue.
https://github.com/kornia/kornia/blob/edbcae696909bca677c71615a71cb89ba87321e1/kornia/losses/focal.py#L144 and
https://github.com/qubvel/segmentation_models.pytorch/blob/e7cbea0bbd546167b9d1757b2cfeca3fbe26a33e/segmentation_models_pytorch/losses/_functional.py#L36

@wyli Please help review the current implementation. Some unit tests will be added after the review.

[wyli]

thanks @qingpeng9802, it looks good to me, one minor thing is that if we assume target is a one-hot binary tensor, then I think - (1 - input_ls.exp()).pow(gamma) * input_ls * target could be simplified to compute for target==1 only.

[qingpeng9802]

/black

[qingpeng9802]

thanks @qingpeng9802, it looks good to me, one minor thing is that if we assume target is a one-hot binary tensor, then I think - (1 - input_ls.exp()).pow(gamma) * input_ls * target could be simplified to compute for target==1 only.

loss: torch.Tensor = torch.where(target>0.5, - (1 - input_ls.exp()).pow(gamma) * input_ls, 0)

and

input_ls[target>0.5]=0.
loss: torch.Tensor = - (1 - input_ls.exp()).pow(gamma) * input_ls

are benchmarked. Both are significantly slower than the original one. It looks like torch.jit.script can fuse element-wise operators efficiently so torch.where or indexing might break the optimization. Therefore, the original one is kept.

[wyli]

sure, thanks for looking into these @qingpeng9802!

[qingpeng9802]

/black

[wyli]

/build