Add features to the YOLO model from the latest YOLO variants #817

senarvi · 2022-03-28T11:51:33Z

What does this PR do?

The YOLO model has been largely refactored, so that it's easy to incorporate new features from different YOLO variants, such as different algorithms for assigning targets to anchors. It also supports defining the network structure as a PyTorch module, in addition to loading a Darknet configuration file. The implementation now supports the most important features of YOLOv3, YOLOv4, YOLOv5, YOLOv7, Scaled-YOLOv4, and YOLOX.

Supports several new algorithms for matching targets to anchors.
Adds support for DIoU and CIoU losses.
By default the target for confidence of a detection that has been assigned a target is 1.0. This pull request adds a support for using the overlap between the target and the predicted box instead.
The code is refactored from one big class into model, anchor matching, and loss function classes.
Target class labels may be specified as a matrix of class probabilities, allowing multiple classes per object.
Automatic padding in convolutional and max pooling layers works now in every case.
Weight decay is applied only to convolutional layer weights.
Supports label smoothing.
Command line interface is now using LightningCLI.
Network architectures can be written as PyTorch modules. YOLOv4, YOLOv5, YOLOv7, and YOLOX architectures are included.
Supports deep supervision (auxiliary detection heads).
Calculates MAP metric using TorchMetrics.
Adds complete type hints for static type checking.
Removes iou and giou metrics and losses from Bolts, as they are available from Torchvision.

Fixes #816

Before submitting

Was this discussed/approved via a Github issue? (no need for typos and docs improvements) - issue for discussion: Upgrade the YOLO model with features from new variants #816
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests? [not needed for typos/docs]
Did you verify new and existing tests pass locally with your changes?
If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

* Supports several algorithms for matching targets to anchors. * Added support for DIoU and CIoU losses. * Target for confidence can be selected between 1.0 and the overlap between the target and the predicted box. * Target classes may be specified as a matrix of class probabilities, allowing multiple classes per object. * Fixed rounding of maxpool layer size. * Weight decay is applied only to convolutional layer weights. * Command line interface is now using LightningCLI.

* Calculates MAP metric using TorchMetrics. * Convolutional and max pooling layers automatically add the correct amount of padding. * Added YOLOv4 network.

Loading a checkpoint with a different number of output classes is possible with strict=False.

…ities.rank_zero is not available

* Added the deeper P6 variant of YOLOv4. * Width of each backbone stage can be configured freely. * The number of input channels in the PyTorch architectures can be changed. * Added predict_step().

… of Torchvision

…ames

Borda · 2023-05-18T16:23:46Z

@senarvi we have fixed master a few munites ago, so shall be 🟢 could you pls merge it here 🐿️

senarvi · 2023-05-19T20:32:38Z

@senarvi we have fixed master a few munites ago, so shall be 🟢 could you pls merge it here 🐿️

Great. Seems like the tests are passing now.

Borda · 2023-05-22T21:10:33Z

@senarvi we have fixed master a few munites ago, so shall be 🟢 could you pls merge it here 🐿️

Great. Seems like the tests are passing now.

perfect, it just sai about some conflicts, but I can't see them, mind check it?
Then I set ti on auto-merge so shall by landing any time soo 🎉

mypy doesn't understand "= ... if ... else ..." and ruff doesn't accept the longer form.

senarvi · 2023-05-25T20:08:47Z

@senarvi we have fixed master a few munites ago, so shall be 🟢 could you pls merge it here 🐿️

Great. Seems like the tests are passing now.

perfect, it just sai about some conflicts, but I can't see them, mind check it? Then I set ti on auto-merge so shall by landing any time soo 🎉

@Borda managed to fix it finally. ruff formatted the code in a way that caused mypy to fail. Formatting in a way that mypy accepts caused ruff to fail. In the end I added a "type: ignore" comment. It wasn't merged automatically, but all the tests have passed.

Borda · 2023-05-29T11:15:08Z

src/pl_bolts/models/detection/yolo/README.md

@@ -0,0 +1,59 @@
+# YOLO
+
+The YOLO model has evolved quite a bit, since the original publication in 2016. The original source code was written in C, using a framework called [Darknet](https://github.com/pjreddie/darknet). The final revision by the original author was called YOLOv3 and described in an [arXiv paper](https://arxiv.org/abs/1804.02767). Later various other authors have written implementations that improve different aspects of the model or the training procedure. [YOLOv4 implementation](https://github.com/AlexeyAB/darknet) was still based on Darknet and [YOLOv5](https://github.com/ultralytics/yolov5) was written using PyTorch. Most other implementations are based on these.


cc: @lantiga

lantiga

Very cool contribution, thank you @senarvi

senarvi added 6 commits March 28, 2022 13:49

Fixed ONNX export

476adb9

meshgrid() call made future-proof by using the indexing argument

8d70ca1

torch.jit.script fails with a lambda function

35a98ba

YOLOV4Tiny, YOLOV5, and YOLOX network architectures in plain PyTorch

a91793c

Improvements to YOLO

b237099

* Calculates MAP metric using TorchMetrics. * Convolutional and max pooling layers automatically add the correct amount of padding. * Added YOLOv4 network.

senarvi requested review from Borda, ethanwharris and awaelchli as code owners March 28, 2022 11:51

github-actions bot added the model label Mar 28, 2022

senarvi mentioned this pull request Mar 28, 2022

Upgrade the YOLO model with features from new variants #816

Closed

senarvi added 7 commits March 29, 2022 18:46

YOLO output layer name includes the number of outputs

82f4de1

Loading a checkpoint with a different number of output classes is possible with strict=False.

Complete type hints

8a201a0

Updated CHANGELOG.

2737fa4

Torchvision import made conditional

26987eb

Use expand() instead of broadcast_to() for backward compatibility

09bce80

Merge branch 'master' into yolo-update

471107f

Use pytorch_lightning.utilities.distributed if pytorch_lightning.util…

a80536e

…ities.rank_zero is not available

mergify bot added the has conflicts label May 4, 2022

senarvi added 2 commits June 8, 2022 09:07

YOLOV4P6 network architecture

b1b8db3

* Added the deeper P6 variant of YOLOv4. * Width of each backbone stage can be configured freely. * The number of input channels in the PyTorch architectures can be changed. * Added predict_step().

Merge branch 'master' into yolo-update

f499b76

mergify bot removed the has conflicts label Jun 8, 2022

senarvi added 3 commits June 8, 2022 09:45

Fixed document generation, when MeanAveragePrecision is not available

5661dba

Use arxiv URL to avoid a too long line

0ad5867

Use torch.div() instead of //

84a949f

mergify bot added the has conflicts label Jun 28, 2022

redleaf-kim and others added 4 commits August 2, 2022 23:58

remove under_review decorators

cf1646c

add yolo cfg with giou & update related test function

fecf88c

add serveral yolo config & layers function test

b5abc8f

Merge branch 'Lightning-AI:master' into yolo_review

198ebc1

senarvi added 3 commits April 6, 2023 05:44

Use giou in YOLO tests to allow them to pass also with older versions…

2242736

… of Torchvision

Use double underscores in links in the docstring to avoid duplicate n…

4807f3c

…ames

Check that targets are given in training mode

64a3e47

senarvi force-pushed the yolo-update branch 2 times, most recently from 0d6f4b8 to 64a3e47 Compare April 6, 2023 10:55

Fixed docstring formatting

26c9a7c

Borda force-pushed the master branch from afdc09a to c5137b4 Compare May 19, 2023 17:15

Merge branch 'master' into yolo-update

c891ca4

mergify bot added the has conflicts label May 20, 2023

Borda approved these changes May 22, 2023

View reviewed changes

Borda enabled auto-merge (squash) May 22, 2023 21:09

senarvi added 2 commits May 23, 2023 11:36

Merge branch 'master' into yolo-update

e119ee9

Add

98e4c2e

auto-merge was automatically disabled May 23, 2023 11:39
Head branch was pushed to by a user without write access

mergify bot removed the has conflicts label May 23, 2023

Removed files that popped up back in the merge

bf6295b

mergify bot added the ready label May 23, 2023

senarvi added 2 commits May 24, 2023 06:06

Code formatting fixed by ruff

9042812

Work around a problem with mypy and if-else ternary operator

a399bed

mergify bot removed the ready label May 24, 2023

Fixed bcefunc assignment so that both ruff and mypy are happy

c976147

mypy doesn't understand "= ... if ... else ..." and ruff doesn't accept the longer form.

mergify bot added the ready label May 25, 2023

Borda reviewed May 29, 2023

View reviewed changes

lantiga approved these changes May 29, 2023

View reviewed changes

Borda merged commit 5f5adaf into Lightning-Universe:master May 29, 2023

senarvi deleted the yolo-update branch May 29, 2023 17:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add features to the YOLO model from the latest YOLO variants #817

Add features to the YOLO model from the latest YOLO variants #817

senarvi commented Mar 28, 2022 •

edited

Loading

Borda commented May 18, 2023

senarvi commented May 19, 2023

Borda commented May 22, 2023

senarvi commented May 25, 2023

Borda May 29, 2023

lantiga left a comment

		@@ -0,0 +1,59 @@
		# YOLO

		The YOLO model has evolved quite a bit, since the original publication in 2016. The original source code was written in C, using a framework called [Darknet](https://github.com/pjreddie/darknet). The final revision by the original author was called YOLOv3 and described in an [arXiv paper](https://arxiv.org/abs/1804.02767). Later various other authors have written implementations that improve different aspects of the model or the training procedure. [YOLOv4 implementation](https://github.com/AlexeyAB/darknet) was still based on Darknet and [YOLOv5](https://github.com/ultralytics/yolov5) was written using PyTorch. Most other implementations are based on these.

Add features to the YOLO model from the latest YOLO variants #817

Add features to the YOLO model from the latest YOLO variants #817

Conversation

senarvi commented Mar 28, 2022 • edited Loading

What does this PR do?

Before submitting

PR review

Did you have fun?

Borda commented May 18, 2023

senarvi commented May 19, 2023

Borda commented May 22, 2023

senarvi commented May 25, 2023

Borda May 29, 2023

Choose a reason for hiding this comment

lantiga left a comment

Choose a reason for hiding this comment

senarvi commented Mar 28, 2022 •

edited

Loading