-
Notifications
You must be signed in to change notification settings - Fork 330
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Port DeepLabV3 segmentation model from keras_cv.models.legacy
to the Task
API
#1831
Conversation
cc: @jbischof |
Task
API
Task
APIkeras_cv.models.legacy
to the Task
API
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the awesome PR!
Hi @ianstenbit |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
High-level question:
Should we just directly offer DeepLabV3Plus? I see from the papers that this is a relatively straightforward addition on top of DeepLabV3, and I see that you wrote a Keras tutorial on this: https://keras.io/examples/vision/deeplabv3_plus/
Unless there's a reason to offer both, I'd suggest that we update this to implement DeepLabV3Plus, and then we can verify the forward pass numerics against your KerasIO tutorial.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work!
keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py
Outdated
Show resolved
Hide resolved
|
||
inputs = backbone.input | ||
|
||
final_backbone_pyramid_output = backbone.get_layer( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it safe to assume that this dictionary is sorted? We may need to extract the max manually.
Also: why not use the output of the backbone as the max layer?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was done as per this request.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think in general we probably can just use the backbone output here.
I brought this up originally because I wasn't clear on whether DeepLabV3Plus always wants to use the highest-available P-level here or if it always wants to use e.g. P4. It seems like it may want the highest P-level always, which I think should mean it always matches up with the backbone output.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think the last P-level is exactly the same as the output, although it is the same shape in the width and height dimensions. There are often extra normalization and activations (and even extra conv layers!) afterward that we omit for 1:1 comparability with the previous levels (e.g., ResNetV2, MobileNet)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jbischof
I attempted to train with the output of the backbone
as the features, and it seems that the outputs are significantly worse in that case, compared to the P5
features.
Here's a comparison: https://wandb.ai/geekyrakshit/deeplabv3-keras-cv/reports/Backbone-Outputs-vs-P5-Features--Vmlldzo0NjQ5NDUw
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm having trouble reading that chart. Which color is P5? Which is backbone?
keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py
Outdated
Show resolved
Hide resolved
keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py
Outdated
Show resolved
Hide resolved
keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus_test.py
Outdated
Show resolved
Hide resolved
"backbone": self.backbone, | ||
"spatial_pyramid_pooling": self.spatial_pyramid_pooling, | ||
"projection_filters": self.projection_filters, | ||
"segmentation_head": self.segmentation_head, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still struggle with serialization. Do we need to explicitly serialize and deserialize a submodel like this for saving to work properly? I see a saving test but I know there are gotchas.
keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus_test.py
Outdated
Show resolved
Hide resolved
used as a feature extractor for the DeepLabV3+ Encoder. Should | ||
either be a `keras_cv.models.backbones.backbone.Backbone` or a | ||
`tf.keras.Model` that implements the `pyramid_level_inputs` | ||
property with keys "P2", "P3", "P4", and "P5" and layer names as |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need all five keys? It just seems like "P2" now
Do we have a reference implementation for this model where we can check the forward pass and maybe grab some pretrained weights? Also: don't forget to set up presets! Thanks for your hard work 🚀 |
@soumik12345 do you think you'll be able to finish up this PR soon? If not I may push some changes to your branch so that we can get this finished -- thank you! |
Hi @ianstenbit Please let me know if there are any immediate changes that I can make. |
Yeah I think the architecture + tests are looking good -- just want to verify the performance by training it against a benchmark 😄 |
"pixel-level semantic labeling task, which consists of fine " | ||
"annotations for train and val sets (3475 annotated images) " | ||
"and 34 classes. This model achieves a final Mean IoU of " | ||
"0.36 on the validation set for fine annotations." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does this compare to the expected performance of this model on this dataset?
@soumik12345 we'd like to go ahead and merge this for now (without worrying about verifying the performance just yet, and without porting to Keras Core). Would you be willing to:
and then I can merge it? I tried to push to to your fork to make the changes myself but I don't have access. |
Hi @ianstenbit removed the presets and fixed linting. |
Wonderful, thank you for the contribution! If the CI is happy then I will go ahead and merge this 😄 |
Looks like the linter is still unhappy.
You'll need to add |
Keras Core (incl. accelerator) testing is not yet relevant for this, so this is good to go |
…e `Task` API (keras-team#1831) * add: DeepLabV3 backbone + preset placeholders * add: presents for DeepLabV3 backbone * update: minor changes * update: minor changes * add: license info * update: refactored DeepLabV3 as a single Task * add: deeplab_v3 module inside segmentation module * update: refactor segmentation head * add: serialization test for SegmentationHead * remove: legacy segmentation models dir * remove: legacy segmentation imports * update: added weight_decay in compile() * add: docstring for deeplabv3 * update: remove compile() and train_step() from DeepLabV3 * update: imports * update: copyright info * remove: Segmentation Head layer implementation + serialization test * update: refactor DeepLabV3 segmentation head as a keras Sequential * update: made abstractions to implement encoder-decoder architecture * update: _make_deeplabv3_encoder * refactor: docstrings for DeepLabV3 * refactor: docstrings for DeepLabV3 * refactor: DeepLabV3 * update: refactored DeepLabV3 to implement DeepLabV3+ * update: docstring * update: docstring * update: removed segmentation_head_activation * update: removed input_shape and input_tensor * update: remove outdated comment * update: docstring * update: replaced low_level_feature_layer_name with low_level_feature_pyramid_level * update: docastring * update: DeepLabV3 * update: remove unused imports * update: changed DeepLabV3 to DeepLabV3+ * update: remove build * update: docstring * update: add docstring for dropout * update: made encoder output upsampling dynamic * add: tests for DeepLabV3+ * update: order of parameters * update: example in docstring * update: renamed feature_map * update: docs for dropout * update: renamed tests * update: update tests * update: removed test_no_nans * update: fix linting * update: remove cleanup_global_session * update: tests * update: fix linting * update: incorporated feedback * update: incorporated feedback * update: incorporated feedback * add: preset for deeplabv3+ * update: remove presets * fix: linting * update: make CI happy
What does this PR do?
This PR moves the DeepLabV3 segmentation model from
keras_cv.models.legacy
and aligns it with theTask
API.A Kaggle TPU notebook demostrating a few training steps: https://www.kaggle.com/code/soumikrakshit/deeplabv3-tpu/
Before submitting
Pull Request section?
to it if that's the case.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.