[Feature] Add MaskFormer #2789

shiyutang · 2022-11-30T07:45:29Z

PR types

New features

PR changes

Models

Description

Add Maskformer.
目前已经使用PaddleSeg中已有的transforms，并在paddle2.4下环境测试精度mIOU为47.93%，满足复现目标46.7%：

在small 配置上验证，训练精度和评估精度均可以达到50.4，超过论文中精度的49.8：

完成tipc部署：

…nto paddle2.2.2

configs/maskformer/maskformer_ade20k_swin_tiny.yml

configs/maskformer/README.md

configs/maskformer/maskformer_ade20k_swin_base.yml

juncaipeng · 2023-01-12T09:25:42Z

paddleseg/transforms/transforms.py

@@ -245,11 +245,18 @@ class ResizeByShort:
        short_size (int): The target size of short side.
    """

-    def __init__(self, short_size):
+    def __init__(self, short_size, max_size=2048):


这个是ResizeByShort的标准用法？会改变原先ResizeByShort的逻辑。如果不是标准用法，可以自己实现一个ResizeByShortWithMax

设置的max_size很大，就不会影响之前的逻辑。

paddleseg/models/maskformer.py

juncaipeng · 2023-01-12T09:47:23Z

paddleseg/models/losses/maskformer_loss.py

+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+


如果参考的比较多，写上参考实现的链接

加个空行吧，和copyright混合了

juncaipeng · 2023-01-12T09:48:41Z

paddleseg/core/train.py

-            loss_list.append(coef_i * loss_i(logits, labels))
+            if targets is not None:
+                loss_list.append(coef_i * loss_i(logits, targets))
+            else:


train.py里面的实现，我再想想。

关于loss计算，马上会更新一个范式，会放到model里面实现特定的loss计算。所以这部分等那个pr合入后再改下。

…into paddle2.2.2

shiyutang · 2023-01-13T02:25:19Z

paddleseg/datasets/ade.py

-            data['label'] = label
-            return data
+            data['label'] = label - 1
+            data['img'] = paddle.to_tensor(data['img'])


修改为和之前一致

shiyutang · 2023-01-12T09:55:23Z

paddleseg/models/losses/maskformer_loss.py

+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+


paddleseg/models/maskformer.py

configs/maskformer/maskformer_ade20k_swin_base.yml

paddleseg/datasets/ade.py

paddleseg/models/maskformer.py

shiyutang · 2023-01-19T02:16:35Z

重新训练后，目前精度可以达到47.93
训练精度：

评估精度：

shiyutang · 2023-01-28T02:19:02Z

在small 配置上验证，训练精度可以达到50.4，超过论文中精度的49.8：

juncaipeng

此外也要添加tipc和验证预测模型

juncaipeng · 2023-01-28T11:38:14Z

paddleseg/utils/utils.py

@@ -138,6 +138,7 @@ def download_pretrained_model(pretrained_model):
            pretrained_model = download_file_and_uncompress(
                pretrained_model,
                savepath=_dir,
+                cover=True,


这个会覆盖原先下载的预训练权重等，导致每次都下载，很费时

如果不添加cover，如果之前用户本地有model.params就会加载之前的，但是那个模型参数可能并不是这个模型的，就存在加载不上参数的问题，导致精度差异不符合预期。

juncaipeng · 2023-01-28T11:40:57Z

configs/maskformer/maskformer_swin_base_ade20k_512x512_160k.yml

+      hue_range: 18
+      hue_prob: 1.0
+    - type: RandomHorizontalFlip
+    - type: Padding


前面已经有RandomPaddingCrop，不需要padding了吧。

如果四个yml文件只是model不一样，一个yml文件作为base，其中三个yml文件可以使__base__包含。

已删除，除了model还有训练数据处理不一样，目前已经修改为最大化继承。

juncaipeng · 2023-01-28T11:43:02Z

configs/maskformer/maskformer_swin_base_ade20k_512x512_160k.yml

+  num_classes: 150
+  backbone:
+    type: SwinTransformer_base_patch4_window7_384_maskformer
+  pretrained: https://bj.bcebos.com/paddleseg/paddleseg/dygraph/ade20k/maskformer_ade20k_swin_base/pretrain/model.pdparams


pretrained还是针对整个maskformer模型，不是针对backbone啊

不是，这个是只有swin的模型，训练日志里加载的参数数量也可以看出来

juncaipeng · 2023-01-28T12:10:55Z

paddleseg/cvlibs/param_init.py

+
+
+def th_multihead_fill(layer, qkv_same_embed_dim=True):


这两个th开头的初始化函数，是所有模型都通用的还是maskformer专用的？

这个是可以用于所有模型里multihead attention算子的初始化

juncaipeng · 2023-01-28T12:16:11Z

paddleseg/core/train.py

@@ -176,6 +179,26 @@ def train(model,
            edges = None
            if 'edge' in data.keys():
                edges = data['edge'].astype('int64')
+
+            targets = None
+            if "instances" in data.keys():


这个写法肯定是不行的，过于trick。这个数据处理不可以放到dataset里面吗？

这个是对一个batch的数据进行处理，从而不能放入dataset，已经放入到损失当中。

juncaipeng · 2023-01-30T02:36:13Z

paddleseg/core/train.py

-            loss_list.append(coef_i * loss_i(logits, labels))
+            if targets is not None:
+                loss_list.append(coef_i * loss_i(logits, targets))
+            else:


关于loss计算，马上会更新一个范式，会放到model里面实现特定的loss计算。所以这部分等那个pr合入后再改下。

juncaipeng · 2023-01-30T02:37:08Z

paddleseg/datasets/maskedade.py

+import paddle.nn as nn
+
+from paddleseg.datasets import ADE20K
+from paddleseg.utils.download import download_file_and_uncompress


download_file_and_uncompress 没用到

juncaipeng · 2023-01-30T02:37:55Z

paddleseg/datasets/maskedade.py

+@manager.DATASETS.add_component
+class MaskedADE20K(ADE20K):
+    """
+    ADE20K dataset `http://sceneparsing.csail.mit.edu/`.


说明一下这个MaskedADE20K具体是什么

paddleseg/transforms/transforms.py

paddleseg/models/maskformer.py

shiyutang · 2023-03-01T03:44:02Z

单卡和多卡均能在修改代码后训练：
单卡

多卡

juncaipeng

LGTM

juncaipeng

LGTM

shiyutang added 19 commits November 8, 2022 17:15

scratch_1

4790c5d

scratch_2

41fcd07

forward_align_finished

35ba0d4

update

d2bf25b

backwardaligned_dot2

933b367

backward_blankin

360f1db

train_divergy

db3a3e2

fix_optimizer

7a90e9b

fix_metric

0abed97

align_train_load+metric467

0e466d4

newest

080bac2

paddle2.2.2

7de03d8

update_init_aug

cc31da5

rm_redundant

65d36ee

update

a75a0c3

update

ea94836

update

e05f390

update_init

6e25d85

train_with2.4and new aug

24f8aed

shiyutang requested a review from juncaipeng December 2, 2022 09:49

shiyutang added the TobeReviewed label Dec 2, 2022

shiyutang and others added 6 commits December 5, 2022 10:31

validate_on2.4

4847dc7

update_init

a3d151a

add_multiple_backbone+initfree

d6fad22

Merge branch 'develop' into paddle2.2.2

8738173

fix_format

14933b9

Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleSeg i…

72c42c2

…nto paddle2.2.2

juncaipeng reviewed Jan 12, 2023

View reviewed changes

shiyutang added 2 commits January 13, 2023 11:52

fix_by_comment_test_train_ok_no_acc

3d54334

Merge branch 'paddle2.2.2' of https://github.com/shiyutang/PaddleSeg …

b86a511

…into paddle2.2.2

shiyutang commented Jan 13, 2023

View reviewed changes

valid_47.6

dbc43c0

shiyutang added 2 commits January 19, 2023 10:46

validate_train_47.93

d7479be

valid_train_small_50.4

c403cad

juncaipeng reviewed Jan 30, 2023

View reviewed changes

shiyutang added 3 commits January 30, 2023 20:22

fix_by_comment

f658d73

maskformer_tipc

85c36a3

fix_large_yml

094d1a2

cloud2009 mentioned this pull request Feb 20, 2023

【PaddlePaddle Hackathon 4】模型套件开源贡献任务合集 PaddlePaddle/Paddle#50631

Closed

compact_train_loss

f16a816

juncaipeng previously approved these changes Mar 1, 2023

View reviewed changes

Merge branch 'develop' into paddle2.2.2

fb9668a

shiyutang dismissed juncaipeng’s stale review via fb9668a March 1, 2023 04:08

juncaipeng approved these changes Mar 1, 2023

View reviewed changes

juncaipeng merged commit 7c2f466 into PaddlePaddle:develop Mar 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add MaskFormer #2789

[Feature] Add MaskFormer #2789

shiyutang commented Nov 30, 2022 •

edited

Loading

juncaipeng Jan 12, 2023

shiyutang Jan 13, 2023

juncaipeng Jan 12, 2023

shiyutang Jan 12, 2023

juncaipeng Jan 28, 2023

juncaipeng Jan 12, 2023

juncaipeng Jan 30, 2023

shiyutang Jan 31, 2023

shiyutang Jan 13, 2023

shiyutang Jan 12, 2023

shiyutang commented Jan 19, 2023 •

edited

Loading

shiyutang commented Jan 28, 2023 •

edited

Loading

juncaipeng left a comment

juncaipeng Jan 28, 2023

shiyutang Jan 30, 2023 •

edited

Loading

juncaipeng Jan 28, 2023

shiyutang Jan 30, 2023

juncaipeng Jan 28, 2023

shiyutang Jan 30, 2023

juncaipeng Jan 28, 2023

shiyutang Jan 30, 2023

juncaipeng Jan 28, 2023

shiyutang Jan 30, 2023

juncaipeng Jan 30, 2023

juncaipeng Jan 30, 2023

shiyutang Jan 30, 2023

juncaipeng Jan 30, 2023

shiyutang Jan 30, 2023

shiyutang commented Mar 1, 2023 •

edited

Loading

juncaipeng left a comment

juncaipeng left a comment

[Feature] Add MaskFormer #2789

[Feature] Add MaskFormer #2789

Conversation

shiyutang commented Nov 30, 2022 • edited Loading

PR types

PR changes

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shiyutang commented Jan 19, 2023 • edited Loading

shiyutang commented Jan 28, 2023 • edited Loading

juncaipeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shiyutang Jan 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shiyutang commented Mar 1, 2023 • edited Loading

juncaipeng left a comment

Choose a reason for hiding this comment

juncaipeng left a comment

Choose a reason for hiding this comment

shiyutang commented Nov 30, 2022 •

edited

Loading

shiyutang commented Jan 19, 2023 •

edited

Loading

shiyutang commented Jan 28, 2023 •

edited

Loading

shiyutang Jan 30, 2023 •

edited

Loading

shiyutang commented Mar 1, 2023 •

edited

Loading