Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

使用dropout在eval模式下出错的问题 #31092

Closed
yangkerrr opened this issue Feb 21, 2021 · 9 comments
Closed

使用dropout在eval模式下出错的问题 #31092

yangkerrr opened this issue Feb 21, 2021 · 9 comments
Assignees
@ysh329

Comments

@yangkerrr
Copy link

出错部分代码：
class _routing(nn.Layer):

def __init__(self, in_channels, num_experts, dropout_rate):
    super(_routing, self).__init__()
    
    self.dropout = nn.Dropout(p=dropout_rate)
    self.fc = nn.Linear(in_channels, num_experts)

def forward(self, x):
    x = paddle.flatten(x).clone()
    x = self.dropout(x)
    x = self.fc(x)
    return F.sigmoid(x)

可以正常训练，但验证时报错
报错信息：
Traceback (most recent call last):
File "train.py", line 145, in
main(args)
File "train.py", line 140, in main
losses=losses)
File "/home/aistudio/work/PaddleSeg/paddleseg/core/train.py", line 183, in train
model, val_dataset, num_workers=num_workers)
File "/home/aistudio/work/PaddleSeg/paddleseg/core/val.py", line 110, in evaluate
crop_size=crop_size)
File "/home/aistudio/work/PaddleSeg/paddleseg/core/infer.py", line 169, in inference
logits = model(im)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/unet.py", line 63, in forward
x, short_cuts = self.encode(x)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/unet.py", line 95, in forward
x = self.double_conv(x)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/container.py", line 86, in forward
input = layer(input)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/layers/layer_libs.py", line 47, in forward
x = self._conv(x)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/layers/CondConv.py", line 77, in forward
routing_weights = self._routing_fn(pooled_inputs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/layers/CondConv.py", line 19, in forward
x = self.dropout(x)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/common.py", line 689, in forward
name=self.name)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/functional/common.py", line 924, in dropout
if seed is not None else 0, 'dropout_implementation', mode)
ValueError: (InvalidArgument) Input dimension number(num_col_dims) must be between 0 and 1, but received number is 1.
[Hint: Expected (num_col_dims > 0 && num_col_dims < rank) == true, but received (num_col_dims > 0 && num_col_dims < rank):0 != true:1.] (at /paddle/paddle/fluid/framework/eigen.h:94)
[operator < dropout > error]
terminate called without an active exception

C++ Traceback (most recent call last):

0 paddle::framework::SignalHandle(char const*, int)
1 paddle::platform::GetCurrentTraceBackStringabi:cxx11

Error Message Summary:

FatalError: Process abort signal is detected by the operating system.
[TimeInfo: *** Aborted at 1613917048 (unix time) try "date -d @1613917048" if you are using GNU date ***]
[SignalInfo: *** SIGABRT (@0x3e8000028cc) received by PID 10444 (TID 0x7f9180ffd700) from PID 10444 ***]
@paddle-bot-old
Copy link

您好，我们已经收到了您的问题，会安排技术人员尽快解答您的问题，请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时，您也可以通过查看官网API文档、常见问题、历史Issue、AI社区来寻求解答。祝您生活愉快～

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the API，FAQ，Github Issue and AI community to get the answer.Have a nice day!

@ysh329
Copy link
Contributor

ysh329 commented Feb 22, 2021 •
edited
Loading

这个是训练OK，但是预测出问题嘛，你使用的是什么Paddle的版本，CPU还是GPU环境，GPU的话CUDA版本是多少

@yangkerrr
Copy link
Author

这个是训练OK，但是预测出问题嘛，你使用的是什么Paddle的版本，CPU还是GPU环境，GPU的话CUDA版本是多少

Paddle2.0，GPU环境，CUDA11.0（AIStudio内环境）

@sellinaxmj

This comment has been minimized.

Sign in to view
@ysh329
Copy link
Contributor

ysh329 commented Feb 23, 2021

@sellinaxmj 新建issue，就不是一个问题好吧

@ysh329
Copy link
Contributor

ysh329 commented Feb 23, 2021

这个是训练OK，但是预测出问题嘛，你使用的是什么Paddle的版本，CPU还是GPU环境，GPU的话CUDA版本是多少

Paddle2.0，GPU环境，CUDA11.0（AIStudio内环境）

收到，我问下API相关负责人

@huangjun12
Copy link
Contributor

出错部分代码：
class _routing(nn.Layer):

def __init__(self, in_channels, num_experts, dropout_rate):
    super(_routing, self).__init__()
    
    self.dropout = nn.Dropout(p=dropout_rate)
    self.fc = nn.Linear(in_channels, num_experts)

def forward(self, x):
    x = paddle.flatten(x).clone()
    x = self.dropout(x)
    x = self.fc(x)
    return F.sigmoid(x)

可以正常训练，但验证时报错
报错信息：
Traceback (most recent call last):
File "train.py", line 145, in
main(args)
File "train.py", line 140, in main
losses=losses)
File "/home/aistudio/work/PaddleSeg/paddleseg/core/train.py", line 183, in train
model, val_dataset, num_workers=num_workers)
File "/home/aistudio/work/PaddleSeg/paddleseg/core/val.py", line 110, in evaluate
crop_size=crop_size)
File "/home/aistudio/work/PaddleSeg/paddleseg/core/infer.py", line 169, in inference
logits = model(im)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/unet.py", line 63, in forward
x, short_cuts = self.encode(x)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/unet.py", line 95, in forward
x = self.double_conv(x)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/container.py", line 86, in forward
input = layer(input)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/layers/layer_libs.py", line 47, in forward
x = self._conv(x)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/layers/CondConv.py", line 77, in forward
routing_weights = self._routing_fn(pooled_inputs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/PaddleSeg/paddleseg/models/layers/CondConv.py", line 19, in forward
x = self.dropout(x)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call
outputs = self.forward(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/common.py", line 689, in forward
name=self.name)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/functional/common.py", line 924, in dropout
if seed is not None else 0, 'dropout_implementation', mode)
ValueError: (InvalidArgument) Input dimension number(num_col_dims) must be between 0 and 1, but received number is 1.
[Hint: Expected (num_col_dims > 0 && num_col_dims < rank) == true, but received (num_col_dims > 0 && num_col_dims < rank):0 != true:1.] (at /paddle/paddle/fluid/framework/eigen.h:94)
[operator < dropout > error]
terminate called without an active exception

C++ Traceback (most recent call last):

0 paddle::framework::SignalHandle(char const*, int)
1 paddle::platform::GetCurrentTraceBackStringabi:cxx11

Error Message Summary:

FatalError: Process abort signal is detected by the operating system.
[TimeInfo: *** Aborted at 1613917048 (unix time) try "date -d @1613917048" if you are using GNU date ***]
[SignalInfo: *** SIGABRT (@0x3e8000028cc) received by PID 10444 (TID 0x7f9180ffd700) from PID 10444 ***]

看起来时数据问题，检查一下train和eval时dropout的输入数据维度一样嘛

@baiwei-yndx
Copy link

建议看一下reader.yml 评估模块drop_empty: ture（同样问题，我原来为drop_empty: false改为ture即可）

@paddle-bot-old
Copy link

Since you haven't replied for more than a year, we have closed this issue/pr.
If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up.
由于您超过一年未回复，我们将关闭这个issue/pr。
若问题未解决或有后续问题，请随时重新打开，我们会继续跟进。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ysh329 ysh329

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ysh329 @huangjun12 @sellinaxmj @baiwei-yndx @yangkerrr