目前的paddle框架不支持导出半精度ONNX模型 #57194

vvwomen · 2023-09-11T10:17:49Z

需求描述 Feature Description

paddle框架提供的paddle.onnx.export 不支持导出半精度ONNX模型，会直接报错[ERROR] Float16 is not supported.

paddle.onnx.export最小的实现示例：
`
import paddle

from paddlenlp.transformers import UIEX # 从模型代码中导入模型

model = UIEX.from_pretrained("uie-x-base") # 实例化模型

model.to(dtype="float16") # 加载预训练模型参数

model.eval() # 将模型设置为评估状态

input_spec = [
paddle.static.InputSpec(shape=[None, None], dtype="int64", name="input_ids"),
paddle.static.InputSpec(shape=[None, None], dtype="int64", name="token_type_ids"),
paddle.static.InputSpec(shape=[None, None], dtype="int64", name="position_ids"),
paddle.static.InputSpec(shape=[None, None], dtype="int64", name="attention_mask"),
paddle.static.InputSpec(shape=[None, None, 4], dtype="int64", name="bbox"),
paddle.static.InputSpec(shape=[None, 3, 224, 224], dtype="float16", name="image"),
] # # 定义输入数据

print("Exporting ONNX model to %s" % "./uiex_fp16.onnx")
paddle.onnx.export(model, "./uiex_fp16", input_spec=input_spec) # ONNX模型导出
print("ONNX model exported.")
`

同paddle2onnx也不支持导出半精度ONNX模型，paddle2onnx最小的实现示例：

`
paddle2onnx --model_dir ./paddle_model_static_onnx_temp_dir/ --model_filename model.pdmodel --params_filename model.pdiparams --save_file ./bs4_paddle2onnx.onnx --opset_version 11

`
其中，model及参数均为fp16数据类型

网上提供的工具https://zenn.dev/pinto0309/scraps/588ed8342e2182 将FP32的onnx模型转为FP16后，此FP16的onnx模型存在问题不能在onnxruntime上使用。

补充信息:

paddle版本信息

paddle.onnx.export 报错信息

paddle2onnx的报错信息

`
[Paddle2ONNX] Start to parse PaddlePaddle model...
[Paddle2ONNX] Model file path: ./paddle_model_static_onnx_temp_dir/model.pdmodel
[Paddle2ONNX] Paramters file path: ./paddle_model_static_onnx_temp_dir/model.pdiparams
[Paddle2ONNX] Start to parsing Paddle model...
[ERROR] Float16 is not supported.

`

Bobholamovic · 2023-09-12T02:03:01Z

你好，目前Paddle确实暂时不支持导出半精度ONNX模型。我们有计划在近期对这个功能进行开发。感谢提议！

Zheng-Bicheng · 2023-09-25T16:10:40Z

@vvwomen 您好，您这个fp16的模型在什么位置下载的呀

Zheng-Bicheng · 2023-09-25T16:21:03Z

我用如下方法是没有办法导出Paddle静态图模型的：

import paddle

from paddlenlp.transformers import UIEX # 从模型代码中导入模型

model = UIEX.from_pretrained("uie-x-base") # 实例化模型

model.to(dtype="float16") # 加载预训练模型参数

model.eval() # 将模型设置为评估状态

model = paddle.jit.to_static(model)

input_spec = [
paddle.static.InputSpec(shape=[None, None], dtype="int64", name="input_ids"),
paddle.static.InputSpec(shape=[None, None], dtype="int64", name="token_type_ids"),
paddle.static.InputSpec(shape=[None, None], dtype="int64", name="position_ids"),
paddle.static.InputSpec(shape=[None, None], dtype="int64", name="attention_mask"),
paddle.static.InputSpec(shape=[None, None, 4], dtype="int64", name="bbox"),
paddle.static.InputSpec(shape=[None, 3, 224, 224], dtype="float16", name="image"),
]

paddle.jit.save(model,"./model",input_spec=input_spec)

vvwomen · 2023-09-26T01:28:23Z

paddle提供的API是不支持生成FP16的静态图的。你可以先生成FP32的静态图，然后使用convert_to_mixed_precision 将FP32的静态图转为FP16的静态图即可。

Zheng-Bicheng · 2023-09-26T02:08:21Z

需求描述 Feature Description

paddle框架提供的paddle.onnx.export 不支持导出半精度ONNX模型，会直接报错[ERROR] Float16 is not supported.

paddle.onnx.export最小的实现示例： ` import paddle

from paddlenlp.transformers import UIEX # 从模型代码中导入模型

model = UIEX.from_pretrained("uie-x-base") # 实例化模型

model.to(dtype="float16") # 加载预训练模型参数

model.eval() # 将模型设置为评估状态

input_spec = [ paddle.static.InputSpec(shape=[None, None], dtype="int64", name="input_ids"), paddle.static.InputSpec(shape=[None, None], dtype="int64", name="token_type_ids"), paddle.static.InputSpec(shape=[None, None], dtype="int64", name="position_ids"), paddle.static.InputSpec(shape=[None, None], dtype="int64", name="attention_mask"), paddle.static.InputSpec(shape=[None, None, 4], dtype="int64", name="bbox"), paddle.static.InputSpec(shape=[None, 3, 224, 224], dtype="float16", name="image"), ] # # 定义输入数据

print("Exporting ONNX model to %s" % "./uiex_fp16.onnx") paddle.onnx.export(model, "./uiex_fp16", input_spec=input_spec) # ONNX模型导出 print("ONNX model exported.") `

同paddle2onnx也不支持导出半精度ONNX模型，paddle2onnx最小的实现示例：

` paddle2onnx --model_dir ./paddle_model_static_onnx_temp_dir/ --model_filename model.pdmodel --params_filename model.pdiparams --save_file ./bs4_paddle2onnx.onnx --opset_version 11

` 其中，model及参数均为fp16数据类型

网上提供的工具https://zenn.dev/pinto0309/scraps/588ed8342e2182 将FP32的onnx模型转为FP16后，此FP16的onnx模型存在问题不能在onnxruntime上使用。

补充信息:

paddle版本信息

paddle.onnx.export 报错信息

paddle2onnx的报错信息

` [Paddle2ONNX] Start to parse PaddlePaddle model... [Paddle2ONNX] Model file path: ./paddle_model_static_onnx_temp_dir/model.pdmodel [Paddle2ONNX] Paramters file path: ./paddle_model_static_onnx_temp_dir/model.pdiparams [Paddle2ONNX] Start to parsing Paddle model... [ERROR] Float16 is not supported.

`

我的报错和您不一致，我这边出现的错误如下：

Traceback (most recent call last):
  File "D:\PycharmProjects\SupportFP16\main.py", line 21, in <module>
    paddle.onnx.export(model, "./uiex_fp16", input_spec=input_spec) # ONNX模型导出
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\onnx\export.py", line 101, in export
    p2o.dygraph2onnx(
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle2onnx\convert.py", line 56, in dygraph2onnx
    paddle.jit.save(layer, os.path.join(paddle_model_dir, "model"), input_spec)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\fluid\wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\jit\api.py", line 752, in wrapper
    func(layer, path, input_spec, **configs)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\fluid\wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\fluid\dygraph\base.py", line 75, in __impl__
    return func(*args, **kwargs)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\jit\api.py", line 1065, in save
    static_forward.concrete_program_specify_input_spec(
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\jit\dy2static\program_translator.py", line 709, in concrete_program_specify_input_spec
    concrete_program, _ = self.get_concrete_program(
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\jit\dy2static\program_translator.py", line 589, in get_concrete_program
    concrete_program, partial_program_layer = self._program_cache[
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\jit\dy2static\program_translator.py", line 1249, in __getitem__
    self._caches[item_id] = self._build_once(item)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\jit\dy2static\program_translator.py", line 1193, in _build_once
    concrete_program = ConcreteProgram.from_func_spec(
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\fluid\wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\fluid\dygraph\base.py", line 75, in __impl__
    return func(*args, **kwargs)
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\jit\dy2static\program_translator.py", line 1070, in from_func_spec
    error_data.raise_new_exception()
  File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\jit\dy2static\error.py", line 452, in raise_new_exception
    raise new_exception from None
TypeError: In transformed code:

    File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddlenlp\transformers\ernie_layout\modeling.py", line 1169, in forward
	sequence_output, _ = self.ernie_layout(
    File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddlenlp\transformers\ernie_layout\modeling.py", line 781, in forward
            position_ids=visual_position_ids,
        )
        final_emb = paddle.concat([text_layout_emb, visual_emb], axis=1)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE

        extended_attention_mask = final_attention_mask.unsqueeze(1).unsqueeze(2)

    File "D:\miniconda3\envs\support_fp16\lib\site-packages\paddle\tensor\manipulation.py", line 1143, in concat
	raise TypeError(

    TypeError: All the Tensors in the input must have the same data type.


进程已结束，退出代码为 1

Zheng-Bicheng · 2023-09-26T02:08:42Z

您的环境是在哪一个版本的paddle下测试的

vvwomen · 2023-09-26T02:23:54Z

我觉得不是环境的问题，应该是paddlenlp套件里的 paddlenlp/transformers/ernie_layout/modeling.py 文件中的 _calc_img_embeddings 函数中有这样一句 visual_embeddings = self.visual_act_fn(self.visual_proj(self.visual(image.astype(paddle.float32)))) 如果你传的image是这样的paddle.static.InputSpec(shape=[None, 3, 224, 224], dtype="float16", name="image")，那此时fp16的image经过 _calc_img_embeddings 的那一句会被转为float32，然后再放到resnet50中计算，而加载的模型权重是fp16的，就造成了输入数据和模型权重的数据类型不一致的问题，因此产生上述报错。或许你可以手动将_calc_img_embeddings 函数中的这句image.astype(paddle.float32) 改为 image.astype(paddle.float16) .

Zheng-Bicheng · 2023-09-26T02:33:27Z

paddle提供的API是不支持生成FP16的静态图的。你可以先生成FP32的静态图，然后使用convert_to_mixed_precision 将FP32的静态图转为FP16的静态图即可。

我还简单测试了一下convert_to_mixed_precision，这个API似乎也没办法直接将模型转换成FP16。我输入的模型是ResNet-18，转换后节点参数的类型还是fp32。

from paddle.inference import PrecisionType, PlaceType
from paddle.inference import convert_to_mixed_precision
import argparse

if __name__ == "__main__":
    black_list = set()

    src_model = "ResNet18_infer/fp32_inference/inference.pdmodel"
    src_params = "ResNet18_infer/fp32_inference/inference.pdiparams"
    dst_model = "./ResNet18_infer/fp16_inference/inference.pdmodel"
    dst_params = "./ResNet18_infer/fp16_inference/inference.pdiparams"

    convert_to_mixed_precision(
        src_model,     # fp32模型文件路径
        src_params,    # fp32权重文件路径
        dst_model,     # 混合精度模型文件保存路径
        dst_params,    # 混合精度权重文件保存路径
        PrecisionType.Half, # 转换精度，如 PrecisionType.Half
        PlaceType.GPU,      # 后端，如 PlaceType.GPU
        False,               # 保留输入输出精度信息，若为 True 则输入输出保留为 fp32 类型，否则转为 precision 类型
        black_list          # 黑名单列表，哪些 op 不需要进行精度类型转换
    )

vvwomen · 2023-09-26T02:54:17Z

我是这样的
convert_to_mixed_precision( model_file="./model.pdmodel", # fp32模型文件路径 params_file="./model.pdiparams", # fp32权重文件路径 mixed_model_file="./model_half.pdmodel", # 混合精度模型文件保存路径 mixed_params_file="./model_half.pdiparams", # 混合精度权重文件保存路径 mixed_precision=PrecisionType.Half, # 转换精度，如 PrecisionType.Half backend=PlaceType.GPU, # 后端，如 PlaceType.GPU keep_io_types=True, # 保留输入输出精度信息，若为 True 则输入输出保留为 fp32 类型，否则转为 precision 类型 )

因为 keep_io_types=True 所以我的image仍然是fp32的格式，但是我这

可以看到中间的conv2d节点，其输入输出均为fp16了，所以我这里是正常的

Zheng-Bicheng · 2023-09-28T02:38:36Z

会在下面的PR中修复Paddle2ONNX 无法原生导出FP16模型的Bug

[Hackathon No.82] Support fp16 Paddle2ONNX#1151

Zheng-Bicheng · 2023-09-28T02:59:46Z

@vvwomen 大佬您好，这个模型的输入你能给一个例子吗？我测试一下

vvwomen · 2023-09-28T03:27:59Z

我构造个随机数的例子吧：
input_ids = paddle.randint(0, 1000, (8, 512), dtype=paddle.int64) token_type_ids = paddle.randint(0, 1000, (8, 512), dtype=paddle.int64) position_ids = paddle.randint(0, 1000, (8, 512), dtype=paddle.int64) attention_mask = paddle.ones((8, 512), dtype=paddle.int64) bbox = paddle.randint(0, 1000, (8, 512, 4), dtype=paddle.int64) image = paddle.cast(paddle.rand((8, 3, 224, 224), dtype=paddle.float32),"float16") inpuda_dict={ "input_ids":input_ids, "token_type_ids": token_type_ids, "position_ids": position_ids, "attention_mask": attention_mask, "bbox": bbox, "image": image, }

Zheng-Bicheng · 2023-09-28T03:29:48Z

我构造个随机数的例子吧： input_ids = paddle.randint(0, 1000, (8, 512), dtype=paddle.int64) token_type_ids = paddle.randint(0, 1000, (8, 512), dtype=paddle.int64) position_ids = paddle.randint(0, 1000, (8, 512), dtype=paddle.int64) attention_mask = paddle.ones((8, 512), dtype=paddle.int64) bbox = paddle.randint(0, 1000, (8, 512, 4), dtype=paddle.int64) image = paddle.cast(paddle.rand((8, 3, 224, 224), dtype=paddle.float32),"float16") inpuda_dict={ "input_ids":input_ids, "token_type_ids": token_type_ids, "position_ids": position_ids, "attention_mask": attention_mask, "bbox": bbox, "image": image, }

好的感谢，我是做CV的，不了解NLP

paddle-bot · 2024-10-01T06:33:32Z

Since you haven't replied for more than a year, we have closed this issue/pr.
If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up.
由于您超过一年未回复，我们将关闭这个issue/pr。
若问题未解决或有后续问题，请随时重新打开，我们会继续跟进。

vvwomen added status/new-issue 新建 type/feature-request 新需求申请 labels Sep 11, 2023

paddle-bot bot assigned Bobholamovic Sep 11, 2023

tiandou-tangdou mentioned this issue Sep 11, 2023

Paddle2ONNX不支持导出半精度ONNX模型 PaddlePaddle/Paddle2ONNX#1148

Open

Bobholamovic added the status/following-up 跟进中 label Sep 12, 2023

paddle-bot bot removed the status/new-issue 新建 label Sep 12, 2023

Bobholamovic added the status/discussing 需求调研中 label Sep 12, 2023

paddle-bot bot removed the status/following-up 跟进中 label Sep 12, 2023

paddle-bot bot closed this as completed Oct 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

目前的paddle框架不支持导出半精度ONNX模型 #57194

目前的paddle框架不支持导出半精度ONNX模型 #57194

vvwomen commented Sep 11, 2023 •

edited

Loading

Bobholamovic commented Sep 12, 2023

Zheng-Bicheng commented Sep 25, 2023

Zheng-Bicheng commented Sep 25, 2023 •

edited

Loading

vvwomen commented Sep 26, 2023

Zheng-Bicheng commented Sep 26, 2023

需求描述 Feature Description

补充信息:

paddle版本信息

paddle.onnx.export 报错信息

paddle2onnx的报错信息

Zheng-Bicheng commented Sep 26, 2023

vvwomen commented Sep 26, 2023

Zheng-Bicheng commented Sep 26, 2023 •

edited

Loading

vvwomen commented Sep 26, 2023

Zheng-Bicheng commented Sep 28, 2023

Zheng-Bicheng commented Sep 28, 2023

vvwomen commented Sep 28, 2023

Zheng-Bicheng commented Sep 28, 2023

paddle-bot bot commented Oct 1, 2024

目前的paddle框架不支持导出半精度ONNX模型 #57194

目前的paddle框架不支持导出半精度ONNX模型 #57194

Comments

vvwomen commented Sep 11, 2023 • edited Loading

需求描述 Feature Description

补充信息:

paddle版本信息

paddle.onnx.export 报错信息

paddle2onnx的报错信息

Bobholamovic commented Sep 12, 2023

Zheng-Bicheng commented Sep 25, 2023

Zheng-Bicheng commented Sep 25, 2023 • edited Loading

vvwomen commented Sep 26, 2023

Zheng-Bicheng commented Sep 26, 2023

需求描述 Feature Description

补充信息:

paddle版本信息

paddle.onnx.export 报错信息

paddle2onnx的报错信息

Zheng-Bicheng commented Sep 26, 2023

vvwomen commented Sep 26, 2023

Zheng-Bicheng commented Sep 26, 2023 • edited Loading

vvwomen commented Sep 26, 2023

Zheng-Bicheng commented Sep 28, 2023

Zheng-Bicheng commented Sep 28, 2023

vvwomen commented Sep 28, 2023

Zheng-Bicheng commented Sep 28, 2023

paddle-bot bot commented Oct 1, 2024

vvwomen commented Sep 11, 2023 •

edited

Loading

Zheng-Bicheng commented Sep 25, 2023 •

edited

Loading

Zheng-Bicheng commented Sep 26, 2023 •

edited

Loading