From c76ddb3e8a40503a7fee5a615de8047cf1e6932b Mon Sep 17 00:00:00 2001 From: zachary sun <70642955+sunzhongkai588@users.noreply.github.com> Date: Tue, 2 Jul 2024 16:58:00 +0800 Subject: [PATCH 01/16] fix (#6740) --- docs/api/paddle/fluid/Overview_cn.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/api/paddle/fluid/Overview_cn.rst b/docs/api/paddle/fluid/Overview_cn.rst index 44fe15b4332..618a71b9566 100644 --- a/docs/api/paddle/fluid/Overview_cn.rst +++ b/docs/api/paddle/fluid/Overview_cn.rst @@ -3,4 +3,4 @@ paddle.fluid --------------------- .. warning:: - 从飞桨框架 2.5 版本开始，我们已经废弃了 `paddle.fluid` namespace 下的 API，请使用其他的替代 API。 + 从飞桨框架 2.5 版本开始，我们已经废弃了 `paddle.fluid` namespace 下的 API，请使用其他的替代 API，请参考 :ref:`cn_guides_api_mapping` 获取更多信息。如有疑问，请于 `Issue 区 `_ 按规范提交 Issue。 From 702912bd9b5ce5380138c3293802bb281305c431 Mon Sep 17 00:00:00 2001 From: zhwesky2010 <1183042833@qq.com> Date: Wed, 3 Jul 2024 15:39:02 +0800 Subject: [PATCH 02/16] add introduction of paconvert (#6741) --- .../convert_from_pytorch/index_cn.rst | 7 +- .../paconvert_introduction_cn.md | 184 ++++++++++++++++++ docs/guides/model_convert/index_cn.rst | 3 +- 3 files changed, 190 insertions(+), 4 deletions(-) create mode 100644 docs/guides/model_convert/convert_from_pytorch/paconvert_introduction_cn.md diff --git a/docs/guides/model_convert/convert_from_pytorch/index_cn.rst b/docs/guides/model_convert/convert_from_pytorch/index_cn.rst index c9d82028683..cc44f001d6f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/index_cn.rst +++ b/docs/guides/model_convert/convert_from_pytorch/index_cn.rst @@ -4,24 +4,25 @@ 您可以通过下面的内容，如何将 PyTorch 训练代码迁移到飞桨: - +- `代码自动转换工具 <./paconvert_introduction_cn.html>`_ : 介绍 Pytorch 代码自动转 Paddle 工具使用方法。 +- `PyTorch API 映射表 <./pytorch_api_mapping_cn.html>`_ : 说明 PyTorch 最新 release 版本 与 Paddle develop 版本 API 对应关系。 - `CV - 快速上手 <./cv_quick_start_cn.html>`_ : 以 MobileNetV3 为例，介绍如何从 PyTorch 迁移到飞桨。 - `CV - 迁移经验总结 <./cv_experience_cn.html>`_ : 介绍 CV 各个方向从 PyTorch 迁移到飞桨的基本流程、常用工具、定位问题的思路及解决方法。 - `NLP - 快速上手 <./nlp_fast_explore_cn.html>`_ : 以 Bert 为例，介绍如何从 PyTorch 迁移到飞桨。 - `NLP - 迁移经验总结 <./nlp_migration_experiences_cn.html>`_ : 介绍 NLP 各个方向从 PyTorch 迁移到飞桨的基本流程、常用工具、定位问题的思路及解决方法。 - `解读网络结构转换 <./convert_net_structure_cn.html>`_ : 介绍网络结构转换的思路和方法。 - `解读 Bert 模型权重转换 <./convert_bert_weights_cn.html>`_ : 介绍如何进行不同框架下的模型权重转换。 -- `PyTorch API 映射表 <./pytorch_api_mapping_cn.html>`_ : 说明 PyTorch 2.1.0 版本与 Paddle develop 版本 API 对应关系。 - `PyTorch 自定义算子转写教程 <./pytorch_custom_op_convert_cn.html>`_ : 介绍 PyTorch 中自定义算子转写成 Paddle 自定义算子的思路和方法。 .. toctree:: :hidden: + paconvert_introduction_cn.md + pytorch_api_mapping_cn.md cv_quick_start_cn.md cv_experience_cn.md nlp_fast_explore_cn.md nlp_migration_experiences_cn.md convert_net_structure_cn.md convert_bert_weights_cn.md - pytorch_api_mapping_cn.md pytorch_custom_op_convert_cn.md diff --git a/docs/guides/model_convert/convert_from_pytorch/paconvert_introduction_cn.md b/docs/guides/model_convert/convert_from_pytorch/paconvert_introduction_cn.md new file mode 100644 index 00000000000..74777dca163 --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/paconvert_introduction_cn.md @@ -0,0 +1,184 @@ +# 代码自动转换工具 + +![](https://img.shields.io/badge/version-v2.0-brightgreen) ![](https://img.shields.io/badge/docs-latest-brightgreen) ![](https://img.shields.io/badge/PRs-welcome-brightgreen) ![](https://img.shields.io/badge/pre--commit-Yes-brightgreen) + +**[PaConvert Github](https://github.com/PaddlePaddle/PaConvert)** + +**Pa**ddlePaddle Code **Convert** Toolkits + +## 🤗 公告 🤗 +- 本工具由 Paddle 官方团队维护与建设，所有转换代码均已经过测试，欢迎使用，高效迁移 Pytorch 代码到 PaddlePaddle + +- 当前共支持约 1300+个 Pytorch API 的一键转换，我们通过 300+个 Pytorch 模型测试，代码行数平均转换率约为 **90+%** + +- 本工具基于 [PyTorch 最新 release 与 Paddle develop API 映射表](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.html) 实现，表中 API 均经过详细对比分析，欢迎查阅 + +- 有使用问题和建议欢迎在 [PaConvert GitHub Issues](https://github.com/PaddlePaddle/PaConvert/issues) 中提出 + +## 概述 + +`代码自动转换工具` 能自动将其它深度学习框架训练或推理的**代码**，转换为 PaddlePaddle 的**代码**，方便快速自动地 **模型代码迁移**。 + +目前仅支持自动转换 Pytorch 代码，其它深度学习框架的支持后续新增中，转换时会尽量保持原代码的风格与结构，将其它深度学习框架的 API 接口 转换为 PaddlePaddle 的 API 接口。 + +转换过程中不会改动原文件，会将原项目中的文件一一转换到 `out_dir` 文件夹中（如不指定`out_dir`，则默认在当前目录下新建`paddle_project/`）。对不同类型的文件的处理逻辑分别为： + +- Python 代码文件：识别代码中调用其它深度学习框架的接口并转换为 PaddlePaddle 的接口 +- requirements.txt： 替换其中的安装依赖为 `paddlepaddle-gpu` +- 其他文件：原样拷贝 + +## 安装与使用 + +由于使用了一些较新的 Python 功能特性，你需要使用 `>=python3.8` 的解释器。 + +1. 使用 pip 安装 + +```bash +pip install -U paconvert +paconvert --in_dir torch_project [--out_dir paddle_project] [--exclude_dirs exclude_dirs] [--log_dir log_dir] [--log_level "DEBUG"] [--run_check 1] +``` + +2. 使用源码安装 + +```bash +git clone https://github.com/PaddlePaddle/PaConvert.git +python paconvert/main.py --in_dir torch_project [--out_dir paddle_project] [--exclude_dirs exclude_dirs] [--log_dir log_dir] [--log_level "DEBUG"] [--run_check 1] +``` + +**参数介绍** + +``` +--in_dir 输入 torch 项目文件，可以为单个文件或文件夹 +--out_dir 可选，输出 paddle 项目文件，可以为单个文件或文件夹，默认在当前目录下创建./paddle_project/ +--exclude_dirs 可选，排除转换的文件或文件夹，排除多项时请用逗号分隔，默认不排除 +--log_dir 可选，输出日志的路径，默认会在终端上打印日志 +--log_level 可选，打印 log 等级，仅支持"INFO" "DEBUG"，默认"INFO" +--run_check 可选，工具自检 +``` + + +## 转换示例 + +以下面 Pytorch 代码为例，转换前： +``` +import torch +import torch.nn as nn +import torch.optim as optim +import torch.nn.Linear as Linear +import torch.nn.functional as F + +class MyNet(nn.Module): + test = "str" + + def __init__(self): + self._fc1 = torch.nn.Linear(10, 10) + self._fc2 = nn.Linear(10, 10) + self._fc3 = Linear(10, 10) + + @torch.no_grad() + def forward(self, x): + x = self._fc1(x) + x = self._fc2(x) + x = self._fc3(x) + y = torch.add(x, x) + return F.relu(y) + +net = MyNet() + +sgd = optim.SGD(net.parameters(), lr=0.1, momentum=0.9) +lr = optim.lr_scheduler.MultiStepLR(sgd, milestones=[2, 4, 6], gamma=0.8) +``` + +转换后： +``` +import paddle + + +class MyNet(paddle.nn.Layer): + test = 'str' + + def __init__(self): + self._fc1 = paddle.nn.Linear(in_features=10, out_features=10) + self._fc2 = paddle.nn.Linear(in_features=10, out_features=10) + self._fc3 = paddle.nn.Linear(in_features=10, out_features=10) + + @paddle.no_grad() + def forward(self, x): + x = self._fc1(x) + x = self._fc2(x) + x = self._fc3(x) + y = paddle.add(x=x, y=paddle.to_tensor(x)) + return paddle.nn.functional.relu(x=y) + + +net = MyNet() +>>>>>>sgd = torch.optim.SGD(net.parameters(), lr=0.1, momentum=0.9) +tmp_lr = paddle.optimizer.lr.MultiStepDecay(milestones=[2, 4, 6], gamma=0.8, + learning_rate=sgd.get_lr()) +sgd.set_lr_scheduler(tmp_lr) +lr = tmp_lr +``` + +打印信息如下： + +```text +=========================================== +PyTorch to Paddle Convert Start ------>: +=========================================== +Start convert file: /workspace/PaConvert/test_code.py --> /workspace/PaConvert/paddle_project/test_code.py +[test_code.py:1] remove 'import torch' +[test_code.py:2] remove 'import torch.nn as nn' +[test_code.py:3] remove 'import torch.optim as optim' +[test_code.py:4] remove 'import torch.nn.Linear as Linear' +[test_code.py:5] remove 'import torch.nn.functional as F' +[test_code.py] add 'import paddle' in line 1 +[test_code.py:1] [Success] Convert torch.nn.Module to Paddle +[test_code.py:11] [Success] Convert torch.nn.Linear to Paddle +[test_code.py:12] [Success] Convert torch.nn.Linear to Paddle +[test_code.py:13] [Success] Convert torch.nn.Linear to Paddle +[test_code.py:20] [Success] Convert torch.add to Paddle +[test_code.py:21] [Success] Convert torch.nn.functional.relu to Paddle +[test_code.py:15] [Success] Convert torch.no_grad to Paddle +[test_code.py:25] [Success] Convert Class Method: torch.nn.Module.parameters to Paddle +[test_code.py:25] [Not Support] convert torch.optim.SGD to Paddle is not supported currently +[test_code.py:26] [Success] Convert torch.optim.lr_scheduler.MultiStepLR to Paddle +Finish convert /workspace/PaConvert/test_code.py --> /workspace/PaConvert/paddle_project/test_code.py + + +=========================================== +Convert Summary: +=========================================== +There are 10 Pytorch APIs in this Project: + 9 Pytorch APIs have been converted to Paddle successfully! + 1 Pytorch APIs are not supported to convert to Paddle currently! + Convert Rate is: 90.000% + +For these 1 Pytorch APIs that currently do not support to convert, which have been marked by >>> before the line, +please refer to [https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.html] +and convert it by yourself manually. In addition, these APIs will be supported in future. + +Thank you to use Paddle Code Convert Tool. You can make any suggestions +to us by submitting issues to [https://github.com/PaddlePaddle/PaConvert]. + +**************************************************************** +______ _____ _ +| ___ \ / ____| | | +| |_/ /_ _| | ___ _ ____ _____ _ __| |_ +| __/ _ | | / _ \| \_ \ \ / / _ \ \__| __| +| | | (_| | |___| (_) | | | \ V / __/ | | |_ +\_| \__,_|\_____\___/|_| |_|\_/ \___|_| \__| + +*************************************************************** + +``` + +转换完成后，会打印 **转换总结** ，包含 **总 API 数、成功转换 API 数、不支持转换 API 数、转化率** 。例如，上述代码里一共有 10 个 Pytorch API，其中 9 个被成功转换，1 个不支持转换，因此转换率为 `90.00%` 。 + +**对于成功转换的 API**：代码风格会略有变化，会 **补全 API 全名、补全参数关键字、移除注释、移除多余空行** 。因为在代码识别的过程中，**注释、空行** 等无法识别。 + +**对于不支持转换的 API**：将 **补全为 Pytorch API 全名**，同时在行前通过 `>>>>>>` 的形式加以标记，用户需要对该 API 进行人工手动转换，然后删除 `>>>>>>` 标记，否则代码无法运行。 + + +## 贡献代码 + +代码自动转换工具（[PaConvert](https://github.com/PaddlePaddle/PaConvert)）为开源贡献形式，欢迎你向我们贡献代码，详细开发步骤请参考 [贡献代码教程](https://github.com/PaddlePaddle/PaConvert/blob/master/docs/CONTRIBUTING.md) diff --git a/docs/guides/model_convert/index_cn.rst b/docs/guides/model_convert/index_cn.rst index b440f95ea35..fdb6e037212 100644 --- a/docs/guides/model_convert/index_cn.rst +++ b/docs/guides/model_convert/index_cn.rst @@ -7,13 +7,14 @@ - `迁移指南 <./convert_guide_cn.html>`_ : 介绍模型迁移场景及概览。 - `从 PyTorch 迁移到飞桨 <./convert_from_pytorch/index_cn.html>`_ : 介绍如何将 PyTorch 训练代码迁移到飞桨。 + - `代码自动转换工具 <./paconvert_introduction_cn.html>`_ : 介绍 Pytorch 代码自动转 Paddle 工具使用方法。 + - `PyTorch API 映射表 <./pytorch_api_mapping_cn.html>`_ : 说明 PyTorch 最新 release 版本 与 Paddle develop 版本 API 对应关系。 - `CV - 快速上手 <./convert_from_pytorch/cv_quick_start_cn.html>`_ : 以 MobileNetV3 为例，介绍如何从 PyTorch 迁移到飞桨。 - `CV - 迁移经验总结 <./convert_from_pytorch/cv_experience_cn.html>`_ : 介绍 CV 各个方向从 PyTorch 迁移到飞桨的基本流程、常用工具、定位问题的思路及解决方法。 - `NLP - 快速上手 <./convert_from_pytorch/nlp_fast_explore_cn.html>`_ : 以 Bert 为例，介绍如何从 PyTorch 迁移到飞桨。 - `NLP - 迁移经验总结 <./convert_from_pytorch/nlp_migration_experiences_cn.html>`_ : 介绍 NLP 各个方向从 PyTorch 迁移到飞桨的基本流程、常用工具、定位问题的思路及解决方法。 - `解读网络结构转换 <./convert_from_pytorch/convert_net_structure_cn.html>`_ : 介绍网络结构转换的思路和方法。 - `解读 Bert 模型权重转换 <./convert_from_pytorch/convert_bert_weights_cn.html>`_ : 介绍如何进行不同框架下的模型权重转换。 - - `PyTorch API 映射表 <./convert_from_pytorch/pytorch_api_mapping_cn.html>`_ : 说明 PyTorch 1.8 版本与 Paddle 2.0 API 对应关系。 - `PyTorch 自定义算子转写教程 <./pytorch_custom_op_convert_cn.html>`_ : 介绍 PyTorch 中自定义算子转写成 Paddle 自定义算子的思路和方法。 - `使用 X2Paddle 迁移推理模型 <./convert_with_x2paddle_cn.html>`_ : 介绍如何使用 X2Paddle 工具将 PyTorch、ONNX、TensorFlow、Caffe 推理模型迁移到飞桨。 - `迁移飞桨旧版本 <./convert_from_older_versions/index_cn.html>`_ : 介绍如何将飞桨 1.X 版本的训练代码与模型迁移到飞桨最新版。 From 93f49f3e022d7a0fc8e8ef0f479eb00ddc97ea69 Mon Sep 17 00:00:00 2001 From: Botao Zhou <1095497213@qq.com> Date: Mon, 8 Jul 2024 14:41:07 +0800 Subject: [PATCH 03/16] =?UTF-8?q?=E3=80=90Hackathon=206th=20No.28=E3=80=91?= =?UTF-8?q?=E4=B8=BA=20paddle.round=20=E8=BF=9B=E8=A1=8C=E5=8A=9F=E8=83=BD?= =?UTF-8?q?=E5=A2=9E=E5=BC=BA-=E4=B8=AD=E6=96=87=E6=96=87=E6=A1=A3?= =?UTF-8?q?=E4=BF=AE=E6=94=B9-part=20(#6749)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * add decimals for round * fix --- docs/api/paddle/round_cn.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/api/paddle/round_cn.rst b/docs/api/paddle/round_cn.rst index 4aa61d40d79..b63625b7986 100644 --- a/docs/api/paddle/round_cn.rst +++ b/docs/api/paddle/round_cn.rst @@ -25,6 +25,7 @@ round - **x** (Tensor) - 支持任意维度的 Tensor。数据类型为 float32，float64 或 float16。 + - **decimals** (int，可选) - 要舍入到的小数点位数。如果 decimals 为负数，则指定小数点左边的位数。默认为 0。 - **name** (str，可选) - 具体用法请参见 :ref:`api_guide_Name`，一般无需设置，默认值为 None。 返回 From 154eadbb568ef7663234f28e0e941fb7ba8104b4 Mon Sep 17 00:00:00 2001 From: huqi Date: Tue, 9 Jul 2024 11:24:47 +0800 Subject: [PATCH 04/16] =?UTF-8?q?[Docathon][Remove=20File=20No.47-55]?= =?UTF-8?q?=E6=B8=85=E7=90=86=E6=9C=AA=E6=9A=B4=E9=9C=B2=E5=9C=A8=20sphinx?= =?UTF-8?q?=20toctree=20=E4=B8=8B=E7=9A=84=E6=96=87=E6=A1=A3=20(#6632)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix resoure path * Update docs/dev_guides/index_en.rst Co-authored-by: zachary sun <70642955+sunzhongkai588@users.noreply.github.com> --------- Co-authored-by: zachary sun <70642955+sunzhongkai588@users.noreply.github.com> --- docs/dev_guides/index_en.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/dev_guides/index_en.rst b/docs/dev_guides/index_en.rst index a85b8b295a3..27286a73ca8 100644 --- a/docs/dev_guides/index_en.rst +++ b/docs/dev_guides/index_en.rst @@ -6,7 +6,7 @@ We very much welcome you to participate in the construction of the paddle. The f Similarly, if you feel that this document is missing, or that the description is unclear, we also welcome you to contribute to this series of documents. -- `Kernel Primitives API <./kernel_primitive_api/index_cn.html>`_ : Introduce the block-level CUDA functions provided by PaddlePaddle to speed up operator development. +- `Kernel Primitives API <./op_optimization/kernel_primitive_api/index_en.html>`_ : Introduce the block-level CUDA functions provided by PaddlePaddle to speed up operator development. - `custom_device_docs <./custom_device_docs/index_en.html>`_ : Contribution guidelines overview. @@ -14,5 +14,5 @@ Similarly, if you feel that this document is missing, or that the description is :hidden: - kernel_primitive_api/index_en.rst + op_optimization/kernel_primitive_api/index_en.rst custom_device_docs/index_en.rst From 1b4715ee95f77c7aa3e77fb499572ee1dec8ec00 Mon Sep 17 00:00:00 2001 From: Matsumoto Ruko Date: Tue, 9 Jul 2024 11:54:26 +0800 Subject: [PATCH 05/16] [Docathon][Add Overview Doc No.19-21] add doc of docathon 19-21 (#6504) * add doc of docathon 19-21 * fix some issues --- docs/api/paddle/nn/Overview_cn.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/api/paddle/nn/Overview_cn.rst b/docs/api/paddle/nn/Overview_cn.rst index bee31da0886..ee40272f162 100644 --- a/docs/api/paddle/nn/Overview_cn.rst +++ b/docs/api/paddle/nn/Overview_cn.rst @@ -205,6 +205,7 @@ Transformer 相关 " :ref:`paddle.nn.MultiHeadAttention ` ", "多头注意力机制" " :ref:`paddle.nn.functional.scaled_dot_product_attention ` ", "点乘注意力机制，并在此基础上加入了对注意力权重的缩放" + " :ref:`paddle.nn.functional.sparse_attention ` ", "稀疏版本的 Attention API，对 Transformer 模块中的 Attention 矩阵进行了稀疏化，从而减少内存消耗和计算量" " :ref:`paddle.nn.Transformer ` ", "Transformer 模型" " :ref:`paddle.nn.TransformerDecoder ` ", "Transformer 解码器" " :ref:`paddle.nn.TransformerDecoderLayer ` ", "Transformer 解码器层" @@ -264,6 +265,7 @@ Loss 层 " :ref:`paddle.nn.CrossEntropyLoss ` ", "交叉熵损失层" " :ref:`paddle.nn.CTCLoss ` ", "CTCLoss 层" " :ref:`paddle.nn.HSigmoidLoss ` ", "层次 sigmoid 损失层" + " :ref:`paddle.nn.HingeEmbeddingLoss ` ", "HingeEmbeddingLoss 损失层" " :ref:`paddle.nn.KLDivLoss ` ", "Kullback-Leibler 散度损失层" " :ref:`paddle.nn.L1Loss ` ", "L1 损失层" " :ref:`paddle.nn.MarginRankingLoss ` ", "MarginRankingLoss 层" @@ -271,6 +273,7 @@ Loss 层 " :ref:`paddle.nn.NLLLoss ` ", "NLLLoss 层" " :ref:`paddle.nn.GaussianNLLLoss ` ", "GaussianNLLLoss 层" " :ref:`paddle.nn.PoissonNLLLoss `", "PoissonNLLLoss 层" + " :ref:`paddle.nn.RNNTLoss `", "RNNTLoss 层" " :ref:`paddle.nn.SmoothL1Loss ` ", "平滑 L1 损失层" " :ref:`paddle.nn.SoftMarginLoss ` ", "SoftMarginLoss 层" " :ref:`paddle.nn.TripletMarginLoss ` ", "TripletMarginLoss 层" @@ -516,6 +519,8 @@ Embedding 相关函数 " :ref:`paddle.nn.functional.triplet_margin_loss ` ", "用于计算 TripletMarginLoss" " :ref:`paddle.nn.functional.triplet_margin_with_distance_loss ` ", "用户自定义距离函数用于计算 triplet margin loss 损失" " :ref:`paddle.nn.functional.multi_label_soft_margin_loss ` ", "用于计算多分类的 hinge loss 损失函数" + " :ref:`paddle.nn.functional.hinge_embedding_loss ` ", "计算输入 input 和标签 label（包含 1 和 -1） 间的 `hinge embedding loss` 损失" + " :ref:`paddle.nn.functional.rnnt_loss ` ", "计算 RNNT loss，也可以叫做 softmax with RNNT" " :ref:`paddle.nn.functional.multi_margin_loss ` ", "用于计算 multi margin loss 损失函数" " :ref:`paddle.nn.functional.adaptive_log_softmax_with_loss ` ", "自适应 logsoftmax 损失函数" @@ -548,6 +553,7 @@ Embedding 相关函数 " :ref:`paddle.nn.functional.temporal_shift ` ", "用于对输入 X 做时序通道 T 上的位移操作，为 TSM 中使用的操作" " :ref:`paddle.nn.functional.upsample ` ", "用于调整一个 batch 中图片的大小" " :ref:`paddle.nn.functional.class_center_sample ` ", "用于 PartialFC 类别中心采样" + " :ref:`paddle.nn.functional.channel_shuffle ` ", "将一个形为[N, C, H, W]或是[N, H, W, C]的 Tensor 按通道分成 g 组，得到形为[N, g, C/g, H, W]或[N, H, W, g, C/g]的 Tensor，然后转置为[N, C/g, g, H, W]或[N, H, W, C/g, g]的形状，最后重新排列为原来的形状" .. _about_initializer: From b76b0b49e4fc572ea3ee9c606c96a4a862d745f7 Mon Sep 17 00:00:00 2001 From: AyaseNana <49900969+NKNaN@users.noreply.github.com> Date: Wed, 10 Jul 2024 11:39:48 +0800 Subject: [PATCH 06/16] =?UTF-8?q?=E3=80=90Hackathon=206th=20No.9=E3=80=91A?= =?UTF-8?q?dd=20cartesian=5Fprod=20API=20to=20Paddle=20(#6751)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/api/paddle/Overview_cn.rst | 1 + docs/api/paddle/cartesian_prod_cn.rst | 28 +++++++++++++++++++++++++++ 2 files changed, 29 insertions(+) create mode 100644 docs/api/paddle/cartesian_prod_cn.rst diff --git a/docs/api/paddle/Overview_cn.rst b/docs/api/paddle/Overview_cn.rst index 4c65cbe931f..0705f61847e 100755 --- a/docs/api/paddle/Overview_cn.rst +++ b/docs/api/paddle/Overview_cn.rst @@ -399,6 +399,7 @@ tensor 元素操作相关（如：转置，reshape 等） " :ref:`paddle.broadcast_to ` ", "根据 shape 指定的形状广播 x ，广播后， x 的形状和 shape 指定的形状一致" " :ref:`paddle.broadcast_tensors ` ", "对一组输入 Tensor 进行广播操作, 输入应符合广播规范" + " :ref:`paddle.cartesian_prod ` ", "对输入 Tensor 序列进行笛卡尔积操作" " :ref:`paddle.cast ` ", "将输入的 x 的数据类型转换为 dtype 并输出" " :ref:`paddle.chunk ` ", "将输入 Tensor 分割成多个子 Tensor" " :ref:`paddle.concat ` ", "对输入沿 axis 轴进行联结，返回一个新的 Tensor" diff --git a/docs/api/paddle/cartesian_prod_cn.rst b/docs/api/paddle/cartesian_prod_cn.rst new file mode 100644 index 00000000000..a8af9bd3c13 --- /dev/null +++ b/docs/api/paddle/cartesian_prod_cn.rst @@ -0,0 +1,28 @@ +.. _cn_api_paddle_cartesian_prod: + +cartesian_prod +------------------------------- + +.. py:function:: paddle.cartesian_prod(x, name=None) + + +对指定的 tensor 序列进行笛卡尔积操作。该行为类似于 python 标准库中的 itertools.product 方法。 +相当于将所有的输入 tensors 转换为列表后，对其使用 itertools.product 方法，最终将返回的列表转换为 tensor。 + + +参数 +::::::::: + + - **x** (list[Tensor]|tuple[Tensor]) – 任意数量的 1-D Tensor 序列，支持的数据类型：bfloat16、float16、float32、float64、int32、int64、complex64、complex128。 + + - **name** (str，可选) - 具体用法请参见 :ref:`api_guide_Name`，一般无需设置，默认值为 None。 + +返回 +::::::::: + +笛卡尔积运算后的 Tensor，数据类型与输入 Tensor 相同。 + +代码示例 +:::::::::::: + +COPY-FROM: paddle.cartesian_prod From 8297ac379ebeabfbc3b422399737dbcb7ae0bddb Mon Sep 17 00:00:00 2001 From: RedContritio Date: Wed, 10 Jul 2024 11:44:51 +0800 Subject: [PATCH 07/16] upgrade pytorch-paddle mapping table generating process (#6752) * add table 'not implemented' in pytorch_api_mapping_cn.md * update pytorch_api_mapping_cn with REFERENCE-MAPPING-TABLE * upgrade apply_reference_from_api_difference to auto generate from api_difference/ --- .../apply_reference_from_api_difference.py | 240 ++- .../pytorch_api_mapping_cn.md | 1574 ++--------------- .../validate_mapping_in_api_difference.py | 54 +- 3 files changed, 296 insertions(+), 1572 deletions(-) diff --git a/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py b/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py index 3e106ed5ae5..6c34eda10ef 100644 --- a/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py +++ b/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py @@ -8,6 +8,7 @@ print(script_dir) from validate_mapping_in_api_difference import ( + DiffMeta, get_meta_from_diff_file, process_mapping_index as reference_mapping_item, ) @@ -64,9 +65,13 @@ def mapping_type_to_description(mapping_type): return "【未知类型】", False -REFERENCE_PATTERN = re.compile( +# 以后没有 REFERENCE-ITEM 需要维护了，全部从 api_difference/ 目录生成 +_REFERENCE_ITEM_PATTERN = re.compile( r"^\| *REFERENCE-MAPPING-ITEM\( *(?P[^,]+) *, *(?P.+) *\) *\|$" ) +REFERENCE_TABLE_PATTERN = re.compile( + r"^\| *REFERENCE-MAPPING-TABLE\( *(?P[^,]+) *(, *max_depth *= *(?P\d+) *)?\) *\|$" +) ALIAS_PATTERN = re.compile( r"^\| *ALIAS-REFERENCE-ITEM\( *(?P[^,]+) *, *(?P[^,]+) *\) *\|$" ) @@ -79,6 +84,7 @@ def mapping_type_to_description(mapping_type): def docs_url_to_relative_page(url): + """将映射文档的 PaddlePaddle/docs url 转换为网页路径""" if not url.startswith(DOCS_REPO_BASEURL): return url @@ -88,110 +94,101 @@ def docs_url_to_relative_page(url): return md_path -def apply_reference_to_row(line, metadata_dict, table_row_idx, line_idx): - reference_match = REFERENCE_PATTERN.match(line) - alias_match = ALIAS_PATTERN.match(line) - not_implemented_match = NOT_IMPLEMENTED_PATTERN.match(line) +def doc_path_to_relative_page(path): + """将映射文档的本地路径转换为网页相对路径""" + md_path = os.path.relpath(path, script_dir) - row_idx_s = str(table_row_idx) + assert md_path.endswith(".md"), f"Unexpected mapping doc path: {path}" - if reference_match: - torch_api = reference_match["torch_api"].strip("`").replace(r"\_", "_") - diff_url = reference_match["diff_url"] + return md_path[:-3] + ".html" - diff_page_url = docs_url_to_relative_page(diff_url) - if torch_api not in metadata_dict: - raise Exception( - f"Cannot find torch_api: {torch_api} in line {line_idx}" - ) +def reference_table_match_to_condition(m): + api_prefix = m["api_prefix"].strip("`") + max_depth = m["max_depth"] + if max_depth is None: + max_depth = 255 + else: + max_depth = int(max_depth) + return api_prefix, max_depth - meta_dict[torch_api]["diff_url"] = diff_page_url - reference_item = metadata_dict.get(torch_api, None) - torch_api_url = reference_item["torch_api_url"] - torch_api_column = f"[`{torch_api}`]({torch_api_url})" +def get_referenced_api_columns(torch_api, metadata_dict, alias=None): + assert ( + torch_api in metadata_dict + ), f'Error: cannot find mapping doc of api "{torch_api}"' + api_data: DiffMeta = metadata_dict[torch_api] - mapping_type = reference_item["mapping_type"] - mapping_type_column = mapping_type + diff_page_url = doc_path_to_relative_page(api_data["source_file"]) - _mapping_type_desc, show_diff_url = mapping_type_to_description( - mapping_type - ) - mapping_url_column = "" - if show_diff_url: - mapping_url_column = f"[详细对比]({diff_page_url})" - - if "paddle_api" not in reference_item: - if mapping_type not in ["组合替代实现", "可删除", "功能缺失"]: - print( - f"Cannot find paddle_api for torch_api: {torch_api} in line {line_idx}" - ) - paddle_api_column = "" - else: - paddle_api = reference_item["paddle_api"] - paddle_api_url = reference_item["paddle_api_url"] - paddle_api_column = f"[`{paddle_api}`]({paddle_api_url})" + torch_api_url = api_data["torch_api_url"] + api_disp_name = torch_api if alias is None else alias + torch_api_column = f"[`{api_disp_name}`]({torch_api_url})" - content = [ - row_idx_s, - torch_api_column, - paddle_api_column, - mapping_type_column, - mapping_url_column, - ] + mapping_type = api_data["mapping_type"] + mapping_type_column = mapping_type - output = "| " + " | ".join(content) + " |\n" - return output - elif alias_match: - alias_name = alias_match["alias_name"].strip("`").replace(r"\_", "_") - torch_api = alias_match["torch_api"].strip("`").replace(r"\_", "_") - - if torch_api not in metadata_dict: - raise Exception( - f"Cannot find torch_api: {torch_api} in line {line_idx}" - ) + _mapping_type_desc, show_diff_url = mapping_type_to_description( + mapping_type + ) + desc_column = "" + if show_diff_url: + desc_column = f"[详细对比]({diff_page_url})" + if alias is not None: + desc_column = f"`{torch_api}` 别名，{desc_column}" + + if "paddle_api" not in api_data: + if mapping_type not in ["组合替代实现", "可删除", "功能缺失"]: + print(f"Error: cannot find paddle_api for torch_api: {torch_api}") + paddle_api_column = "" + else: + paddle_api = api_data["paddle_api"] + paddle_api_url = api_data["paddle_api_url"] + paddle_api_column = f"[`{paddle_api}`]({paddle_api_url})" + + return [ + torch_api_column, + paddle_api_column, + mapping_type_column, + desc_column, + ] - diff_page_url = metadata_dict[torch_api].get("diff_url", "") - reference_item = metadata_dict.get(torch_api, None) - torch_api_url = reference_item["torch_api_url"] - alisa_name_column = f"[`{alias_name}`]({torch_api_url})" +def apply_reference_to_row_ex(line, metadata_dict, context, line_idx): + reference_table_match = REFERENCE_TABLE_PATTERN.match(line) + alias_match = ALIAS_PATTERN.match(line) + not_implemented_match = NOT_IMPLEMENTED_PATTERN.match(line) - mapping_type = reference_item["mapping_type"] - mapping_type_column = mapping_type + row_idx_s = str(context["table_row_idx"]) + + if reference_table_match: + condition = reference_table_match_to_condition(reference_table_match) + api_list = context["c2a_dict"][ + condition + ] # 这个键一定存在，否则说明前面出错了 + output_lines = [] + cur_row_idx = context["table_row_idx"] + for api in api_list: + content = get_referenced_api_columns(api, metadata_dict) + content.insert(0, str(cur_row_idx)) + output = "| " + " | ".join(content) + " |\n" + output_lines.append(output) + cur_row_idx += 1 + # 因为外面会给 table_row_idx 自动加 1，所以这里减去 1 + context["table_row_idx"] = cur_row_idx - 1 + return output_lines + elif alias_match: + alias_name = alias_match["alias_name"].strip("`").replace(r"\_", "_") + torch_api = alias_match["torch_api"].strip("`").replace(r"\_", "_") - _mapping_type_desc, show_diff_url = mapping_type_to_description( - mapping_type + content = get_referenced_api_columns( + torch_api, metadata_dict, alias=alias_name ) - desc_column = f"`{torch_api}` 别名" - - if show_diff_url: - desc_column += f"，[详细对比]({diff_page_url})" - - if "paddle_api" not in reference_item: - if mapping_type not in ["组合替代实现", "可删除", "功能缺失"]: - print( - f"Cannot find paddle_api for torch_api: {torch_api} in line {line_idx}" - ) - paddle_api_column = "" - else: - paddle_api = reference_item["paddle_api"] - paddle_api_url = reference_item["paddle_api_url"] - paddle_api_column = f"[`{paddle_api}`]({paddle_api_url})" - - content = [ - row_idx_s, - alisa_name_column, - paddle_api_column, - mapping_type_column, - desc_column, - ] + content.insert(0, row_idx_s) output = "| " + " | ".join(content) + " |\n" - return output - + return [output] elif not_implemented_match: torch_api = ( not_implemented_match["torch_api"].strip("`").replace(r"\_", "_") @@ -213,12 +210,12 @@ def apply_reference_to_row(line, metadata_dict, table_row_idx, line_idx): ] output = "| " + " | ".join(content) + " |\n" - return output + return [output] else: raise ValueError( f"found manual-maintaining row at line [{line_idx}]: {line}" ) - return line + return [line] def reference_mapping_item_processer(line, line_idx, state, output, context): @@ -232,11 +229,7 @@ def reference_mapping_item_processer(line, line_idx, state, output, context): # check column names in common process output.append(line) return True - elif state == 1: - # check seperator of table to ignore in common process - output.append(line) - return True - elif state == 2: + elif state == 1 or state == 2: # check seperator of table to process in common process output.append(line) return True @@ -246,17 +239,56 @@ def reference_mapping_item_processer(line, line_idx, state, output, context): return True elif state == 6: # check content of table to process in common process - referenced_row = apply_reference_to_row( - line, metadata_dict, context["table_row_idx"], line_idx + 1 + output_lines = apply_reference_to_row_ex( + line, metadata_dict, context, line_idx + 1 ) - output.append(referenced_row) + output += output_lines return True print(state) return False +def reference_table_scanner(line, _line_idx, state, output, context): + if not line.startswith("|"): + return True + + if state >= 0 and state <= 2: + return True + elif state == 5: + return True + elif state == 6: + # check content of table to process in common process + rtm = REFERENCE_TABLE_PATTERN.match(line) + if rtm: + condition = reference_table_match_to_condition(rtm) + context["table_conditions"].append(condition) + return True + + return False + + +def get_c2a_dict(conditions, meta_dict): + c2a_dict = {c: [] for c in conditions} + conditions.sort( + key=lambda c: (-len(c[0]), c[1]) + ) # 先按照字符串长度降序，随后按照最大深度升序 + for api in meta_dict: + for api_prefix, max_depth in conditions: + if not api.startswith(api_prefix): + continue + depth = len(api.split(".")) - 1 + if depth > max_depth: + continue + c2a_dict[(api_prefix, max_depth)].append(api) + break + else: + print(f"Warning: cannot find a suitable condition for api {api}") + + return c2a_dict + + if __name__ == "__main__": # convert from pytorch basedir cfp_basedir = os.path.dirname(__file__) @@ -287,7 +319,21 @@ def reference_mapping_item_processer(line, line_idx, state, output, context): "metadata_dict": meta_dict, "ret_code": 0, "output": [], + "table_conditions": [], } + + # 第一遍预读，用来分析有哪些表格和匹配条件 + ret_code = reference_mapping_item( + mapping_index_file, reference_table_scanner, reference_context + ) + assert ret_code == 0 + reference_context["output"] = [] + + # 现在 c2a_dict 包含每个条件对应的 api 列表 + c2a_dict = get_c2a_dict(reference_context["table_conditions"], meta_dict) + reference_context["c2a_dict"] = c2a_dict + + # 第二遍正式读，读并处理 ret_code = reference_mapping_item( mapping_index_file, reference_mapping_item_processer, reference_context ) diff --git a/docs/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.md b/docs/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.md index 929dbf1d314..65b47bb117a 100644 --- a/docs/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.md +++ b/docs/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.md @@ -36,390 +36,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.BoolTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.BoolTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.ByteTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ByteTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.DoubleTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.DoubleTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.FloatTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.FloatTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.Generator`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.Generator.md) | -| REFERENCE-MAPPING-ITEM(`torch.HalfTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.HalfTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.IntTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.IntTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.LongTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.LongTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.ShortTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ShortTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.Size`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.Size.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.Tensor__upper.md) | -| REFERENCE-MAPPING-ITEM(`torch.__version__`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.__version__.md) | -| REFERENCE-MAPPING-ITEM(`torch.abs`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.abs.md) | -| REFERENCE-MAPPING-ITEM(`torch.abs_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.abs_.md) | -| REFERENCE-MAPPING-ITEM(`torch.absolute`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.absolute.md) | -| REFERENCE-MAPPING-ITEM(`torch.acos`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.acos.md) | -| REFERENCE-MAPPING-ITEM(`torch.acosh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.acosh.md) | -| REFERENCE-MAPPING-ITEM(`torch.add`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.add.md) | -| REFERENCE-MAPPING-ITEM(`torch.addbmm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.addbmm.md) | -| REFERENCE-MAPPING-ITEM(`torch.addcdiv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.addcdiv.md) | -| REFERENCE-MAPPING-ITEM(`torch.addcmul`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.addcmul.md) | -| REFERENCE-MAPPING-ITEM(`torch.addmm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.addmm.md) | -| REFERENCE-MAPPING-ITEM(`torch.addmv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.addmv.md) | -| REFERENCE-MAPPING-ITEM(`torch.addr`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.addr.md) | -| REFERENCE-MAPPING-ITEM(`torch.adjoint`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.adjoint.md) | -| REFERENCE-MAPPING-ITEM(`torch.all`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.all.md) | -| REFERENCE-MAPPING-ITEM(`torch.allclose`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.allclose.md) | -| REFERENCE-MAPPING-ITEM(`torch.alpha_dropout`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.alpha_dropout.md) | -| REFERENCE-MAPPING-ITEM(`torch.feature_alpha_dropout`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.feature_alpha_dropout.md) | -| REFERENCE-MAPPING-ITEM(`torch.amax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.amax.md) | -| REFERENCE-MAPPING-ITEM(`torch.amin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.amin.md) | -| REFERENCE-MAPPING-ITEM(`torch.aminmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.aminmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.angle`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.angle.md) | -| REFERENCE-MAPPING-ITEM(`torch.any`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.any.md) | -| REFERENCE-MAPPING-ITEM(`torch.arange`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.arange.md) | -| REFERENCE-MAPPING-ITEM(`torch.arccos`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.arccos.md) | -| REFERENCE-MAPPING-ITEM(`torch.arccosh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.arccosh.md) | -| REFERENCE-MAPPING-ITEM(`torch.arcsin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.arcsin.md) | -| REFERENCE-MAPPING-ITEM(`torch.arcsinh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.arcsinh.md) | -| REFERENCE-MAPPING-ITEM(`torch.arctan`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.arctan.md) | -| REFERENCE-MAPPING-ITEM(`torch.arctan2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.arctan2.md) | -| REFERENCE-MAPPING-ITEM(`torch.arctanh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.arctanh.md) | -| REFERENCE-MAPPING-ITEM(`torch.argmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.argmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.argmin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.argmin.md) | -| REFERENCE-MAPPING-ITEM(`torch.argsort`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.argsort.md) | -| REFERENCE-MAPPING-ITEM(`torch.argwhere`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.argwhere.md) | -| REFERENCE-MAPPING-ITEM(`torch.as_strided`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.as_strided.md) | -| REFERENCE-MAPPING-ITEM(`torch.as_tensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.as_tensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.asarray`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.asarray.md) | -| REFERENCE-MAPPING-ITEM(`torch.asin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.asin.md) | -| REFERENCE-MAPPING-ITEM(`torch.asinh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.asinh.md) | -| REFERENCE-MAPPING-ITEM(`torch.atan`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.atan.md) | -| REFERENCE-MAPPING-ITEM(`torch.atan2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.atan2.md) | -| REFERENCE-MAPPING-ITEM(`torch.atanh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.atanh.md) | -| REFERENCE-MAPPING-ITEM(`torch.atleast_1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.atleast_1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.atleast_2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.atleast_2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.atleast_3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.atleast_3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.autocast`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autocast.md) | -| REFERENCE-MAPPING-ITEM(`torch.baddbmm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.baddbmm.md) | -| REFERENCE-MAPPING-ITEM(`torch.bernoulli`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.bernoulli.md) | -| REFERENCE-MAPPING-ITEM(`torch.bincount`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.bincount.md) | -| REFERENCE-MAPPING-ITEM(`torch.bitwise_and`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.bitwise_and.md) | -| REFERENCE-MAPPING-ITEM(`torch.bitwise_not`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.bitwise_not.md) | -| REFERENCE-MAPPING-ITEM(`torch.bitwise_or`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.bitwise_or.md) | -| REFERENCE-MAPPING-ITEM(`torch.bitwise_xor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.bitwise_xor.md) | -| REFERENCE-MAPPING-ITEM(`torch.bmm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.bmm.md) | -| REFERENCE-MAPPING-ITEM(`torch.broadcast_shapes`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.broadcast_shapes.md) | -| REFERENCE-MAPPING-ITEM(`torch.broadcast_tensors`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.broadcast_tensors.md) | -| REFERENCE-MAPPING-ITEM(`torch.broadcast_to`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.broadcast_to.md) | -| REFERENCE-MAPPING-ITEM(`torch.bucketize`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.bucketize.md) | -| REFERENCE-MAPPING-ITEM(`torch.cat`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cat.md) | -| REFERENCE-MAPPING-ITEM(`torch.cdist`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cdist.md) | -| REFERENCE-MAPPING-ITEM(`torch.ceil`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.ceil.md) | -| REFERENCE-MAPPING-ITEM(`torch.celu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.celu.md) | -| REFERENCE-MAPPING-ITEM(`torch.chain_matmul`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.chain_matmul.md) | -| REFERENCE-MAPPING-ITEM(`torch.cholesky`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cholesky.md) | -| REFERENCE-MAPPING-ITEM(`torch.cholesky_inverse`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cholesky_inverse.md) | -| REFERENCE-MAPPING-ITEM(`torch.cholesky_solve`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cholesky_solve.md) | -| REFERENCE-MAPPING-ITEM(`torch.chunk`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.chunk.md) | -| REFERENCE-MAPPING-ITEM(`torch.clamp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.clamp.md) | -| REFERENCE-MAPPING-ITEM(`torch.clamp_max`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.clamp_max.md) | -| REFERENCE-MAPPING-ITEM(`torch.clamp_min`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.clamp_min.md) | -| REFERENCE-MAPPING-ITEM(`torch.clip`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.clip.md) | -| REFERENCE-MAPPING-ITEM(`torch.clone`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.clone.md) | -| REFERENCE-MAPPING-ITEM(`torch.column_stack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.column_stack.md) | -| REFERENCE-MAPPING-ITEM(`torch.combinations`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.combinations.md) | -| REFERENCE-MAPPING-ITEM(`torch.complex`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.complex.md) | -| REFERENCE-MAPPING-ITEM(`torch.concat`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.concat.md) | -| REFERENCE-MAPPING-ITEM(`torch.conj`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.conj.md) | -| REFERENCE-MAPPING-ITEM(`torch.conj_physical`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.conj_physical.md) | -| REFERENCE-MAPPING-ITEM(`torch.copysign`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.copysign.md) | -| REFERENCE-MAPPING-ITEM(`torch.corrcoef`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.corrcoef.md) | -| REFERENCE-MAPPING-ITEM(`torch.cos`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cos.md) | -| REFERENCE-MAPPING-ITEM(`torch.cosh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cosh.md) | -| REFERENCE-MAPPING-ITEM(`torch.cosine_similarity`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cosine_similarity.md) | -| REFERENCE-MAPPING-ITEM(`torch.count_nonzero`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.count_nonzero.md) | -| REFERENCE-MAPPING-ITEM(`torch.cov`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cov.md) | -| REFERENCE-MAPPING-ITEM(`torch.cross`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cross.md) | -| REFERENCE-MAPPING-ITEM(`torch.cummax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cummax.md) | -| REFERENCE-MAPPING-ITEM(`torch.cummin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cummin.md) | -| REFERENCE-MAPPING-ITEM(`torch.cumprod`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cumprod.md) | -| REFERENCE-MAPPING-ITEM(`torch.cumsum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cumsum.md) | -| REFERENCE-MAPPING-ITEM(`torch.cumulative_trapezoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cumulative_trapezoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.deg2rad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.deg2rad.md) | -| REFERENCE-MAPPING-ITEM(`torch.det`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.det.md) | -| REFERENCE-MAPPING-ITEM(`torch.device`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.device.md) | -| REFERENCE-MAPPING-ITEM(`torch.diag`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.diag.md) | -| REFERENCE-MAPPING-ITEM(`torch.diag_embed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.diag_embed.md) | -| REFERENCE-MAPPING-ITEM(`torch.diagflat`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.diagflat.md) | -| REFERENCE-MAPPING-ITEM(`torch.diagonal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.diagonal.md) | -| REFERENCE-MAPPING-ITEM(`torch.diagonal_scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.diagonal_scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.diff`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.diff.md) | -| REFERENCE-MAPPING-ITEM(`torch.digamma`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.digamma.md) | -| REFERENCE-MAPPING-ITEM(`torch.dist`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.dist.md) | -| REFERENCE-MAPPING-ITEM(`torch.div`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.div.md) | -| REFERENCE-MAPPING-ITEM(`torch.divide`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.divide.md) | -| REFERENCE-MAPPING-ITEM(`torch.dot`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.dot.md) | -| REFERENCE-MAPPING-ITEM(`torch.dropout`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.dropout.md) | -| REFERENCE-MAPPING-ITEM(`torch.dsplit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.dsplit.md) | -| REFERENCE-MAPPING-ITEM(`torch.dstack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.dstack.md) | -| REFERENCE-MAPPING-ITEM(`torch.einsum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.einsum.md) | -| REFERENCE-MAPPING-ITEM(`torch.empty`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.empty.md) | -| REFERENCE-MAPPING-ITEM(`torch.empty_like`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.empty_like.md) | -| REFERENCE-MAPPING-ITEM(`torch.enable_grad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.enable_grad.md) | -| REFERENCE-MAPPING-ITEM(`torch.eq`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.eq.md) | -| REFERENCE-MAPPING-ITEM(`torch.equal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.equal.md) | -| REFERENCE-MAPPING-ITEM(`torch.erf`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.erf.md) | -| REFERENCE-MAPPING-ITEM(`torch.erfc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.erfc.md) | -| REFERENCE-MAPPING-ITEM(`torch.erfinv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.erfinv.md) | -| REFERENCE-MAPPING-ITEM(`torch.exp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.exp.md) | -| REFERENCE-MAPPING-ITEM(`torch.exp2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.exp2.md) | -| REFERENCE-MAPPING-ITEM(`torch.expm1`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.expm1.md) | -| REFERENCE-MAPPING-ITEM(`torch.eye`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.eye.md) | -| REFERENCE-MAPPING-ITEM(`torch.finfo`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.finfo.md) | -| REFERENCE-MAPPING-ITEM(`torch.fix`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.fix.md) | -| REFERENCE-MAPPING-ITEM(`torch.flatten`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.flatten.md) | -| REFERENCE-MAPPING-ITEM(`torch.flip`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.flip.md) | -| REFERENCE-MAPPING-ITEM(`torch.fliplr`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.fliplr.md) | -| REFERENCE-MAPPING-ITEM(`torch.flipud`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.flipud.md) | -| REFERENCE-MAPPING-ITEM(`torch.floor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.floor.md) | -| REFERENCE-MAPPING-ITEM(`torch.floor_divide`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.floor_divide.md) | -| REFERENCE-MAPPING-ITEM(`torch.fmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.fmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.fmin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.fmin.md) | -| REFERENCE-MAPPING-ITEM(`torch.fmod`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.fmod.md) | -| REFERENCE-MAPPING-ITEM(`torch.frac`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.frac.md) | -| REFERENCE-MAPPING-ITEM(`torch.frexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.frexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.from_dlpack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.from_dlpack.md) | -| REFERENCE-MAPPING-ITEM(`torch.from_numpy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.from_numpy.md) | -| REFERENCE-MAPPING-ITEM(`torch.full`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.full.md) | -| REFERENCE-MAPPING-ITEM(`torch.full_like`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.full_like.md) | -| REFERENCE-MAPPING-ITEM(`torch.gather`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.gather.md) | -| REFERENCE-MAPPING-ITEM(`torch.gcd`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.gcd.md) | -| REFERENCE-MAPPING-ITEM(`torch.ge`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.ge.md) | -| REFERENCE-MAPPING-ITEM(`torch.ger`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.ger.md) | -| REFERENCE-MAPPING-ITEM(`torch.get_default_dtype`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.get_default_dtype.md) | -| REFERENCE-MAPPING-ITEM(`torch.get_rng_state`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.get_rng_state.md) | -| REFERENCE-MAPPING-ITEM(`torch.greater`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.greater.md) | -| REFERENCE-MAPPING-ITEM(`torch.gt`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.gt.md) | -| REFERENCE-MAPPING-ITEM(`torch.heaviside`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.heaviside.md) | -| REFERENCE-MAPPING-ITEM(`torch.histc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.histc.md) | -| REFERENCE-MAPPING-ITEM(`torch.histogram`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.histogram.md) | -| REFERENCE-MAPPING-ITEM(`torch.histogramdd`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.histogramdd.md) | -| REFERENCE-MAPPING-ITEM(`torch.hsplit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.hsplit.md) | -| REFERENCE-MAPPING-ITEM(`torch.hstack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.hstack.md) | -| REFERENCE-MAPPING-ITEM(`torch.hypot`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.hypot.md) | -| REFERENCE-MAPPING-ITEM(`torch.i0`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.i0.md) | -| REFERENCE-MAPPING-ITEM(`torch.iinfo`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.iinfo.md) | -| REFERENCE-MAPPING-ITEM(`torch.imag`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.imag.md) | -| REFERENCE-MAPPING-ITEM(`torch.index_add`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.index_add.md) | -| REFERENCE-MAPPING-ITEM(`torch.index_copy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.index_copy.md) | -| REFERENCE-MAPPING-ITEM(`torch.index_fill`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.index_fill.md) | -| REFERENCE-MAPPING-ITEM(`torch.index_select`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.index_select.md) | -| REFERENCE-MAPPING-ITEM(`torch.inference_mode`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.inference_mode.md) | -| REFERENCE-MAPPING-ITEM(`torch.initial_seed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.initial_seed.md) | -| REFERENCE-MAPPING-ITEM(`torch.inner`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.inner.md) | -| REFERENCE-MAPPING-ITEM(`torch.inverse`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.inverse.md) | -| REFERENCE-MAPPING-ITEM(`torch.is_complex`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.is_complex.md) | -| REFERENCE-MAPPING-ITEM(`torch.is_floating_point`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.is_floating_point.md) | -| REFERENCE-MAPPING-ITEM(`torch.is_grad_enabled`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.is_grad_enabled.md) | -| REFERENCE-MAPPING-ITEM(`torch.is_nonzero`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.is_nonzero.md) | -| REFERENCE-MAPPING-ITEM(`torch.is_tensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.is_tensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.isclose`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.isclose.md) | -| REFERENCE-MAPPING-ITEM(`torch.isfinite`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.isfinite.md) | -| REFERENCE-MAPPING-ITEM(`torch.isinf`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.isinf.md) | -| REFERENCE-MAPPING-ITEM(`torch.isnan`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.isnan.md) | -| REFERENCE-MAPPING-ITEM(`torch.istft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.istft.md) | -| REFERENCE-MAPPING-ITEM(`torch.kron`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.kron.md) | -| REFERENCE-MAPPING-ITEM(`torch.kthvalue`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.kthvalue.md) | -| REFERENCE-MAPPING-ITEM(`torch.lcm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.lcm.md) | -| REFERENCE-MAPPING-ITEM(`torch.ldexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.ldexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.le`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.le.md) | -| REFERENCE-MAPPING-ITEM(`torch.lerp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.lerp.md) | -| REFERENCE-MAPPING-ITEM(`torch.less`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.less.md) | -| REFERENCE-MAPPING-ITEM(`torch.less_equal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.less_equal.md) | -| REFERENCE-MAPPING-ITEM(`torch.lgamma`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.lgamma.md) | -| REFERENCE-MAPPING-ITEM(`torch.linspace`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.linspace.md) | -| REFERENCE-MAPPING-ITEM(`torch.load`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.load.md) | -| REFERENCE-MAPPING-ITEM(`torch.log`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.log.md) | -| REFERENCE-MAPPING-ITEM(`torch.log10`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.log10.md) | -| REFERENCE-MAPPING-ITEM(`torch.log1p`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.log1p.md) | -| REFERENCE-MAPPING-ITEM(`torch.log2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.log2.md) | -| REFERENCE-MAPPING-ITEM(`torch.logaddexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logaddexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.logaddexp2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logaddexp2.md) | -| REFERENCE-MAPPING-ITEM(`torch.logcumsumexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logcumsumexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.logdet`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logdet.md) | -| REFERENCE-MAPPING-ITEM(`torch.logical_and`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logical_and.md) | -| REFERENCE-MAPPING-ITEM(`torch.logical_not`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logical_not.md) | -| REFERENCE-MAPPING-ITEM(`torch.logical_or`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logical_or.md) | -| REFERENCE-MAPPING-ITEM(`torch.logical_xor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logical_xor.md) | -| REFERENCE-MAPPING-ITEM(`torch.logit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logit.md) | -| REFERENCE-MAPPING-ITEM(`torch.logspace`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.logspace.md) | -| REFERENCE-MAPPING-ITEM(`torch.lt`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.lt.md) | -| REFERENCE-MAPPING-ITEM(`torch.lu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.lu.md) | -| REFERENCE-MAPPING-ITEM(`torch.lu_unpack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.lu_unpack.md) | -| REFERENCE-MAPPING-ITEM(`torch.manual_seed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.manual_seed.md) | -| REFERENCE-MAPPING-ITEM(`torch.masked_fill`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.masked_fill.md) | -| REFERENCE-MAPPING-ITEM(`torch.masked_select`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.masked_select.md) | -| REFERENCE-MAPPING-ITEM(`torch.matmul`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.matmul.md) | -| REFERENCE-MAPPING-ITEM(`torch.matrix_power`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.matrix_power.md) | -| REFERENCE-MAPPING-ITEM(`torch.max`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.max.md) | -| REFERENCE-MAPPING-ITEM(`torch.max_pool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.max_pool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.max_pool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.max_pool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.max_pool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.max_pool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.maximum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.maximum.md) | -| REFERENCE-MAPPING-ITEM(`torch.mean`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.mean.md) | -| REFERENCE-MAPPING-ITEM(`torch.median`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.median.md) | -| REFERENCE-MAPPING-ITEM(`torch.meshgrid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.meshgrid.md) | -| REFERENCE-MAPPING-ITEM(`torch.min`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.min.md) | -| REFERENCE-MAPPING-ITEM(`torch.minimum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.minimum.md) | -| REFERENCE-MAPPING-ITEM(`torch.mm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.mm.md) | -| REFERENCE-MAPPING-ITEM(`torch.mode`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.mode.md) | -| REFERENCE-MAPPING-ITEM(`torch.moveaxis`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.moveaxis.md) | -| REFERENCE-MAPPING-ITEM(`torch.movedim`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.movedim.md) | -| REFERENCE-MAPPING-ITEM(`torch.msort`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.msort.md) | -| REFERENCE-MAPPING-ITEM(`torch.mul`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.mul.md) | -| REFERENCE-MAPPING-ITEM(`torch.multinomial`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.multinomial.md) | -| REFERENCE-MAPPING-ITEM(`torch.multiply`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.multiply.md) | -| REFERENCE-MAPPING-ITEM(`torch.mv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.mv.md) | -| REFERENCE-MAPPING-ITEM(`torch.nan_to_num`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nan_to_num.md) | -| REFERENCE-MAPPING-ITEM(`torch.nanmean`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nanmean.md) | -| REFERENCE-MAPPING-ITEM(`torch.nanmedian`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nanmedian.md) | -| REFERENCE-MAPPING-ITEM(`torch.nanquantile`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nanquantile.md) | -| REFERENCE-MAPPING-ITEM(`torch.nansum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.nansum.md) | -| REFERENCE-MAPPING-ITEM(`torch.narrow`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.narrow.md) | -| REFERENCE-MAPPING-ITEM(`torch.narrow_copy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.narrow_copy.md) | -| REFERENCE-MAPPING-ITEM(`torch.ne`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.ne.md) | -| REFERENCE-MAPPING-ITEM(`torch.neg`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.neg.md) | -| REFERENCE-MAPPING-ITEM(`torch.negative`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.negative.md) | -| REFERENCE-MAPPING-ITEM(`torch.nextafter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nextafter.md) | -| REFERENCE-MAPPING-ITEM(`torch.no_grad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.no_grad.md) | -| REFERENCE-MAPPING-ITEM(`torch.nonzero`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nonzero.md) | -| REFERENCE-MAPPING-ITEM(`torch.norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.normal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.normal.md) | -| REFERENCE-MAPPING-ITEM(`torch.not_equal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.not_equal.md) | -| REFERENCE-MAPPING-ITEM(`torch.numel`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.numel.md) | -| REFERENCE-MAPPING-ITEM(`torch.ones`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.ones.md) | -| REFERENCE-MAPPING-ITEM(`torch.ones_like`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.ones_like.md) | -| REFERENCE-MAPPING-ITEM(`torch.outer`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.outer.md) | -| REFERENCE-MAPPING-ITEM(`torch.pca_lowrank`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.pca_lowrank.md) | -| REFERENCE-MAPPING-ITEM(`torch.permute`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.permute.md) | -| REFERENCE-MAPPING-ITEM(`torch.pinverse`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.pinverse.md) | -| REFERENCE-MAPPING-ITEM(`torch.poisson`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.poisson.md) | -| REFERENCE-MAPPING-ITEM(`torch.polar`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.polar.md) | -| REFERENCE-MAPPING-ITEM(`torch.polygamma`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.polygamma.md) | -| REFERENCE-MAPPING-ITEM(`torch.pow`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.pow.md) | -| REFERENCE-MAPPING-ITEM(`torch.prod`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.prod.md) | -| REFERENCE-MAPPING-ITEM(`torch.qr`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.qr.md) | -| REFERENCE-MAPPING-ITEM(`torch.quantile`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.quantile.md) | -| REFERENCE-MAPPING-ITEM(`torch.rad2deg`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rad2deg.md) | -| REFERENCE-MAPPING-ITEM(`torch.rand`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rand.md) | -| REFERENCE-MAPPING-ITEM(`torch.rand_like`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rand_like.md) | -| REFERENCE-MAPPING-ITEM(`torch.randint`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.randint.md) | -| REFERENCE-MAPPING-ITEM(`torch.randint_like`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.randint_like.md) | -| REFERENCE-MAPPING-ITEM(`torch.randn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.randn.md) | -| REFERENCE-MAPPING-ITEM(`torch.randn_like`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.randn_like.md) | -| REFERENCE-MAPPING-ITEM(`torch.randperm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.randperm.md) | -| REFERENCE-MAPPING-ITEM(`torch.range`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.range.md) | -| REFERENCE-MAPPING-ITEM(`torch.ravel`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.ravel.md) | -| REFERENCE-MAPPING-ITEM(`torch.real`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.real.md) | -| REFERENCE-MAPPING-ITEM(`torch.reciprocal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.reciprocal.md) | -| REFERENCE-MAPPING-ITEM(`torch.relu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.relu.md) | -| REFERENCE-MAPPING-ITEM(`torch.remainder`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.remainder.md) | -| REFERENCE-MAPPING-ITEM(`torch.renorm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.renorm.md) | -| REFERENCE-MAPPING-ITEM(`torch.repeat_interleave`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.repeat_interleave.md) | -| REFERENCE-MAPPING-ITEM(`torch.reshape`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.reshape.md) | -| REFERENCE-MAPPING-ITEM(`torch.roll`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.roll.md) | -| REFERENCE-MAPPING-ITEM(`torch.rot90`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rot90.md) | -| REFERENCE-MAPPING-ITEM(`torch.round`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.round.md) | -| REFERENCE-MAPPING-ITEM(`torch.row_stack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.row_stack.md) | -| REFERENCE-MAPPING-ITEM(`torch.rrelu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rrelu.md) | -| REFERENCE-MAPPING-ITEM(`torch.rsqrt`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rsqrt.md) | -| REFERENCE-MAPPING-ITEM(`torch.save`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.save.md) | -| REFERENCE-MAPPING-ITEM(`torch.scalar_tensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scalar_tensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.scatter_add`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter_add.md) | -| REFERENCE-MAPPING-ITEM(`torch.searchsorted`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.searchsorted.md) | -| REFERENCE-MAPPING-ITEM(`torch.seed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.seed.md) | -| REFERENCE-MAPPING-ITEM(`torch.select`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.select.md) | -| REFERENCE-MAPPING-ITEM(`torch.select_scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.select_scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.selu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.selu.md) | -| REFERENCE-MAPPING-ITEM(`torch.set_default_dtype`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.set_default_dtype.md) | -| REFERENCE-MAPPING-ITEM(`torch.set_default_tensor_type`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.set_default_tensor_type.md) | -| REFERENCE-MAPPING-ITEM(`torch.set_grad_enabled`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.set_grad_enabled.md) | -| REFERENCE-MAPPING-ITEM(`torch.set_printoptions`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.set_printoptions.md) | -| REFERENCE-MAPPING-ITEM(`torch.set_rng_state`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.set_rng_state.md) | -| REFERENCE-MAPPING-ITEM(`torch.sgn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sgn.md) | -| REFERENCE-MAPPING-ITEM(`torch.sigmoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sigmoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.sign`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sign.md) | -| REFERENCE-MAPPING-ITEM(`torch.signbit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.signbit.md) | -| REFERENCE-MAPPING-ITEM(`torch.sin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sin.md) | -| REFERENCE-MAPPING-ITEM(`torch.sinc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sinc.md) | -| REFERENCE-MAPPING-ITEM(`torch.sinh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sinh.md) | -| REFERENCE-MAPPING-ITEM(`torch.slice_scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.slice_scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.slogdet`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.slogdet.md) | -| REFERENCE-MAPPING-ITEM(`torch.softmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.softmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.sort`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sort.md) | -| REFERENCE-MAPPING-ITEM(`torch.sparse_coo_tensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sparse_coo_tensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.sparse_csr_tensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sparse_csr_tensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.split`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.split.md) | -| REFERENCE-MAPPING-ITEM(`torch.sqrt`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sqrt.md) | -| REFERENCE-MAPPING-ITEM(`torch.square`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.square.md) | -| REFERENCE-MAPPING-ITEM(`torch.squeeze`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.squeeze.md) | -| REFERENCE-MAPPING-ITEM(`torch.stack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.stack.md) | -| REFERENCE-MAPPING-ITEM(`torch.std`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.std.md) | -| REFERENCE-MAPPING-ITEM(`torch.std_mean`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.std_mean.md) | -| REFERENCE-MAPPING-ITEM(`torch.stft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.stft.md) | -| REFERENCE-MAPPING-ITEM(`torch.sub`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sub.md) | -| REFERENCE-MAPPING-ITEM(`torch.subtract`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.subtract.md) | -| REFERENCE-MAPPING-ITEM(`torch.sum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.sum.md) | -| REFERENCE-MAPPING-ITEM(`torch.svd`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.svd.md) | -| REFERENCE-MAPPING-ITEM(`torch.svd_lowrank`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.svd_lowrank.md) | -| REFERENCE-MAPPING-ITEM(`torch.swapaxes`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.swapaxes.md) | -| REFERENCE-MAPPING-ITEM(`torch.swapdims`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.swapdims.md) | -| REFERENCE-MAPPING-ITEM(`torch.symeig`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.symeig.md) | -| REFERENCE-MAPPING-ITEM(`torch.t`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.t.md) | -| REFERENCE-MAPPING-ITEM(`torch.take`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.take.md) | -| REFERENCE-MAPPING-ITEM(`torch.take_along_dim`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.take_along_dim.md) | -| REFERENCE-MAPPING-ITEM(`torch.tan`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.tan.md) | -| REFERENCE-MAPPING-ITEM(`torch.tensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.tensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.tensor_split`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.tensor_split.md) | -| REFERENCE-MAPPING-ITEM(`torch.tensordot`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.tensordot.md) | -| REFERENCE-MAPPING-ITEM(`torch.tile`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.tile.md) | -| REFERENCE-MAPPING-ITEM(`torch.topk`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.topk.md) | -| REFERENCE-MAPPING-ITEM(`torch.trace`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.trace.md) | -| REFERENCE-MAPPING-ITEM(`torch.transpose`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.transpose.md) | -| REFERENCE-MAPPING-ITEM(`torch.trapezoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.trapezoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.triangular_solve`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.triangular_solve.md) | -| REFERENCE-MAPPING-ITEM(`torch.tril`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.tril.md) | -| REFERENCE-MAPPING-ITEM(`torch.tril_indices`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.tril_indices.md) | -| REFERENCE-MAPPING-ITEM(`torch.triu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.triu.md) | -| REFERENCE-MAPPING-ITEM(`torch.triu_indices`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.triu_indices.md) | -| REFERENCE-MAPPING-ITEM(`torch.true_divide`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.true_divide.md) | -| REFERENCE-MAPPING-ITEM(`torch.trunc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.trunc.md) | -| REFERENCE-MAPPING-ITEM(`torch.unbind`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.unbind.md) | -| REFERENCE-MAPPING-ITEM(`torch.unflatten`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.unflatten.md) | -| REFERENCE-MAPPING-ITEM(`torch.unique`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.unique.md) | -| REFERENCE-MAPPING-ITEM(`torch.unique_consecutive`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.unique_consecutive.md) | -| REFERENCE-MAPPING-ITEM(`torch.unsqueeze`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.unsqueeze.md) | -| REFERENCE-MAPPING-ITEM(`torch.vander`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.vander.md) | -| REFERENCE-MAPPING-ITEM(`torch.var`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.var.md) | -| REFERENCE-MAPPING-ITEM(`torch.var_mean`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.var_mean.md) | -| REFERENCE-MAPPING-ITEM(`torch.vdot`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.vdot.md) | -| REFERENCE-MAPPING-ITEM(`torch.view_as_complex`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.view_as_complex.md) | -| REFERENCE-MAPPING-ITEM(`torch.view_as_real`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.view_as_real.md) | -| REFERENCE-MAPPING-ITEM(`torch.vsplit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.vsplit.md) | -| REFERENCE-MAPPING-ITEM(`torch.vstack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.vstack.md) | -| REFERENCE-MAPPING-ITEM(`torch.where`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.where.md) | -| REFERENCE-MAPPING-ITEM(`torch.xlogy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.xlogy.md) | -| REFERENCE-MAPPING-ITEM(`torch.zeros`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.zeros.md) | -| REFERENCE-MAPPING-ITEM(`torch.zeros_like`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.zeros_like.md) | -| NOT-IMPLEMENTED-ITEM(`torch.bitwise_left_shift`, https://pytorch.org/docs/stable/generated/torch.bitwise_left_shift.html#torch.bitwise_left_shift) | -| NOT-IMPLEMENTED-ITEM(`torch.bitwise_right_shift`, https://pytorch.org/docs/stable/generated/torch.bitwise_right_shift.html#torch.bitwise_right_shift) | -| NOT-IMPLEMENTED-ITEM(`torch.can_cast`, https://pytorch.org/docs/stable/generated/torch.can_cast.html) | -| NOT-IMPLEMENTED-ITEM(`torch.geqrf`, https://pytorch.org/docs/stable/generated/torch.geqrf.html?highlight=geqrf#torch.geqrf) | -| NOT-IMPLEMENTED-ITEM(`torch.get_deterministic_debug_mode`, https://pytorch.org/docs/stable/generated/torch.get_deterministic_debug_mode.html#torch.get_deterministic_debug_mode) | -| NOT-IMPLEMENTED-ITEM(`torch.gradient`, https://pytorch.org/docs/stable/generated/torch.gradient.html#torch.gradient) | -| NOT-IMPLEMENTED-ITEM(`torch.is_conj`, https://pytorch.org/docs/stable/generated/torch.is_conj.html#torch.is_conj) | -| NOT-IMPLEMENTED-ITEM(`torch.isin`, https://pytorch.org/docs/stable/generated/torch.isin.html#torch.isin) | -| NOT-IMPLEMENTED-ITEM(`torch.layout`, https://pytorch.org/docs/stable/tensor_attributes.html#torch.layout) | -| NOT-IMPLEMENTED-ITEM(`torch.positive`, https://pytorch.org/docs/stable/generated/torch.positive.html#torch.positive) | -| NOT-IMPLEMENTED-ITEM(`torch.resolve_conj`, https://pytorch.org/docs/stable/generated/torch.resolve_conj.html#torch.resolve_conj) | -| NOT-IMPLEMENTED-ITEM(`torch.resolve_neg`, https://pytorch.org/docs/stable/generated/torch.resolve_neg.html#torch.resolve_neg) | -| NOT-IMPLEMENTED-ITEM(`torch.scatter_reduce`, https://pytorch.org/docs/stable/generated/torch.scatter_reduce.html#torch.scatter_reduce) | -| NOT-IMPLEMENTED-ITEM(`torch.set_deterministic_debug_mode`, https://pytorch.org/docs/stable/generated/torch.set_deterministic_debug_mode.html#torch.set_deterministic_debug_mode) | +| REFERENCE-MAPPING-TABLE(`torch.`, max_depth=1) | ***持续更新...*** @@ -429,151 +46,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.nn.AdaptiveAvgPool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.AdaptiveAvgPool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.AdaptiveAvgPool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.AdaptiveMaxPool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveMaxPool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.AdaptiveMaxPool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveMaxPool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.AdaptiveMaxPool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveMaxPool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.AlphaDropout`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AlphaDropout.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.FeatureAlphaDropout`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.FeatureAlphaDropout.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.AvgPool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AvgPool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.AvgPool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AvgPool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.AvgPool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AvgPool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.BCELoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.BCELoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.BCEWithLogitsLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.BCEWithLogitsLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.BatchNorm1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.BatchNorm1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.BatchNorm2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.BatchNorm2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.BatchNorm3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.BatchNorm3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Bilinear`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Bilinear.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.CELU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.CELU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.CTCLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.CTCLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ChannelShuffle`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ChannelShuffle.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ConstantPad1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ConstantPad2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ConstantPad3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Conv1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Conv1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Conv2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Conv2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Conv3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Conv3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ConvTranspose1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConvTranspose1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ConvTranspose2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConvTranspose2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ConvTranspose3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConvTranspose3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.CosineEmbeddingLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.CosineEmbeddingLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.CosineSimilarity`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.CosineSimilarity.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.CrossEntropyLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.CrossEntropyLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.DataParallel`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.DataParallel.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Dropout`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Dropout.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Dropout1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Dropout1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Dropout2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Dropout2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Dropout3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Dropout3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ELU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ELU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Embedding`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Embedding.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Flatten`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Flatten.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Fold`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Fold.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.FractionalMaxPool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.FractionalMaxPool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.FractionalMaxPool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.FractionalMaxPool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.GELU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.GELU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.GRU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.GRU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.GRUCell`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.GRUCell.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.GaussianNLLLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.GaussianNLLLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.GroupNorm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.GroupNorm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Hardshrink`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Hardshrink.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Hardsigmoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Hardsigmoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Hardswish`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Hardswish.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Hardtanh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Hardtanh.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.HingeEmbeddingLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.HingeEmbeddingLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.HuberLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.HuberLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Identity`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Identity.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.InstanceNorm1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.InstanceNorm1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.InstanceNorm2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.InstanceNorm2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.InstanceNorm3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.InstanceNorm3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.KLDivLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.KLDivLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.L1Loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.L1Loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.LSTM`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LSTM.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.LSTMCell`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LSTMCell.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.LayerNorm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LayerNorm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.LeakyReLU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LeakyReLU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Linear`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Linear.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.LocalResponseNorm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LocalResponseNorm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.LogSigmoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LogSigmoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.LogSoftmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LogSoftmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MSELoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MSELoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MarginRankingLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MarginRankingLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MaxPool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxPool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MaxPool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxPool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MaxPool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxPool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MaxUnpool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MaxUnpool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MaxUnpool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Mish`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Mish.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ModuleDict`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ModuleDict.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ModuleList`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ModuleList.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MultiLabelMarginLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MultiLabelMarginLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MultiLabelSoftMarginLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MultiLabelSoftMarginLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MultiMarginLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MultiMarginLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.MultiheadAttention`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MultiheadAttention.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.NLLLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.NLLLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.PReLU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PReLU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.PairwiseDistance`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PairwiseDistance.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Parameter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Parameter.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ParameterList`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ParameterList.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.PixelShuffle`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelShuffle.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.PixelUnshuffle`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelUnshuffle.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.PoissonNLLLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PoissonNLLLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.RNN`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.RNN.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.RNNBase`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.RNNBase.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.RNNCell`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.RNNCell.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.RReLU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.RReLU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ReLU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ReLU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ReLU6`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ReLU6.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ReflectionPad1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ReflectionPad1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ReflectionPad2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ReflectionPad2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ReflectionPad3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ReflectionPad3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ReplicationPad1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ReplicationPad1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ReplicationPad2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ReplicationPad2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ReplicationPad3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ReplicationPad3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.SELU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.SELU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Sequential`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Sequential.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.SiLU`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.SiLU.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Sigmoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Sigmoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.SmoothL1Loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.SmoothL1Loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.SoftMarginLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.SoftMarginLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Softmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Softmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Softmax2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Softmax2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Softplus`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Softplus.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Softshrink`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Softshrink.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Softsign`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Softsign.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.SyncBatchNorm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.SyncBatchNorm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.SyncBatchNorm.convert_sync_batchnorm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.SyncBatchNorm.convert_sync_batchnorm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Tanh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Tanh.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Tanhshrink`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Tanhshrink.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Threshold`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Threshold.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Transformer`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Transformer.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.TransformerDecoder`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.TransformerDecoder.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.TransformerDecoderLayer`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.TransformerDecoderLayer.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.TransformerEncoder`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.TransformerEncoder.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.TransformerEncoderLayer`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.TransformerEncoderLayer.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.TripletMarginLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.TripletMarginLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.TripletMarginWithDistanceLoss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.TripletMarginWithDistanceLoss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Unflatten`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Unflatten.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Unfold`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Unfold.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Upsample`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Upsample.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.UpsamplingBilinear2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingBilinear2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.UpsamplingNearest2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingNearest2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.ZeroPad2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ZeroPad2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.modules.batchnorm._BatchNorm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nn.modules.batchnorm._BatchNorm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.modules.utils._ntuple`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.modules.utils._ntuple.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.modules.utils._pair`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.modules.utils._pair.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.parallel.DistributedDataParallel`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.parallel.DistributedDataParallel.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.parameter.Parameter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.nn.parameter.Parameter.md) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.GLU`, https://pytorch.org/docs/stable/generated/torch.nn.GLU.html#torch.nn.GLU) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyBatchNorm1d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyBatchNorm1d.html#torch.nn.LazyBatchNorm1d) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyBatchNorm2d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyBatchNorm2d.html#torch.nn.LazyBatchNorm2d) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyBatchNorm3d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyBatchNorm3d.html#torch.nn.LazyBatchNorm3d) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyInstanceNorm1d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyInstanceNorm1d.html#torch.nn.LazyInstanceNorm1d) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyInstanceNorm2d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyInstanceNorm2d.html#torch.nn.LazyInstanceNorm2d) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyInstanceNorm3d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyInstanceNorm3d.html#torch.nn.LazyInstanceNorm3d) | +| REFERENCE-MAPPING-TABLE(`torch.nn.`) | ***持续更新...*** @@ -582,114 +55,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional._Reduction.get_enum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional._Reduction.get_enum.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.adaptive_avg_pool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.adaptive_avg_pool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.adaptive_avg_pool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.adaptive_avg_pool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.adaptive_avg_pool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.adaptive_avg_pool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.adaptive_max_pool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.adaptive_max_pool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.adaptive_max_pool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.adaptive_max_pool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.adaptive_max_pool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.adaptive_max_pool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.affine_grid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.affine_grid.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.alpha_dropout`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.alpha_dropout.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.avg_pool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.avg_pool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.avg_pool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.avg_pool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.avg_pool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.avg_pool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.batch_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.batch_norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.bilinear`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.bilinear.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.binary_cross_entropy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.binary_cross_entropy.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.binary_cross_entropy_with_logits`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.binary_cross_entropy_with_logits.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.celu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.celu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.conv1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.conv2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.conv3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.conv_transpose1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.conv_transpose2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.conv_transpose3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.cosine_embedding_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.cosine_embedding_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.cosine_similarity`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.cosine_similarity.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.cross_entropy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.cross_entropy.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.ctc_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.ctc_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.dropout`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.dropout.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.dropout1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.dropout1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.dropout2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.dropout2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.dropout3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.dropout3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.elu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.elu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.elu_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.elu_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.embedding`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.embedding.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.fold`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fold.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.fractional_max_pool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.fractional_max_pool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.gaussian_nll_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.gaussian_nll_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.gelu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.gelu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.glu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.glu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.grid_sample`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.grid_sample.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.group_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.group_norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.gumbel_softmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.gumbel_softmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.hardshrink`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardshrink.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.hardsigmoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardsigmoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.hardswish`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardswish.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.hardtanh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardtanh.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.hardtanh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardtanh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.hinge_embedding_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hinge_embedding_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.huber_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.huber_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.instance_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.instance_norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.interpolate`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.interpolate.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.kl_div`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.kl_div.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.l1_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.l1_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.layer_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.layer_norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.leaky_relu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.leaky_relu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.leaky_relu_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.leaky_relu_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.linear`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.linear.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.local_response_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.local_response_norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.log_softmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.log_softmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.logsigmoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.log_sigmoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.margin_ranking_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.margin_ranking_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.max_pool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.max_pool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.max_pool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.max_pool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.max_pool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.max_pool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.max_unpool1d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.max_unpool1d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.max_unpool2d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.max_unpool2d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.max_unpool3d`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.max_unpool3d.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.mish`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.mish.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.mse_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.mse_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.multi_margin_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.multi_margin_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.multilabel_soft_margin_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.multilabel_soft_margin_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.nll_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.nll_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.normalize`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.normalize.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.one_hot`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.one_hot.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.pad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pad.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.pairwise_distance`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pairwise_distance.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.pdist`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pdist.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.pixel_shuffle`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_shuffle.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.pixel_unshuffle`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_unshuffle.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.poisson_nll_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.poisson_nll_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.prelu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.prelu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.relu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.relu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.relu6`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.relu6.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.relu_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.relu_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.rrelu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.rrelu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.rrelu_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.rrelu_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.selu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.selu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.sigmoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.sigmoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.silu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.silu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.smooth_l1_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.smooth_l1_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.soft_margin_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.soft_margin_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.softmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.softmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.softmin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.softmin.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.softplus`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.softplus.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.softshrink`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.softshrink.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.softsign`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.softsign.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.tanh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.tanh.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.tanhshrink`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.tanhshrink.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.threshold`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.threshold.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.triplet_margin_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.triplet_margin_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.triplet_margin_with_distance_loss`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.triplet_margin_with_distance_loss.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.unfold`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.unfold.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.upsample`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.upsample_bilinear`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_bilinear.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.functional.upsample_nearest`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_nearest.md) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.functional.multilabel_margin_loss`, https://pytorch.org/docs/stable/generated/torch.nn.functional.multilabel_margin_loss.html#torch.nn.functional.multilabel_margin_loss) | - +| REFERENCE-MAPPING-TABLE(`torch.nn.functional`) | ***持续更新...*** @@ -699,453 +65,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.H`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.H.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.T`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.T__upper.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.abs`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.abs.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.abs_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.abs_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.absolute`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.absolute.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.acos_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.acos_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.acosh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.acosh.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.acosh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.acosh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.add`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.add.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.add_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.add_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.addbmm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.addbmm.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.addcdiv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.addcdiv.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.addcmul`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.addcmul.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.addmm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.addmm.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.addmm_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.addmm_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.addmv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.addmv.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.addr`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.addr.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.adjoint`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.adjoint.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.all`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.all.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.allclose`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.allclose.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.amax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.amax.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.amin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.amin.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.aminmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.aminmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.angle`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.angle.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.any`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.any.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.apply_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.apply_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arccos`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arccos.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arccos_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arccos_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arccosh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arccosh.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arccosh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arccosh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arcsin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arcsin.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arcsin_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arcsin_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arcsinh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arcsinh.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arcsinh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arcsinh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arctan`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arctan.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arctan2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arctan2.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arctan_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arctan_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arctanh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arctanh.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.arctanh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.arctanh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.argmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.argmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.argmin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.argmin.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.argsort`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.argsort.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.argwhere`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.argwhere.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.as_strided`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.as_strided.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.asin_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.asin_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.asinh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.asinh.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.asinh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.asinh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.atan_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.atan_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.atanh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.atanh.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.atanh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.atanh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.backward`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.backward.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.baddbmm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.baddbmm.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bernoulli`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bernoulli.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bernoulli_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bernoulli_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bfloat16`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bfloat16.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bincount`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bincount.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bitwise_and`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bitwise_and.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bitwise_and_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bitwise_and_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bitwise_not`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bitwise_not.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bitwise_not_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bitwise_not_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bitwise_or`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bitwise_or.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bitwise_or_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bitwise_or_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bitwise_xor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bitwise_xor.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bitwise_xor_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bitwise_xor_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bmm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bmm.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.bool`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.bool.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.broadcast_to`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.broadcast_to.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.byte`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.byte.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cdouble`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cdouble.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ceil`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ceil.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ceil_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ceil_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cfloat`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cfloat.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.char`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.char.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cholesky`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cholesky.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cholesky_inverse`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cholesky_inverse.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cholesky_solve`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cholesky_solve.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.chunk`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.chunk.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.clamp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.clamp.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.clamp_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.clamp_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.clip`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.clip.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.clip_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.clip_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.clone`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.clone.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.coalesce`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.coalesce.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.conj`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.conj.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.conj_physical`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.conj_physical.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.contiguous`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.contiguous.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.copy_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.copy_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.corrcoef`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.corrcoef.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cos`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cos.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cos_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cos_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cosh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cosh.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cosh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cosh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.count_nonzero`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.count_nonzero.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cov`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cov.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cpu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cpu.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cross`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cross.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cuda`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cuda.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cummax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummax.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cummin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummin.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cumprod`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cumprod.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cumprod_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cumprod_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cumsum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cumsum.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.cumsum_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cumsum_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.deg2rad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.deg2rad.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.det`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.det.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.detach`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.detach.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.detach_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.detach_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.device`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.device.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.diag`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.diag.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.diag_embed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.diag_embed.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.diagflat`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.diagflat.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.diagonal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.diagonal.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.diagonal_scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.diagonal_scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.diff`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.diff.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.digamma`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.digamma.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.digamma_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.digamma_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.dim`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.dim.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.dist`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.dist.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.div`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.div.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.div_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.div_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.divide`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.divide.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.divide_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.divide_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.dot`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.dot.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.double`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.double.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.dsplit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.dsplit.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.dtype`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tenosr.dtype.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.element_size`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.element_size.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.eq`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.eq.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.eq_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.eq_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.equal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.equal.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.erf`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.erf.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.erf_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.erf_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.erfc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.erfc.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.erfinv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.erfinv.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.erfinv\_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.erfinv_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.erfinv_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.erfinv_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.exp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.exp.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.exp\_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.exp_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.exp_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.exp_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.expand`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.expand.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.expand_as`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.expand_as.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.expm1`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.expm1.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.expm1_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.expm1_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.exponential_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.exponential_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.fill\_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fill_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.fill_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fill_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.fill_diagonal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fill_diagonal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.fix`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fix.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.flatten`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.flatten.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.flip`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.flip.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.fliplr`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fliplr.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.flipud`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.flipud.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.float`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.float.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.float_power`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.float_power.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.float_power_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.float_power_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.floor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.floor.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.floor_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.floor_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.floor_divide`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.floor_divide.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.floor_divide_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.floor_divide_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.fmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.fmin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fmin.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.fmod`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fmod.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.frac`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.frac.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.frac_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.frac_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.frexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.frexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.gather`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.gather.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.gcd`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.gcd.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.gcd_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.gcd_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ge`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ge.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ge_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ge_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ger`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ger.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.get_device`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.get_device.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.grad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.grad.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.greater`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.greater.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.greater_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.greater_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.greater_equal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.greater_equal.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.greater_equal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.greater_equal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.gt`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.gt.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.gt_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.gt_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.half`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.half.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.hardshrink`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.hardshrink.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.heaviside`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.heaviside.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.histc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.histc.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.histogram`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.histogram.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.hsplit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.hsplit.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.hypot`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.hypot.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.hypot_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.hypot_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.i0`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.i0.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.i0_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.i0_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.imag`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.imag.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.index_add`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.index_add.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.index_add_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.index_add_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.index_copy_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.index_copy_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.index_fill`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.index_fill.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.index_fill_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.index_fill_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.index_put`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.index_put.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.index_put_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.index_put_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.index_select`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.index_select.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.indices`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.indices.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.inner`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.inner.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.int`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.int.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.inverse`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.inverse.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.is_complex`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.is_complex.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.is_contiguous`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.is_contiguous.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.is_cuda`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.is_cuda.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.is_floating_point`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.is_floating_point.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.is_leaf`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.is_leaf.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.is_pinned`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.is_pinned.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.is_signed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.is_signed.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.is_sparse`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.is_sparse.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.isclose`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.isclose.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.isfinite`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.isfinite.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.isinf`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.isinf.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.isnan`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.isnan.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.istft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.istft.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.item`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.item.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.kthvalue`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.kthvalue.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lcm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lcm.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lcm_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lcm_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ldexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ldexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ldexp_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ldexp_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.le`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.le.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.le_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.le_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lerp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lerp.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lerp_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lerp_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.less`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.less.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.less_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.less_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.less_equal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.less_equal.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.less_equal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.less_equal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lgamma`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lgamma.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lgamma_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lgamma_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.log`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.log.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.log10`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.log10.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.log10_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.log10_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.log1p`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.log1p.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.log1p_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.log1p_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.log2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.log2.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.log2_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.log2_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.log_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.log_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logaddexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logaddexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logaddexp2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logaddexp2.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logcumsumexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logcumsumexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logdet`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logdet.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logical_and`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_and.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logical_and_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_and_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logical_not`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_not.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logical_not_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_not_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logical_or`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_or.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logical_or_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_or_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logical_xor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_xor.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logical_xor_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_xor_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logit.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logit_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logit_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.logsumexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logsumexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.long`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.long.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lstsq`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lstsq.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lt`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lt.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lt_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lt_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.lu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.lu.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.mH`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.mH.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.mT`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.mT.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.masked_fill`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.masked_fill.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.masked_fill_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.masked_fill_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.masked_scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.masked_scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.masked_scatter_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.masked_scatter_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.masked_select`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.masked_select.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.matmul`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matmul.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.matrix_power`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matrix_power.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.maximum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.maximum.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.mean`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.mean.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.median`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.median.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.minimum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.minimum.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.mm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.mm.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.mode`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.mode.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.moveaxis`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.moveaxis.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.movedim`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.movedim.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.msort`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.msort.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.mul`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.mul.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.multinomial`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.multinomial.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.multiply`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.multiply.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.multiply_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.multiply_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.mv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.mv.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.nan_to_num`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nan_to_num.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.nan_to_num_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nan_to_num_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.nanmean`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nanmean.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.nanmedian`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nanmedian.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.nanquantile`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nanquantile.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.nansum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nansum.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.narrow`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.narrow.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.narrow_copy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.narrow_copy.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ndim`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ndim.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ndimension`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ndimension.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ne`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ne.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ne_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ne_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.neg`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.neg.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.neg_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.neg_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.negative`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.negative.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.negative_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.negative_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.nelement`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nelement.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.new_empty`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.new_empty.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.new_full`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.new_full.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.new_ones`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.new_ones.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.new_tensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.new_tensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.new_zeros`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.new_zeros.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.nextafter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nextafter.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.nonzero`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nonzero.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.normal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.normal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.not_equal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.not_equal.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.not_equal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.not_equal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.numel`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.numel.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.numpy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.numpy.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.outer`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.outer.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.permute`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.permute.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.pin_memory`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.pin_memory.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.pinverse`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.pinverse.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.polygamma`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.polygamma.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.polygamma_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.polygamma_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.pow`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.pow.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.pow_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.pow_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.prod`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.prod.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.qr`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.qr.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.quantile`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.quantile.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.rad2deg`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.rad2deg.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.ravel`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ravel.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.real`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.real.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.reciprocal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.reciprocal.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.reciprocal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.reciprocal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.register_hook`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.register_hook.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.remainder`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.remainder.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.remainder_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.remainder_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.renorm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.renorm.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.renorm_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.renorm_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.repeat`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.repeat.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.repeat_interleave`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.repeat_interleave.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.requires_grad_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.requires_grad_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.reshape`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.reshape.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.reshape_as`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.reshape_as.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.resize_as_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.resize_as_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.retain_grad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.retain_grad.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.roll`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.roll.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.rot90`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.rot90.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.round`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.round.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.round_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.round_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.rsqrt`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.rsqrt.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.rsqrt_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.rsqrt_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.scatter_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.scatter_add`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.scatter_add_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.select`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.select.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sgn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sgn.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.shape`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.shape.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.short`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.short.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sigmoid`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sigmoid.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sigmoid_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sigmoid_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sign`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sign.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.signbit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.signbit.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sin`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sin.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sin_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sin_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sinc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sinc.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sinh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sinh.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sinh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sinh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.size`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.size.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.slice_scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.slice_scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.slogdet`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.slogdet.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.softmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.softmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sort`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sort.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.split`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.split.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sparse_mask`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sparse_mask.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sqrt`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sqrt.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sqrt_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sqrt_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.square`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.square.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.square_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.square_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.squeeze`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.squeeze.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.squeeze_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.squeeze_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.std`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.std.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.stft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.stft.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sub`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sub.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sub_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sub_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.subtract`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.subtract.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.subtract_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.subtract_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.sum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.sum.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.svd`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.svd.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.swapaxes`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.swapaxes.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.swapdims`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.swapdims.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.symeig`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.symeig.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.t`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.t.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.take`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.take.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.take_along_dim`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.take_along_dim.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.tan`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.tan.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.tan_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.tan_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.tanh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.tanh.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.tanh_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.tanh_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.tensor_split`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.tensor_split.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.tile`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.tile.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.to`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.to.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.to_dense`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.to_dense.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.to_sparse`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.to_sparse.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.tolist`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.tolist.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.topk`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.topk.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.trace`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.trace.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.transpose`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.transpose.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.triangular_solve`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.triangular_solve.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.tril`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.tril.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.tril_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.tril_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.triu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.triu.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.triu_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.triu_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.true_divide`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.true_divide.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.true_divide_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.true_divide_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.trunc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.trunc.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.trunc_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.trunc_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.type`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.type.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.type_as`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.type_as.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.unbind`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unbind.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.unflatten`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unflatten.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.unfold`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unfold.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.uniform_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.uniform_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.unique`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unique.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.unique_consecutive`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unique_consecutive.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.unsqueeze`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unsqueeze.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.unsqueeze_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unsqueeze_.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.values`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.values.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.var`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.var.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.vdot`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.vdot.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.view`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.view.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.view_as`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.view_as.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.vsplit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.vsplit.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.where`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.where.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.xlogy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.xlogy.md) | -| REFERENCE-MAPPING-ITEM(`torch.Tensor.zero_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.zero_.md) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.bitwise_left_shift`, https://pytorch.org/docs/stable/generated/torch.Tensor.bitwise_left_shift.html#torch.Tensor.bitwise_left_shift) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.bitwise_right_shift`, https://pytorch.org/docs/stable/generated/torch.Tensor.bitwise_right_shift.html#torch.Tensor.bitwise_right_shift) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.copysign`, https://pytorch.org/docs/1.13/generated/torch.Tensor.copysign.html?highlight=torch+tensor+copysign#torch.Tensor.copysign) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.dequantize`, https://pytorch.org/docs/1.13/generated/torch.Tensor.dequantize.html?highlight=torch+tensor+dequantize#torch.Tensor.dequantize) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.geqrf`, https://pytorch.org/docs/1.13/generated/torch.Tensor.geqrf.html?highlight=torch+tensor+geqrf#torch.Tensor.geqrf) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.is_coalesced`, https://pytorch.org/docs/stable/generated/torch.Tensor.is_coalesced.html#torch.Tensor.is_coalesced) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.is_conj`, https://pytorch.org/docs/stable/generated/torch.Tensor.is_conj.html#torch.Tensor.is_conj) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.mul_`, https://pytorch.org/docs/stable/generated/torch.Tensor.mul_.html?highlight=mul_) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.resize_`, https://pytorch.org/docs/stable/generated/torch.Tensor.resize_.html?highlight=resize#torch.Tensor.resize_) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.resolve_conj`, https://pytorch.org/docs/stable/generated/torch.Tensor.resolve_conj.html#torch.Tensor.resolve_conj) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.resolve_neg`, https://pytorch.org/docs/stable/generated/torch.Tensor.resolve_neg.html#torch.Tensor.resolve_neg) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.scatter_reduce`, https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_reduce.html#torch.Tensor.scatter_reduce) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.select_scatter`, https://pytorch.org/docs/stable/generated/torch.Tensor.select_scatter.html#torch.Tensor.select_scatter) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.sparse_resize_`, https://pytorch.org/docs/stable/generated/torch.Tensor.sparse_resize_.html#torch.Tensor.sparse_resize_) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.sparse_resize_and_clear_`, https://pytorch.org/docs/stable/generated/torch.Tensor.sparse_resize_and_clear_.html#torch.Tensor.sparse_resize_and_clear_) | -| NOT-IMPLEMENTED-ITEM(`torch.Tensor.sum_to_size`, https://pytorch.org/docs/stable/generated/torch.Tensor.sum_to_size.html?highlight=sum_to_size#torch.Tensor.sum_to_size) | +| REFERENCE-MAPPING-TABLE(`torch.Tensor.`) | ***持续更新...*** @@ -1154,20 +74,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.calculate_gain`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.calculate_gain.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.constant_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.constant_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.dirac_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.dirac_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.eye_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.eye_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.kaiming_normal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.kaiming_normal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.kaiming_uniform_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.kaiming_uniform_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.normal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.normal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.ones_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.ones_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.orthogonal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.orthogonal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.trunc_normal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.trunc_normal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.uniform_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.uniform_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.xavier_normal_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.xavier_normal_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.xavier_uniform_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.xavier_uniform_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.init.zeros_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/init/torch.nn.init.zeros_.md) | +| REFERENCE-MAPPING-TABLE(`torch.nn.init.`) | ***持续更新...*** @@ -1176,16 +83,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.nn.utils.clip_grad_norm_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.nn.utils.clip_grad_norm_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.utils.clip_grad_value_`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.nn.utils.clip_grad_value_.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.utils.parameters_to_vector`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.utils.parameters_to_vector.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.utils.parametrizations.spectral_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.nn.utils.parametrizations.spectral_norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.utils.remove_weight_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.nn.utils.remove_weight_norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.utils.spectral_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.nn.utils.spectral_norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.utils.vector_to_parameters`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.utils.vector_to_parameters.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.utils.weight_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.nn.utils.weight_norm.md) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.utils.parametrize.is_parametrized`, https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.is_parametrized.html#torch.nn.utils.parametrize.is_parametrized) | - +| REFERENCE-MAPPING-TABLE(`torch.nn.utils.`) | ***持续更新...*** @@ -1194,43 +92,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.add_module`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.add_module.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.apply`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.apply.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.bfloat16`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.bfloat16.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.buffers`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.buffers.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.children`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.children.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.cpu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cpu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.cuda`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cuda.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.double`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.double.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.eval`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.eval.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.float`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.float.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.get_buffer`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.get_buffer.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.get_parameter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.get_parameter.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.get_submodule`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.get_submodule.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.half`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.half.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.load_state_dict`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.load_state_dict.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.modules`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nn.Module.modules.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.named_buffers`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.named_buffers.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.named_children`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.named_children.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.named_modules`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.named_modules.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.named_parameters`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.named_parameters.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.parameters`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.parameters.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.register_buffer`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.register_buffer.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.register_forward_hook`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.register_forward_hook.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.register_forward_pre_hook`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.register_forward_pre_hook.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.register_module`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.register_module.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.register_parameter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.register_parameter.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.state_dict`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.state_dict.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.to`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.to.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.train`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.train.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.type`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.type.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.xpu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.xpu.md) | -| REFERENCE-MAPPING-ITEM(`torch.nn.Module.zero_grad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.zero_grad.md) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.Module.register_full_backward_hook`, https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.Module.register_full_backward_pre_hook`, https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_pre_hook) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.Module.requires_grad_`, https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.requires_grad_) | -| NOT-IMPLEMENTED-ITEM(`torch.nn.Module.to_empty`, https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.to_empty) | - +| REFERENCE-MAPPING-TABLE(`torch.nn.Module.`) | ***持续更新...*** @@ -1239,25 +101,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.autograd.Function`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.Function.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.Function.backward`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.Function.backward.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.Function.forward`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.Function.forward.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.backward`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.backward.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.enable_grad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.enable_grad.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.function.FunctionCtx`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.autograd.function.FunctionCtx.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.function.FunctionCtx.mark_non_differentiable`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.function.FunctionCtx.mark_non_differentiable.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.function.FunctionCtx.save_for_backward`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.function.FunctionCtx.save_for_backward.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.function.FunctionCtx.set_materialize_grads`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.function.FunctionCtx.set_materialize_grads.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.functional.hessian`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.functional.hessian.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.functional.jacobian`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.functional.jacobian.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.functional.jvp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.functional.jvp.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.functional.vjp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.functional.vjp.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.grad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.grad.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.graph.saved_tensors_hooks`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.graph.saved_tensors_hooks.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.profiler.profile.export_chrome_trace`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.profiler.profile.export_chrome_trace.md) | -| REFERENCE-MAPPING-ITEM(`torch.autograd.set_grad_enabled`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.set_grad_enabled.md) | -| NOT-IMPLEMENTED-ITEM(`torch.autograd.function.FunctionCtx.mark_dirty`, https://pytorch.org/docs/stable/generated/torch.autograd.function.FunctionCtx.mark_dirty.html#torch.autograd.function.FunctionCtx.mark_dirty) | -| NOT-IMPLEMENTED-ITEM(`torch.autograd.profiler.profile.self_cpu_time_total`, https://pytorch.org/docs/stable/generated/torch.autograd.profiler.profile.self_cpu_time_total.html#torch.autograd.profiler.profile.self_cpu_time_total) | +| REFERENCE-MAPPING-TABLE(`torch.autograd.`) | ***持续更新...*** @@ -1266,53 +110,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.cuda.BoolTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.BoolTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.ByteTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ByteTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.DoubleTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.DoubleTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.Event`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.Event.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.FloatTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.FloatTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.HalfTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.HalfTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.IntTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.IntTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.LongTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.LongTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.ShortTensor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ShortTensor.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.Stream`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.Stream__upper.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.amp.GradScaler`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.GradScaler.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.amp.autocast`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.autocast.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.comm.broadcast`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.comm.broadcast.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.current_device`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.current_device.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.current_stream`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.current_stream.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.device`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.device.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.device_count`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.device_count.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.empty_cache`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.empty_cache.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.get_device_capability`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.get_device_capability.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.get_device_name`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.get_device_name.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.get_device_properties`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.get_device_properties.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.get_rng_state_all`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.get_rng_state_all.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.initial_seed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.initial_seed.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.is_available`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.is_available.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.manual_seed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.manual_seed.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.manual_seed_all`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.manual_seed_all.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.max_memory_allocated`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.max_memory_allocated.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.max_memory_reserved`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.max_memory_reserved.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.memory_allocated`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.memory_allocated.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.memory_reserved`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.memory_reserved.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.nvtx.range_pop`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.nvtx.range_pop.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.nvtx.range_push`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.nvtx.range_push.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.set_device`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.set_device.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.set_rng_state_all`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.set_rng_state_all.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.set_stream`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.set_stream.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.stream`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.stream.md) | -| REFERENCE-MAPPING-ITEM(`torch.cuda.synchronize`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.synchronize.md) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.StreamContext`, https://pytorch.org/docs/stable/generated/torch.cuda.StreamContext.html#torch.cuda.StreamContext) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.comm.gather`, https://pytorch.org/docs/stable/generated/torch.cuda.comm.gather.html#torch-cuda-comm-gather) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.comm.scatter`, https://pytorch.org/docs/stable/generated/torch.cuda.comm.scatter.html#torch-cuda-comm-scatter) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.default_stream`, https://pytorch.org/docs/stable/generated/torch.cuda.default_stream.html?highlight=torch+cuda+default_stream#torch.cuda.default_stream) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.get_arch_list`, https://pytorch.org/docs/stable/generated/torch.cuda.get_arch_list.html?highlight=torch+cuda+get_arch_list#torch.cuda.get_arch_list) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.is_current_stream_capturing`, https://pytorch.org/docs/stable/generated/torch.cuda.is_current_stream_capturing.html#torch.cuda.is_current_stream_capturing) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.is_initialized`, https://pytorch.org/docs/stable/generated/torch.cuda.is_initialized.html?highlight=torch+cuda+is_initialized#torch.cuda.is_initialized) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.mem_get_info`, https://pytorch.org/docs/stable/generated/torch.cuda.mem_get_info.html#torch.cuda.mem_get_info) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.memory_usage`, https://pytorch.org/docs/stable/generated/torch.cuda.memory_usage.html#torch.cuda.memory_usage) | -| NOT-IMPLEMENTED-ITEM(`torch.cuda.set_rng_state`, https://pytorch.org/docs/stable/generated/torch.cuda.set_rng_state.html#torch.cuda.set_rng_state) | +| REFERENCE-MAPPING-TABLE(`torch.cuda.`) | ***持续更新...*** @@ -1321,45 +119,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.distributed.ReduceOp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.ReduceOp.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.ReduceOp.MAX`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.distributed.ReduceOp.MAX.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.ReduceOp.MIN`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.distributed.ReduceOp.MIN.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.ReduceOp.PRODUCT`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.distributed.ReduceOp.PRODUCT.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.ReduceOp.SUM`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.distributed.ReduceOp.SUM.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.all_gather`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.all_gather.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.all_gather_object`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.all_gather_object.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.all_reduce`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.all_reduce.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.all_to_all`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.all_to_all.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.barrier`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.barrier.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.broadcast`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.broadcast.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.broadcast_object_list`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.broadcast_object_list.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.gather`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.gather.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.get_backend`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.get_backend.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.get_rank`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.get_rank.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.get_world_size`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.get_world_size.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.init_process_group`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.init_process_group.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.irecv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.irecv.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.isend`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.isend.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.new_group`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.new_group.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.recv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.recv.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.reduce`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.reduce.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.reduce_scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.reduce_scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.rpc.get_worker_info`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.rpc.get_worker_info.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.rpc.init_rpc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.rpc.init_rpc.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.rpc.rpc_async`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.rpc.rpc_async.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.rpc.rpc_sync`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.rpc.rpc_sync.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.rpc.shutdown`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.rpc.shutdown.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.scatter`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.scatter.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.scatter_object_list`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.scatter_object_list.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributed.send`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.send.md) | -| NOT-IMPLEMENTED-ITEM(`torch.distributed.all_gather_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather_multigpu) | -| NOT-IMPLEMENTED-ITEM(`torch.distributed.all_reduce_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce_multigpu) | -| NOT-IMPLEMENTED-ITEM(`torch.distributed.broadcast_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.broadcast_multigpu) | -| NOT-IMPLEMENTED-ITEM(`torch.distributed.reduce_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.reduce_multigpu) | -| NOT-IMPLEMENTED-ITEM(`torch.distributed.reduce_scatter_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.reduce_scatter_multigpu) | -| NOT-IMPLEMENTED-ITEM(`torch.distributed.rpc.WorkerInfo`, https://pytorch.org/docs/stable/rpc.html#torch.distributed.rpc.WorkerInfo) | -| NOT-IMPLEMENTED-ITEM(`torch.distributed.rpc.functions.async_execution`, https://pytorch.org/docs/stable/rpc.html#torch.distributed.rpc.functions.async_execution) | - +| REFERENCE-MAPPING-TABLE(`torch.distributed.`) | ***持续更新...*** @@ -1368,50 +128,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.distributions.Distribution.log_prob`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.Distribution.log_prob.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.Distribution.rsample`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.Distribution.rsample.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.Distribution.sample`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.Distribution.sample.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.bernoulli.Bernoulli`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.bernoulli.Bernoulli.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.beta.Beta`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.beta.Beta.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.categorical.Categorical`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.categorical.Categorical.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.cauchy.Cauchy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.cauchy.Cauchy.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.dirichlet.Dirichlet`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.dirichlet.Dirichlet.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.distribution.Distribution`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.distribution.Distribution.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.exp_family.ExponentialFamily`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.exp_family.ExponentialFamily.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.geometric.Geometric`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.geometric.Geometric.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.gumbel.Gumbel`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.gumbel.Gumbel.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.independent.Independent`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.independent.Independent.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.kl.kl_divergence`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.kl.kl_divergence.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.kl.register_kl`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.kl.register_kl.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.laplace.Laplace`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.laplace.Laplace.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.log_normal.LogNormal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.log_normal.LogNormal.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.multinomial.Multinomial`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.multinomial.Multinomial.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.normal.Normal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.normal.Normal.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transformed_distribution.TransformedDistribution`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transformed_distribution.TransformedDistribution.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.AbsTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.AbsTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.AffineTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.AffineTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.ComposeTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.ComposeTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.ExpTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.ExpTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.IndependentTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.IndependentTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.PowerTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.PowerTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.ReshapeTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.ReshapeTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.SigmoidTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.SigmoidTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.SoftmaxTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.SoftmaxTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.StackTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.StackTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.StickBreakingTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.StickBreakingTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.TanhTransform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.TanhTransform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.transforms.Transform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.transforms.Transform.md) | -| REFERENCE-MAPPING-ITEM(`torch.distributions.uniform.Uniform`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/distributions/torch.distributions.uniform.Uniform.md) | -| NOT-IMPLEMENTED-ITEM(`torch.distributions.binomial.Binomial`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.binomial.Binomial) | -| NOT-IMPLEMENTED-ITEM(`torch.distributions.constraint_registry.ConstraintRegistry`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.constraint_registry.ConstraintRegistry) | -| NOT-IMPLEMENTED-ITEM(`torch.distributions.constraints.Constraint`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.constraints.Constraint) | -| NOT-IMPLEMENTED-ITEM(`torch.distributions.continuous_bernoulli.ContinuousBernoulli`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.continuous_bernoulli.ContinuousBernoulli) | -| NOT-IMPLEMENTED-ITEM(`torch.distributions.exponential.Exponential`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.exponential.Exponential) | -| NOT-IMPLEMENTED-ITEM(`torch.distributions.one_hot_categorical.OneHotCategorical`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.one_hot_categorical.OneHotCategorical) | -| NOT-IMPLEMENTED-ITEM(`torch.distributions.transforms.CatTransform`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.transforms.CatTransform) | -| NOT-IMPLEMENTED-ITEM(`torch.distributions.transforms.CumulativeDistributionTransform`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.transforms.CumulativeDistributionTransform) | -| NOT-IMPLEMENTED-ITEM(`torch.distributions.transforms.SoftplusTransform`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.transforms.SoftplusTransform) | - +| REFERENCE-MAPPING-TABLE(`torch.distributions.`) | ***持续更新...*** @@ -1420,29 +137,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.fft.fft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.fft.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.fft2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.fft2.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.fftfreq`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.fftfreq.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.fftn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.fftn.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.fftshift`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.fftshift.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.hfft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.hfft.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.hfft2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.hfft2.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.hfftn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.hfftn.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.ifft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.ifft.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.ifft2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.ifft2.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.ifftn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.ifftn.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.ifftshift`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.ifftshift.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.ihfft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.ihfft.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.ihfft2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.ihfft2.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.ihfftn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.ihfftn.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.irfft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.irfft.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.irfft2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.irfft2.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.irfftn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.irfftn.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.rfft`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.rfft.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.rfft2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.rfft2.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.rfftfreq`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.rfftfreq.md) | -| REFERENCE-MAPPING-ITEM(`torch.fft.rfftn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/fft/torch.fft.rfftn.md) | - +| REFERENCE-MAPPING-TABLE(`torch.fft.`) | ***持续更新...*** @@ -1452,12 +147,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.hub.download_url_to_file`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/hub/torch.hub.download_url_to_file.md) | -| REFERENCE-MAPPING-ITEM(`torch.hub.help`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/hub/torch.hub.help.md) | -| REFERENCE-MAPPING-ITEM(`torch.hub.list`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/hub/torch.hub.list.md) | -| REFERENCE-MAPPING-ITEM(`torch.hub.load`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/hub/torch.hub.load.md) | -| NOT-IMPLEMENTED-ITEM(`torch.hub.get_dir`, https://pytorch.org/docs/stable/hub.html?highlight=torch+hub+get_dir#torch.hub.get_dir) | -| NOT-IMPLEMENTED-ITEM(`torch.hub.set_dir`, https://pytorch.org/docs/stable/hub.html?highlight=torch+hub+get_dir#torch.hub.set_dir) | +| REFERENCE-MAPPING-TABLE(`torch.hub.`) | ***持续更新...*** @@ -1468,39 +158,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.linalg.cholesky`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.cholesky.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.cond`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.cond.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.cross`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.cross.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.det`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.det.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.diagonal`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.diagonal.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.eig`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.eig.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.eigh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.eigh.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.eigvals`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.eigvals.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.eigvalsh`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.eigvalsh.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.householder_product`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.householder_product.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.inv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.inv.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.lstsq`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.lstsq.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.lu`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.lu.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.lu_factor`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.lu_factor.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.lu_factor_ex`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.lu_factor_ex.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.matmul`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.matmul.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.matrix_exp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.matrix_exp.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.matrix_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.matrix_norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.matrix_power`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.matrix_power.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.matrix_rank`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.matrix_rank.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.multi_dot`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.multi_dot.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.norm.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.pinv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.pinv.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.qr`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.qr.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.slogdet`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.slogdet.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.solve`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.solve.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.solve_triangular`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.solve_triangular.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.svd`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.svd.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.svdvals`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.svdvals.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.vander`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.vander.md) | -| REFERENCE-MAPPING-ITEM(`torch.linalg.vector_norm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.vector_norm.md) | -| NOT-IMPLEMENTED-ITEM(`torch.linalg.cholesky_ex`, https://pytorch.org/docs/stable/generated/torch.linalg.cholesky_ex.html#torch.linalg.cholesky_ex) | -| NOT-IMPLEMENTED-ITEM(`torch.linalg.inv_ex`, https://pytorch.org/docs/stable/generated/torch.linalg.inv_ex.html#torch.linalg.inv_ex) | +| REFERENCE-MAPPING-TABLE(`torch.linalg.`) | ***持续更新...*** @@ -1510,9 +168,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.onnx.export`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.onnx.export.md) | -| NOT-IMPLEMENTED-ITEM(`torch.onnx.disable_log`, https://pytorch.org/docs/stable/onnx.html#torch.onnx.disable_log) | -| NOT-IMPLEMENTED-ITEM(`torch.onnx.enable_log`, https://pytorch.org/docs/stable/onnx.html#torch.onnx.enable_log) | +| REFERENCE-MAPPING-TABLE(`torch.onnx.`) | ***持续更新...*** @@ -1521,33 +177,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.optim.ASGD`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.ASGD.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Adadelta`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Adadelta.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Adagrad`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Adagrad.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Adam`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Adam.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.AdamW`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.AdamW.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Adamax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Adamax.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.LBFGS`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.LBFGS.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Optimizer`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Optimizer.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Optimizer.add_param_group`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.optim.Optimizer.add_param_group.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Optimizer.load_state_dict`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Optimizer.load_state_dict.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Optimizer.state_dict`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Optimizer.state_dict.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Optimizer.step`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Optimizer.step.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.RMSprop`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.RMSprop.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.Rprop`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Rprop.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.SGD`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.SGD.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.ConstantLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.ConstantLR.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.CosineAnnealingLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.CosineAnnealingLR.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.CosineAnnealingWarmRestarts`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.CosineAnnealingWarmRestarts.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.CyclicLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.CyclicLR.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.ExponentialLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.ExponentialLR.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.LambdaLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.LambdaLR.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.LinearLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.LinearLR.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.MultiStepLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.MultiStepLR.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.MultiplicativeLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.MultiplicativeLR.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.OneCycleLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.OneCycleLR.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.ReduceLROnPlateau`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.ReduceLROnPlateau.md) | -| REFERENCE-MAPPING-ITEM(`torch.optim.lr_scheduler.StepLR`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.lr_scheduler.StepLR.md) | +| REFERENCE-MAPPING-TABLE(`torch.optim.`) | ***持续更新...*** @@ -1557,10 +187,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.profiler.profile`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.profiler.profile.md) | -| REFERENCE-MAPPING-ITEM(`torch.profiler.schedule`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.profiler.schedule.md) | -| NOT-IMPLEMENTED-ITEM(`torch.profiler.ProfilerAction`, https://pytorch.org/docs/stable/profiler.html#torch.profiler.ProfilerAction) | -| NOT-IMPLEMENTED-ITEM(`torch.profiler.ProfilerActivity`, https://pytorch.org/docs/stable/profiler.html#torch.profiler.ProfilerActivity) | +| REFERENCE-MAPPING-TABLE(`torch.profiler.`) | ***持续更新...*** @@ -1570,11 +197,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.sparse.addmm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/sparse/torch.sparse.addmm.md) | -| REFERENCE-MAPPING-ITEM(`torch.sparse.mm`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/sparse/torch.sparse.mm.md) | -| REFERENCE-MAPPING-ITEM(`torch.sparse.softmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/sparse/torch.sparse.softmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.sparse.sum`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/sparse/torch.sparse.sum.md) | -| NOT-IMPLEMENTED-ITEM(`torch.sparse.sampled_addmm`, https://pytorch.org/docs/stable/generated/torch.sparse.sampled_addmm.html#torch.sparse.sampled_addmm) | +| REFERENCE-MAPPING-TABLE(`torch.sparse.`) | ***持续更新...*** @@ -1584,76 +207,7 @@ | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | -| REFERENCE-MAPPING-ITEM(`torch.backends.cuda.is_built`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.backends.cuda.is_built.md) | -| REFERENCE-MAPPING-ITEM(`torch.backends.cudnn.is_available`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.backends.cudnn.is_available.md) | -| REFERENCE-MAPPING-ITEM(`torch.backends.cudnn.version`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.backends.cudnn.version.md) | -| REFERENCE-MAPPING-ITEM(`torch.cpu.amp.autocast`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.cpu.amp.autocast.md) | -| REFERENCE-MAPPING-ITEM(`torch.jit.load`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.jit.load.md) | -| REFERENCE-MAPPING-ITEM(`torch.multiprocessing.spawn`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.multiprocessing.spawn.md) | -| REFERENCE-MAPPING-ITEM(`torch.random.get_rng_state`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.random.get_rng_state.md) | -| REFERENCE-MAPPING-ITEM(`torch.random.initial_seed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.random.initial_seed.md) | -| REFERENCE-MAPPING-ITEM(`torch.random.manual_seed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.random.manual_seed.md) | -| REFERENCE-MAPPING-ITEM(`torch.random.seed`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.random.seed.md) | -| REFERENCE-MAPPING-ITEM(`torch.random.set_rng_state`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.random.set_rng_state.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.digamma`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.digamma.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.erf`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.erf.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.erfc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.erfc.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.erfcx`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.erfcx.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.erfinv`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.erfinv.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.exp2`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.exp2.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.expit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.expit.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.expm1`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.expm1.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.gammaln`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.gammaln.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.i0`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.i0.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.i0e`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.i0e.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.i1`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.i1.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.i1e`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.i1e.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.log1p`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.log1p.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.log_softmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.log_softmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.logit`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.logit.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.logsumexp`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.logsumexp.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.multigammaln`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.multigammaln.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.ndtri`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.ndtri.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.polygamma`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.polygamma.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.psi`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.psi.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.round`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.round.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.sinc`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.sinc.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.softmax`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.softmax.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.xlog1py`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.xlog1py.md) | -| REFERENCE-MAPPING-ITEM(`torch.special.xlogy`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.special.xlogy.md) | -| REFERENCE-MAPPING-ITEM(`torch.testing.assert_allclose`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.testing.assert_allclose.md) | -| REFERENCE-MAPPING-ITEM(`torch.testing.assert_close`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.testing.assert_close.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.cpp_extension.BuildExtension`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.cpp_extension.BuildExtension.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.cpp_extension.BuildExtension.with_options`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.cpp_extension.BuildExtension.with_options.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.cpp_extension.CUDAExtension`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.cpp_extension.CUDAExtension.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.cpp_extension.CUDA_HOME`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.cpp_extension.CUDA_HOME.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.cpp_extension.CppExtension`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.cpp_extension.CppExtension.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.cpp_extension.load`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.cpp_extension.load.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.BatchSampler`, https://github.com/PaddlePaddle/X2Paddle/tree/develop/docs/pytorch_project_convertor/API_docs/utils/torch.utils.data.BatchSampler.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.ChainDataset`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.ChainDataset.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.ConcatDataset`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.ConcatDataset.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.DataLoader`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.DataLoader.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.Dataset`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.Dataset.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.DistributedSampler`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.DistributedSampler.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.IterableDataset`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.IterableDataset.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.RandomSampler`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.RandomSampler.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.Sampler`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.Sampler.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.SequentialSampler`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.utils.data.SequentialSampler.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.Subset`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.Subset.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.SubsetRandomSampler`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.SubsetRandomSampler.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.TensorDataset`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.TensorDataset.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.WeightedRandomSampler`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.WeightedRandomSampler.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data._utils.collate.default_collate`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data._utils.collate.default_collate.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.dataloader.default_collate`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.dataloader.default_collate.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.default_collate`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.default_collate.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.distributed.DistributedSampler`, https://github.com/PaddlePaddle/X2Paddle/tree/develop/docs/pytorch_project_convertor/API_docs/utils/torch.utils.data.distributed.DistributedSampler.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.get_worker_info`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.get_worker_info.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.data.random_split`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.random_split.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.dlpack.from_dlpack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.dlpack.from_dlpack.md) | -| REFERENCE-MAPPING-ITEM(`torch.utils.dlpack.to_dlpack`, https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.dlpack.to_dlpack.md) | -| NOT-IMPLEMENTED-ITEM(`torch.special.entr`, https://pytorch.org/docs/stable/special.html#torch.special.entr) | -| NOT-IMPLEMENTED-ITEM(`torch.utils.cpp_extension.include_paths`, https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.include_paths) | -| NOT-IMPLEMENTED-ITEM(`torch.utils.cpp_extension.load_inline`, https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.load_inline) | +| REFERENCE-MAPPING-TABLE(`torch.`) | ***持续更新...*** @@ -1772,3 +326,93 @@ | ALIAS-REFERENCE-ITEM(`torch.utils.data.sampler.SequentialSampler`, `torch.utils.data.SequentialSampler`) | | ALIAS-REFERENCE-ITEM(`torch.utils.data.sampler.SubsetRandomSampler`, `torch.utils.data.SubsetRandomSampler`) | | ALIAS-REFERENCE-ITEM(`torch.utils.data.sampler.WeightedRandomSampler`, `torch.utils.data.WeightedRandomSampler`) | + + ## 尚未实现的 API 列表 + +| 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | +| ----- | ----------- | ----------------- | ----------- | ------- | +| NOT-IMPLEMENTED-ITEM(`torch.bitwise_left_shift`, https://pytorch.org/docs/stable/generated/torch.bitwise_left_shift.html#torch.bitwise_left_shift) | +| NOT-IMPLEMENTED-ITEM(`torch.bitwise_right_shift`, https://pytorch.org/docs/stable/generated/torch.bitwise_right_shift.html#torch.bitwise_right_shift) | +| NOT-IMPLEMENTED-ITEM(`torch.can_cast`, https://pytorch.org/docs/stable/generated/torch.can_cast.html) | +| NOT-IMPLEMENTED-ITEM(`torch.geqrf`, https://pytorch.org/docs/stable/generated/torch.geqrf.html?highlight=geqrf#torch.geqrf) | +| NOT-IMPLEMENTED-ITEM(`torch.get_deterministic_debug_mode`, https://pytorch.org/docs/stable/generated/torch.get_deterministic_debug_mode.html#torch.get_deterministic_debug_mode) | +| NOT-IMPLEMENTED-ITEM(`torch.gradient`, https://pytorch.org/docs/stable/generated/torch.gradient.html#torch.gradient) | +| NOT-IMPLEMENTED-ITEM(`torch.is_conj`, https://pytorch.org/docs/stable/generated/torch.is_conj.html#torch.is_conj) | +| NOT-IMPLEMENTED-ITEM(`torch.isin`, https://pytorch.org/docs/stable/generated/torch.isin.html#torch.isin) | +| NOT-IMPLEMENTED-ITEM(`torch.layout`, https://pytorch.org/docs/stable/tensor_attributes.html#torch.layout) | +| NOT-IMPLEMENTED-ITEM(`torch.positive`, https://pytorch.org/docs/stable/generated/torch.positive.html#torch.positive) | +| NOT-IMPLEMENTED-ITEM(`torch.resolve_conj`, https://pytorch.org/docs/stable/generated/torch.resolve_conj.html#torch.resolve_conj) | +| NOT-IMPLEMENTED-ITEM(`torch.resolve_neg`, https://pytorch.org/docs/stable/generated/torch.resolve_neg.html#torch.resolve_neg) | +| NOT-IMPLEMENTED-ITEM(`torch.scatter_reduce`, https://pytorch.org/docs/stable/generated/torch.scatter_reduce.html#torch.scatter_reduce) | +| NOT-IMPLEMENTED-ITEM(`torch.set_deterministic_debug_mode`, https://pytorch.org/docs/stable/generated/torch.set_deterministic_debug_mode.html#torch.set_deterministic_debug_mode) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.GLU`, https://pytorch.org/docs/stable/generated/torch.nn.GLU.html#torch.nn.GLU) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyBatchNorm1d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyBatchNorm1d.html#torch.nn.LazyBatchNorm1d) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyBatchNorm2d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyBatchNorm2d.html#torch.nn.LazyBatchNorm2d) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyBatchNorm3d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyBatchNorm3d.html#torch.nn.LazyBatchNorm3d) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyInstanceNorm1d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyInstanceNorm1d.html#torch.nn.LazyInstanceNorm1d) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyInstanceNorm2d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyInstanceNorm2d.html#torch.nn.LazyInstanceNorm2d) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.LazyInstanceNorm3d`, https://pytorch.org/docs/stable/generated/torch.nn.LazyInstanceNorm3d.html#torch.nn.LazyInstanceNorm3d) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.functional.multilabel_margin_loss`, https://pytorch.org/docs/stable/generated/torch.nn.functional.multilabel_margin_loss.html#torch.nn.functional.multilabel_margin_loss) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.bitwise_left_shift`, https://pytorch.org/docs/stable/generated/torch.Tensor.bitwise_left_shift.html#torch.Tensor.bitwise_left_shift) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.bitwise_right_shift`, https://pytorch.org/docs/stable/generated/torch.Tensor.bitwise_right_shift.html#torch.Tensor.bitwise_right_shift) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.copysign`, https://pytorch.org/docs/1.13/generated/torch.Tensor.copysign.html?highlight=torch+tensor+copysign#torch.Tensor.copysign) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.dequantize`, https://pytorch.org/docs/1.13/generated/torch.Tensor.dequantize.html?highlight=torch+tensor+dequantize#torch.Tensor.dequantize) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.geqrf`, https://pytorch.org/docs/1.13/generated/torch.Tensor.geqrf.html?highlight=torch+tensor+geqrf#torch.Tensor.geqrf) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.is_coalesced`, https://pytorch.org/docs/stable/generated/torch.Tensor.is_coalesced.html#torch.Tensor.is_coalesced) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.is_conj`, https://pytorch.org/docs/stable/generated/torch.Tensor.is_conj.html#torch.Tensor.is_conj) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.mul_`, https://pytorch.org/docs/stable/generated/torch.Tensor.mul_.html?highlight=mul_) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.resize_`, https://pytorch.org/docs/stable/generated/torch.Tensor.resize_.html?highlight=resize#torch.Tensor.resize_) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.resolve_conj`, https://pytorch.org/docs/stable/generated/torch.Tensor.resolve_conj.html#torch.Tensor.resolve_conj) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.resolve_neg`, https://pytorch.org/docs/stable/generated/torch.Tensor.resolve_neg.html#torch.Tensor.resolve_neg) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.scatter_reduce`, https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_reduce.html#torch.Tensor.scatter_reduce) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.select_scatter`, https://pytorch.org/docs/stable/generated/torch.Tensor.select_scatter.html#torch.Tensor.select_scatter) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.sparse_resize_`, https://pytorch.org/docs/stable/generated/torch.Tensor.sparse_resize_.html#torch.Tensor.sparse_resize_) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.sparse_resize_and_clear_`, https://pytorch.org/docs/stable/generated/torch.Tensor.sparse_resize_and_clear_.html#torch.Tensor.sparse_resize_and_clear_) | +| NOT-IMPLEMENTED-ITEM(`torch.Tensor.sum_to_size`, https://pytorch.org/docs/stable/generated/torch.Tensor.sum_to_size.html?highlight=sum_to_size#torch.Tensor.sum_to_size) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.utils.parametrize.is_parametrized`, https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.is_parametrized.html#torch.nn.utils.parametrize.is_parametrized) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.Module.register_full_backward_hook`, https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.Module.register_full_backward_pre_hook`, https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_pre_hook) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.Module.requires_grad_`, https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.requires_grad_) | +| NOT-IMPLEMENTED-ITEM(`torch.nn.Module.to_empty`, https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.to_empty) | +| NOT-IMPLEMENTED-ITEM(`torch.autograd.function.FunctionCtx.mark_dirty`, https://pytorch.org/docs/stable/generated/torch.autograd.function.FunctionCtx.mark_dirty.html#torch.autograd.function.FunctionCtx.mark_dirty) | +| NOT-IMPLEMENTED-ITEM(`torch.autograd.profiler.profile.self_cpu_time_total`, https://pytorch.org/docs/stable/generated/torch.autograd.profiler.profile.self_cpu_time_total.html#torch.autograd.profiler.profile.self_cpu_time_total) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.StreamContext`, https://pytorch.org/docs/stable/generated/torch.cuda.StreamContext.html#torch.cuda.StreamContext) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.comm.gather`, https://pytorch.org/docs/stable/generated/torch.cuda.comm.gather.html#torch-cuda-comm-gather) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.comm.scatter`, https://pytorch.org/docs/stable/generated/torch.cuda.comm.scatter.html#torch-cuda-comm-scatter) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.default_stream`, https://pytorch.org/docs/stable/generated/torch.cuda.default_stream.html?highlight=torch+cuda+default_stream#torch.cuda.default_stream) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.get_arch_list`, https://pytorch.org/docs/stable/generated/torch.cuda.get_arch_list.html?highlight=torch+cuda+get_arch_list#torch.cuda.get_arch_list) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.is_current_stream_capturing`, https://pytorch.org/docs/stable/generated/torch.cuda.is_current_stream_capturing.html#torch.cuda.is_current_stream_capturing) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.is_initialized`, https://pytorch.org/docs/stable/generated/torch.cuda.is_initialized.html?highlight=torch+cuda+is_initialized#torch.cuda.is_initialized) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.mem_get_info`, https://pytorch.org/docs/stable/generated/torch.cuda.mem_get_info.html#torch.cuda.mem_get_info) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.memory_usage`, https://pytorch.org/docs/stable/generated/torch.cuda.memory_usage.html#torch.cuda.memory_usage) | +| NOT-IMPLEMENTED-ITEM(`torch.cuda.set_rng_state`, https://pytorch.org/docs/stable/generated/torch.cuda.set_rng_state.html#torch.cuda.set_rng_state) | +| NOT-IMPLEMENTED-ITEM(`torch.distributed.all_gather_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather_multigpu) | +| NOT-IMPLEMENTED-ITEM(`torch.distributed.all_reduce_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce_multigpu) | +| NOT-IMPLEMENTED-ITEM(`torch.distributed.broadcast_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.broadcast_multigpu) | +| NOT-IMPLEMENTED-ITEM(`torch.distributed.reduce_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.reduce_multigpu) | +| NOT-IMPLEMENTED-ITEM(`torch.distributed.reduce_scatter_multigpu`, https://pytorch.org/docs/stable/distributed.html#torch.distributed.reduce_scatter_multigpu) | +| NOT-IMPLEMENTED-ITEM(`torch.distributed.rpc.WorkerInfo`, https://pytorch.org/docs/stable/rpc.html#torch.distributed.rpc.WorkerInfo) | +| NOT-IMPLEMENTED-ITEM(`torch.distributed.rpc.functions.async_execution`, https://pytorch.org/docs/stable/rpc.html#torch.distributed.rpc.functions.async_execution) | +| NOT-IMPLEMENTED-ITEM(`torch.distributions.binomial.Binomial`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.binomial.Binomial) | +| NOT-IMPLEMENTED-ITEM(`torch.distributions.constraint_registry.ConstraintRegistry`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.constraint_registry.ConstraintRegistry) | +| NOT-IMPLEMENTED-ITEM(`torch.distributions.constraints.Constraint`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.constraints.Constraint) | +| NOT-IMPLEMENTED-ITEM(`torch.distributions.continuous_bernoulli.ContinuousBernoulli`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.continuous_bernoulli.ContinuousBernoulli) | +| NOT-IMPLEMENTED-ITEM(`torch.distributions.exponential.Exponential`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.exponential.Exponential) | +| NOT-IMPLEMENTED-ITEM(`torch.distributions.one_hot_categorical.OneHotCategorical`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.one_hot_categorical.OneHotCategorical) | +| NOT-IMPLEMENTED-ITEM(`torch.distributions.transforms.CatTransform`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.transforms.CatTransform) | +| NOT-IMPLEMENTED-ITEM(`torch.distributions.transforms.CumulativeDistributionTransform`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.transforms.CumulativeDistributionTransform) | +| NOT-IMPLEMENTED-ITEM(`torch.distributions.transforms.SoftplusTransform`, https://pytorch.org/docs/stable/distributions.html#torch.distributions.transforms.SoftplusTransform) | +| NOT-IMPLEMENTED-ITEM(`torch.hub.get_dir`, https://pytorch.org/docs/stable/hub.html?highlight=torch+hub+get_dir#torch.hub.get_dir) | +| NOT-IMPLEMENTED-ITEM(`torch.hub.set_dir`, https://pytorch.org/docs/stable/hub.html?highlight=torch+hub+get_dir#torch.hub.set_dir) | +| NOT-IMPLEMENTED-ITEM(`torch.linalg.cholesky_ex`, https://pytorch.org/docs/stable/generated/torch.linalg.cholesky_ex.html#torch.linalg.cholesky_ex) | +| NOT-IMPLEMENTED-ITEM(`torch.linalg.inv_ex`, https://pytorch.org/docs/stable/generated/torch.linalg.inv_ex.html#torch.linalg.inv_ex) | +| NOT-IMPLEMENTED-ITEM(`torch.onnx.disable_log`, https://pytorch.org/docs/stable/onnx.html#torch.onnx.disable_log) | +| NOT-IMPLEMENTED-ITEM(`torch.onnx.enable_log`, https://pytorch.org/docs/stable/onnx.html#torch.onnx.enable_log) | +| NOT-IMPLEMENTED-ITEM(`torch.profiler.ProfilerAction`, https://pytorch.org/docs/stable/profiler.html#torch.profiler.ProfilerAction) | +| NOT-IMPLEMENTED-ITEM(`torch.profiler.ProfilerActivity`, https://pytorch.org/docs/stable/profiler.html#torch.profiler.ProfilerActivity) | +| NOT-IMPLEMENTED-ITEM(`torch.sparse.sampled_addmm`, https://pytorch.org/docs/stable/generated/torch.sparse.sampled_addmm.html#torch.sparse.sampled_addmm) | +| NOT-IMPLEMENTED-ITEM(`torch.special.entr`, https://pytorch.org/docs/stable/special.html#torch.special.entr) | +| NOT-IMPLEMENTED-ITEM(`torch.utils.cpp_extension.include_paths`, https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.include_paths) | +| NOT-IMPLEMENTED-ITEM(`torch.utils.cpp_extension.load_inline`, https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.load_inline) | + +***持续更新...*** diff --git a/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py b/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py index ec28cf4c5f8..16bc1308308 100644 --- a/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py +++ b/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py @@ -77,6 +77,11 @@ def unescape_api(api): def reformat_signature(code): + """ + 从函数签名代码中解析出函数名和参数列表 + - code: 函数签名代码 + - 返回值: 函数名和参数列表 + """ lines = [l for l in code.split("\n") if len(l.strip()) > 0] assert len(lines) > 0, "code have no lines." buffer = "".join([l.strip() for l in lines]) @@ -117,6 +122,11 @@ def reformat_signature(code): def get_meta_from_diff_file(filepath): + """ + 该函数从指定的映射文件中解析出元数据信息 + - filepath: 映射文件路径 + - 返回值: DiffMeta 类型的元数据信息 + """ meta_data: DiffMeta = {"source_file": filepath} state = ParserState.wait_for_title title_pattern = re.compile(r"^## +\[(?P[^\]]+)\] *(?P.+)$") @@ -393,25 +403,32 @@ def validate_mapping_table_macro_row(columns, row_idx, line_idx): if macro_match: macro_type = macro_match["macro_type"] - if macro_type not in ["REFERENCE-MAPPING-ITEM", "NOT-IMPLEMENTED-ITEM"]: + if macro_type not in [ + "REFERENCE-MAPPING-ITEM", + "NOT-IMPLEMENTED-ITEM", + "REFERENCE-MAPPING-TABLE", + ]: print(f"Unknown macro type: {macro_type} at line {line_idx}.") return False - torch_api = macro_match["torch_api"].strip("`") - diff_url = macro_match["diff_url"] + if macro_type == "REFERENCE-MAPPING-TABLE": + pass + else: + torch_api = macro_match["torch_api"].strip("`") + diff_url = macro_match["diff_url"] - if torch_api in INDEX_ALL_APIS: - raise Exception( - f"Duplicate torch api: {torch_api} at line {line_idx}." - ) - INDEX_ALL_APIS[torch_api] = columns[0] + if torch_api in INDEX_ALL_APIS: + raise Exception( + f"Duplicate torch api: {torch_api} at line {line_idx}." + ) + INDEX_ALL_APIS[torch_api] = columns[0] - return torch_api + return torch_api return False -def collect_mapping_item_processor(api_name, line_idx, state, output, context): +def collect_mapping_item_processor(_line, line_idx, state, output, context): if state == 0 or state == 1 or state == 5: return True @@ -435,6 +452,23 @@ def collect_mapping_item_processor(api_name, line_idx, state, output, context): def process_mapping_index(index_path, item_processer, context={}): + """ + 线性处理 `pytorch_api_mapping_cn.md` 文件 + - index_path: 该 md 文件路径 + - item_processer: 对文件每行的处理方式，输入参数 (line, line_idx, state, output, context)。 + 如果处理出错则返回 False，否则返回 True。 + - context: 用于存储处理过程中的上下文信息 + - output: 使用 context["output"] 初始化，如果不调用 item_processer，直接加入原文件对应行，否则 item_processer 处理 output 逻辑。 + - 返回值：是否成功处理，成功返回 0。 + + 其中 state 信息如下： + - 0: 等待表头（如果未进入表格则始终为 0） + - 1: 无需处理的表格，分隔行（表头和内容的分割线） + - 2: 需要处理的表格，分隔行 + - 5: 无需处理的表格，表格内容 + - 6: **需要处理的表格，表格内容** + + """ if not os.path.exists(index_path): raise Exception(f"Cannot find pytorch_api_mapping_cn.md: {index_path}") From 1f34e9b756646079932085c2237bdbcf4b1ace0f Mon Sep 17 00:00:00 2001 From: tianshuo78520a Date: Wed, 10 Jul 2024 18:49:08 +0800 Subject: [PATCH 08/16] Update 3.0.0b1 (#6753) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * 增加3.0.0b0版本使用conda环境import paddle报错问题的解决方案 (#6693) * Update 3.0.0b0 (#6692) * Update 3.0.0b0 * Update Tables * Fix * Fix nccl * Fix en pip * Fix * Update 3.0.0b1 * Fix --------- Co-authored-by: YUNSHEN XIE <1084314248@qq.com> --- docs/faq/install_cn.md | 14 +++ docs/install/Tables.md | 126 ++++++++++++------------- docs/install/Tables_en.md | 126 ++++++++++++------------- docs/install/conda/linux-conda.md | 6 +- docs/install/conda/linux-conda_en.md | 6 +- docs/install/conda/macos-conda.md | 2 +- docs/install/conda/macos-conda_en.md | 2 +- docs/install/conda/windows-conda.md | 6 +- docs/install/conda/windows-conda_en.md | 6 +- docs/install/docker/linux-docker.md | 44 ++++----- docs/install/docker/linux-docker_en.md | 44 ++++----- docs/install/docker/macos-docker.md | 24 ++--- docs/install/docker/macos-docker_en.md | 24 ++--- docs/install/pip/linux-pip.md | 8 +- docs/install/pip/linux-pip_en.md | 8 +- docs/install/pip/macos-pip.md | 2 +- docs/install/pip/macos-pip_en.md | 2 +- docs/install/pip/windows-pip.md | 8 +- docs/install/pip/windows-pip_en.md | 8 +- 19 files changed, 240 insertions(+), 226 deletions(-) diff --git a/docs/faq/install_cn.md b/docs/faq/install_cn.md index eabd50dae60..efea7de0b70 100644 --- a/docs/faq/install_cn.md +++ b/docs/faq/install_cn.md @@ -1,5 +1,19 @@ # 安装常见问题 +#### 问题：conda 环境下安装 paddlepaddle 3.0.0b0 版本，运行`import paddle`时报错,报错信息为找不到 libpython.so 文件 + ++ 问题描述： +> ImportError: libpython3.12.so.1.0: cannot open shared object file: No such file or directory + ++ 问题分析： +遇到该问题是因为 3.0.0b0 版本中增加了对于 libpython 的依赖，但是使用 conda 安装的 python 环境时，未把 libpython.so 文件所在路径加入到环境变量中，导致找不到该文件。 + ++ 解决办法： +例如：执行`find / -name libpython3.12.so.1.0`, 发现 libpython 的路径如`/opt/conda/lib/`，使用如下命令安装即可; + +```bash +export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/conda/lib/ +``` ##### 问题：使用过程中报找不到 tensorrt 库的日志 diff --git a/docs/install/Tables.md b/docs/install/Tables.md index d718c87280b..64c7524d2ac 100644 --- a/docs/install/Tables.md +++ b/docs/install/Tables.md @@ -290,11 +290,11 @@ PaddePaddle 通过编译时指定路径来实现引用各种 BLAS/CUDA/cuDNN 库 - paddlepaddle==[版本号] 例如 paddlepaddle==3.0.0b0 + paddlepaddle==[版本号] 例如 paddlepaddle==3.0.0b1 只支持 CPU 对应版本的 PaddlePaddle，具体版本请参见Pypi - paddlepaddle-gpu==[版本号] 例如 paddlepaddle-gpu==3.0.0b0 + paddlepaddle-gpu==[版本号] 例如 paddlepaddle-gpu==3.0.0b1 默认安装支持 CUDA 11.8 和 cuDNN 8 的对应[版本号]的 PaddlePaddle 安装包 @@ -303,7 +303,7 @@ PaddePaddle 通过编译时指定路径来实现引用各种 BLAS/CUDA/cuDNN 库 您可以在 [Release History](https://pypi.org/project/paddlepaddle-gpu/#history) 中找到 PaddlePaddle-gpu 的各个发行版本。 -需要注意的是，命令中 paddlepaddle-gpu==3.0.0b0 在 windows 环境下，会默认安装支持 CUDA 11.8 和 cuDNN 8 的对应[版本号]的 PaddlePaddle 安装包 +需要注意的是，命令中 paddlepaddle-gpu==3.0.0b1 在 windows 环境下，会默认安装支持 CUDA 11.8 和 cuDNN 8 的对应[版本号]的 PaddlePaddle 安装包

@@ -325,86 +325,86 @@ PaddePaddle 通过编译时指定路径来实现引用各种 BLAS/CUDA/cuDNN 库 cpu-mkl-avx - paddlepaddle-3.0.0b0-cp38-cp38-linux_x86_64.whl - paddlepaddle-3.0.0b0-cp39-cp39-linux_x86_64.whl - paddlepaddle-3.0.0b0-cp310-cp310-linux_x86_64.whl - paddlepaddle-3.0.0b0-cp311-cp311-linux_x86_64.whl - paddlepaddle-3.0.0b0-cp312-cp312-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp38-cp38-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp39-cp39-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp310-cp310-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp311-cp311-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp312-cp312-linux_x86_64.whl cuda11.8-cudnn8.6-mkl-gcc8.2-avx - - paddlepaddle_gpu-3.0.0b0-cp38-cp38-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp39-cp39-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp310-cp310-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp311-cp311-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp312-cp312-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp38-cp38-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp39-cp39-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp310-cp310-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp311-cp311-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp312-cp312-linux_x86_64.whl cuda12.3-cudnn9.0-mkl-gcc12.2-avx - - paddlepaddle_gpu-3.0.0b0-cp38-cp38-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp39-cp39-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp310-cp310-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp311-cp311-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp312-cp312-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp38-cp38-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp39-cp39-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp310-cp310-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp311-cp311-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp312-cp312-linux_x86_64.whl macos-cpu-x86 - - paddlepaddle-3.0.0b0-cp38-cp38-macosx_10_9_x86_64.whl - - paddlepaddle-3.0.0b0-cp39-cp39-macosx_10_9_x86_64.whl - - paddlepaddle-3.0.0b0-cp310-cp310-macosx_10_9_universal2.whl - - paddlepaddle-3.0.0b0-cp311-cp311-macosx_10_9_universal2.whl - - paddlepaddle-3.0.0b0-cp312-cp312-macosx_10_9_universal2.whl + + paddlepaddle-3.0.0b1-cp38-cp38-macosx_10_9_x86_64.whl + + paddlepaddle-3.0.0b1-cp39-cp39-macosx_10_9_x86_64.whl + + paddlepaddle-3.0.0b1-cp310-cp310-macosx_10_9_universal2.whl + + paddlepaddle-3.0.0b1-cp311-cp311-macosx_10_9_universal2.whl + + paddlepaddle-3.0.0b1-cp312-cp312-macosx_10_9_universal2.whl macos-cpu-arm - - paddlepaddle-3.0.0b0-cp38-cp38-macosx_11_0_arm64.whl - - paddlepaddle-3.0.0b0-cp39-cp39-macosx_11_0_arm64.whl - - paddlepaddle-3.0.0b0-cp310-cp310-macosx_11_0_arm64.whl - - paddlepaddle-3.0.0b0-cp311-cp311-macosx_11_0_arm64.whl - - paddlepaddle-3.0.0b0-cp312-cp312-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp38-cp38-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp39-cp39-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp310-cp310-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp311-cp311-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp312-cp312-macosx_11_0_arm64.whl win-cpu-mkl-avx - paddlepaddle-3.0.0b0-cp38-cp38-win_amd64.whl - paddlepaddle-3.0.0b0-cp39-cp39-win_amd64.whl - paddlepaddle-3.0.0b0-cp310-cp310-win_amd64.whl - paddlepaddle-3.0.0b0-cp311-cp311-win_amd64.whl - paddlepaddle-3.0.0b0-cp312-cp312-win_amd64.whl + paddlepaddle-3.0.0b1-cp38-cp38-win_amd64.whl + paddlepaddle-3.0.0b1-cp39-cp39-win_amd64.whl + paddlepaddle-3.0.0b1-cp310-cp310-win_amd64.whl + paddlepaddle-3.0.0b1-cp311-cp311-win_amd64.whl + paddlepaddle-3.0.0b1-cp312-cp312-win_amd64.whl win-cuda11.8-cudnn8.6-mkl-vs2019-avx - paddlepaddle_gpu-3.0.0b0-cp38-cp38-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp39-cp39-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp310-cp310-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp311-cp311-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp312-cp312-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp38-cp38-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp39-cp39-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp310-cp310-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp311-cp311-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp312-cp312-win_amd64.whl win-cuda12.3-cudnn9.0-mkl-vs2019-avx - paddlepaddle_gpu-3.0.0b0-cp38-cp38-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp39-cp39-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp310-cp310-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp311-cp311-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp312-cp312-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp38-cp38-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp39-cp39-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp310-cp310-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp311-cp311-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp312-cp312-win_amd64.whl diff --git a/docs/install/Tables_en.md b/docs/install/Tables_en.md index 7fc0b97d913..453e45ffeec 100644 --- a/docs/install/Tables_en.md +++ b/docs/install/Tables_en.md @@ -282,11 +282,11 @@ PaddePaddle implements references to various BLAS/CUDA/cuDNN libraries by specif - paddlepaddle==[version code] such as paddlepaddle==3.0.0b0 + paddlepaddle==[version code] such as paddlepaddle==3.0.0b1 Only support the corresponding version of the CPU PaddlePaddle, please refer to Pypi for the specific version. - paddlepaddle-gpu==[version code], such as paddlepaddle-gpu==3.0.0b0 + paddlepaddle-gpu==[version code], such as paddlepaddle-gpu==3.0.0b1 The default installation supports the PaddlePaddle installation package corresponding to [version number] of CUDA 11.2 and cuDNN 8 @@ -295,7 +295,7 @@ PaddePaddle implements references to various BLAS/CUDA/cuDNN libraries by specif You can find various distributions of PaddlePaddle-gpu in [the Release History](https://pypi.org/project/paddlepaddle-gpu/#history). -Please note that: in the commands, paddlepaddle-gpu==3.0.0b0 will install the installation package of PaddlePaddle that supports CUDA 11.2 and cuDNN 8 by default under Windows environment. +Please note that: in the commands, paddlepaddle-gpu==3.0.0b1 will install the installation package of PaddlePaddle that supports CUDA 11.2 and cuDNN 8 by default under Windows environment. @@ -319,86 +319,86 @@ Please note that: in the commands, paddlepaddle-gpu==3.0.0b0 will cpu-mkl-avx - paddlepaddle-3.0.0b0-cp38-cp38-linux_x86_64.whl - paddlepaddle-3.0.0b0-cp39-cp39-linux_x86_64.whl - paddlepaddle-3.0.0b0-cp310-cp310-linux_x86_64.whl - paddlepaddle-3.0.0b0-cp311-cp311-linux_x86_64.whl - paddlepaddle-3.0.0b0-cp312-cp312-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp38-cp38-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp39-cp39-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp310-cp310-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp311-cp311-linux_x86_64.whl + paddlepaddle-3.0.0b1-cp312-cp312-linux_x86_64.whl cuda11.8-cudnn8.6-mkl-gcc8.2-avx - - paddlepaddle_gpu-3.0.0b0-cp38-cp38-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp39-cp39-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp310-cp310-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp311-cp311-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp312-cp312-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp38-cp38-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp39-cp39-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp310-cp310-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp311-cp311-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp312-cp312-linux_x86_64.whl cuda12.3-cudnn9.0-mkl-gcc12.2-avx - - paddlepaddle_gpu-3.0.0b0-cp38-cp38-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp39-cp39-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp310-cp310-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp311-cp311-linux_x86_64.whl - - paddlepaddle_gpu-3.0.0b0-cp312-cp312-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp38-cp38-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp39-cp39-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp310-cp310-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp311-cp311-linux_x86_64.whl + + paddlepaddle_gpu-3.0.0b1-cp312-cp312-linux_x86_64.whl macos-cpu-x86 - - paddlepaddle-3.0.0b0-cp38-cp38-macosx_10_9_x86_64.whl - - paddlepaddle-3.0.0b0-cp39-cp39-macosx_10_9_x86_64.whl - - paddlepaddle-3.0.0b0-cp310-cp310-macosx_10_9_universal2.whl - - paddlepaddle-3.0.0b0-cp311-cp311-macosx_10_9_universal2.whl - - paddlepaddle-3.0.0b0-cp312-cp312-macosx_10_9_universal2.whl + + paddlepaddle-3.0.0b1-cp38-cp38-macosx_10_9_x86_64.whl + + paddlepaddle-3.0.0b1-cp39-cp39-macosx_10_9_x86_64.whl + + paddlepaddle-3.0.0b1-cp310-cp310-macosx_10_9_universal2.whl + + paddlepaddle-3.0.0b1-cp311-cp311-macosx_10_9_universal2.whl + + paddlepaddle-3.0.0b1-cp312-cp312-macosx_10_9_universal2.whl macos-cpu-arm - - paddlepaddle-3.0.0b0-cp38-cp38-macosx_11_0_arm64.whl - - paddlepaddle-3.0.0b0-cp39-cp39-macosx_11_0_arm64.whl - - paddlepaddle-3.0.0b0-cp310-cp310-macosx_11_0_arm64.whl - - paddlepaddle-3.0.0b0-cp311-cp311-macosx_11_0_arm64.whl - - paddlepaddle-3.0.0b0-cp312-cp312-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp38-cp38-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp39-cp39-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp310-cp310-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp311-cp311-macosx_11_0_arm64.whl + + paddlepaddle-3.0.0b1-cp312-cp312-macosx_11_0_arm64.whl win-cpu-mkl-avx - paddlepaddle-3.0.0b0-cp38-cp38-win_amd64.whl - paddlepaddle-3.0.0b0-cp39-cp39-win_amd64.whl - paddlepaddle-3.0.0b0-cp310-cp310-win_amd64.whl - paddlepaddle-3.0.0b0-cp311-cp311-win_amd64.whl - paddlepaddle-3.0.0b0-cp312-cp312-win_amd64.whl + paddlepaddle-3.0.0b1-cp38-cp38-win_amd64.whl + paddlepaddle-3.0.0b1-cp39-cp39-win_amd64.whl + paddlepaddle-3.0.0b1-cp310-cp310-win_amd64.whl + paddlepaddle-3.0.0b1-cp311-cp311-win_amd64.whl + paddlepaddle-3.0.0b1-cp312-cp312-win_amd64.whl win-cuda11.8-cudnn8.6-mkl-vs2019-avx - paddlepaddle_gpu-3.0.0b0-cp38-cp38-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp39-cp39-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp310-cp310-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp311-cp311-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp312-cp312-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp38-cp38-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp39-cp39-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp310-cp310-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp311-cp311-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp312-cp312-win_amd64.whl win-cuda12.3-cudnn9.0-mkl-vs2019-avx - paddlepaddle_gpu-3.0.0b0-cp38-cp38-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp39-cp39-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp310-cp310-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp311-cp311-win_amd64.whl - paddlepaddle_gpu-3.0.0b0-cp312-cp312-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp38-cp38-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp39-cp39-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp310-cp310-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp311-cp311-win_amd64.whl + paddlepaddle_gpu-3.0.0b1-cp312-cp312-win_amd64.whl diff --git a/docs/install/conda/linux-conda.md b/docs/install/conda/linux-conda.md index 7fb313d5af1..5b97ec80da0 100644 --- a/docs/install/conda/linux-conda.md +++ b/docs/install/conda/linux-conda.md @@ -90,7 +90,7 @@ python3 -c "import platform;print(platform.architecture()[0]);print(platform.mac 如果您的计算机没有 NVIDIA® GPU，请安装 CPU 版的 PaddlePaddle ``` -conda install paddlepaddle==3.0.0b0 -c paddle +conda install paddlepaddle==3.0.0b1 -c paddle ``` @@ -100,13 +100,13 @@ conda install paddlepaddle==3.0.0b0 -c paddle * 对于 `CUDA 11.8` 安装命令为: ``` - conda install paddlepaddle-gpu==3.0.0b0 paddlepaddle-cuda=11.8 -c paddle -c nvidia + conda install paddlepaddle-gpu==3.0.0b1 paddlepaddle-cuda=11.8 -c paddle -c nvidia ``` * 对于 `CUDA 12.3` 安装命令为: ``` - conda install paddlepaddle-gpu==3.0.0b0 paddlepaddle-cuda=12.3 -c paddle -c nvidia + conda install paddlepaddle-gpu==3.0.0b1 paddlepaddle-cuda=12.3 -c paddle -c nvidia ``` diff --git a/docs/install/conda/linux-conda_en.md b/docs/install/conda/linux-conda_en.md index 4cf7ca92fa1..6a2d3d1d8e7 100644 --- a/docs/install/conda/linux-conda_en.md +++ b/docs/install/conda/linux-conda_en.md @@ -95,7 +95,7 @@ You can choose the following version of PaddlePaddle to start installation: If your computer doesn't have NVIDIA® GPU, please install `the CPU Version of PaddlePaddle` ``` -conda install paddlepaddle==3.0.0b0 -c paddle +conda install paddlepaddle==3.0.0b1 -c paddle ``` @@ -105,13 +105,13 @@ conda install paddlepaddle==3.0.0b0 -c paddle * If you are using CUDA 11.8: ``` - conda install paddlepaddle-gpu==3.0.0b0 paddlepaddle-cuda=11.8 -c paddle -c nvidia + conda install paddlepaddle-gpu==3.0.0b1 paddlepaddle-cuda=11.8 -c paddle -c nvidia ``` * If you are using CUDA 12.3: ``` - conda install paddlepaddle-gpu==3.0.0b0 paddlepaddle-cuda=12.3 -c paddle -c nvidia + conda install paddlepaddle-gpu==3.0.0b1 paddlepaddle-cuda=12.3 -c paddle -c nvidia ``` diff --git a/docs/install/conda/macos-conda.md b/docs/install/conda/macos-conda.md index 22760292664..4adb5843edc 100644 --- a/docs/install/conda/macos-conda.md +++ b/docs/install/conda/macos-conda.md @@ -83,7 +83,7 @@ python3 -c "import platform;print(platform.architecture()[0]);print(platform.mac * 目前在 macOS 环境仅支持 CPU 版 PaddlePaddle，请参考如下命令安装 Paddle: ``` - conda install paddlepaddle==3.0.0b0 -c paddle + conda install paddlepaddle==3.0.0b1 -c paddle ``` ## **三、验证安装** diff --git a/docs/install/conda/macos-conda_en.md b/docs/install/conda/macos-conda_en.md index 4af9bafb88a..d938ea6408b 100644 --- a/docs/install/conda/macos-conda_en.md +++ b/docs/install/conda/macos-conda_en.md @@ -87,7 +87,7 @@ conda config --set show_channel_urls yes * Currently, only the CPU version of PaddlePaddle is supported in the macOS environment. Please use the following command to install PaddlePaddle： ``` - conda install paddlepaddle==3.0.0b0 -c paddle + conda install paddlepaddle==3.0.0b1 -c paddle ``` diff --git a/docs/install/conda/windows-conda.md b/docs/install/conda/windows-conda.md index 045a34169ff..8f165027654 100644 --- a/docs/install/conda/windows-conda.md +++ b/docs/install/conda/windows-conda.md @@ -92,7 +92,7 @@ python -c "import platform;print(platform.architecture()[0]);print(platform.mach ``` -conda install paddlepaddle==3.0.0b0 -c paddle +conda install paddlepaddle==3.0.0b1 -c paddle ``` @@ -102,13 +102,13 @@ conda install paddlepaddle==3.0.0b0 -c paddle * 对于 `CUDA 11.8` 安装命令为: ``` - conda install paddlepaddle-gpu==3.0.0b0 paddlepaddle-cuda=11.8 -c paddle -c nvidia + conda install paddlepaddle-gpu==3.0.0b1 paddlepaddle-cuda=11.8 -c paddle -c nvidia ``` * 对于 `CUDA 12.3` 安装命令为: ``` - conda install paddlepaddle-gpu==3.0.0b0 paddlepaddle-cuda=12.3 -c paddle -c nvidia + conda install paddlepaddle-gpu==3.0.0b1 paddlepaddle-cuda=12.3 -c paddle -c nvidia ``` diff --git a/docs/install/conda/windows-conda_en.md b/docs/install/conda/windows-conda_en.md index af0cd6af23e..f3f0a2b2df3 100644 --- a/docs/install/conda/windows-conda_en.md +++ b/docs/install/conda/windows-conda_en.md @@ -97,7 +97,7 @@ You can choose the following version of PaddlePaddle to start installation: If your computer doesn't have NVIDIA® GPU, please install `the CPU Version of PaddlePaddle` ``` -conda install paddlepaddle==3.0.0b0 -c paddle +conda install paddlepaddle==3.0.0b1 -c paddle ``` @@ -109,13 +109,13 @@ conda install paddlepaddle==3.0.0b0 -c paddle * If you are using CUDA 11.8: ``` - conda install paddlepaddle-gpu==3.0.0b0 paddlepaddle-cuda=11.8 -c paddle -c nvidia + conda install paddlepaddle-gpu==3.0.0b1 paddlepaddle-cuda=11.8 -c paddle -c nvidia ``` * If you are using CUDA 12.3: ``` - conda install paddlepaddle-gpu==3.0.0b0 paddlepaddle-cuda=12.3 -c paddle -c nvidia + conda install paddlepaddle-gpu==3.0.0b1 paddlepaddle-cuda=12.3 -c paddle -c nvidia ``` diff --git a/docs/install/docker/linux-docker.md b/docs/install/docker/linux-docker.md index e79cd188fa3..defc14b2f51 100644 --- a/docs/install/docker/linux-docker.md +++ b/docs/install/docker/linux-docker.md @@ -21,40 +21,40 @@ * CPU 版的 PaddlePaddle： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 ``` * CPU 版的 PaddlePaddle，且镜像中预装好了 jupyter： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter ``` * GPU 版的 PaddlePaddle(**建议拉取最新版本镜像，并确保已经成功安装 NVIDIA Container Toolkit**)： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda11.8-cudnn8.6-trt8.5 + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda11.8-cudnn8.6-trt8.5 ``` ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6 + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6 ``` 如果您的机器不在中国大陆地区，可以直接从 DockerHub 拉取镜像： * CPU 版的 PaddlePaddle： ``` - docker pull paddlepaddle/paddle:3.0.0b0 + docker pull paddlepaddle/paddle:3.0.0b1 ``` * CPU 版的 PaddlePaddle，且镜像中预装好了 jupyter： ``` - docker pull paddlepaddle/paddle:3.0.0b0-jupyter + docker pull paddlepaddle/paddle:3.0.0b1-jupyter ``` * GPU 版的 PaddlePaddle(**建议拉取最新版本镜像，并确保已经成功安装 NVIDIA Container Toolkit**)： ``` - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda11.8-cudnn8.6-trt8.5 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda11.8-cudnn8.6-trt8.5 ``` ``` - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6 ``` 您还可以访问[DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/)获取更多镜像。 @@ -66,7 +66,7 @@ ``` - docker run --name paddle_docker -it -v $PWD:/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 /bin/bash + docker run --name paddle_docker -it -v $PWD:/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 /bin/bash ``` - `--name paddle_docker`：设定 Docker 的名称，`paddle_docker` 是自己设置的名称； @@ -77,7 +77,7 @@ - `-v $PWD:/paddle`：指定将当前路径（PWD 变量会展开为当前路径的绝对路径）挂载到容器内部的 /paddle 目录； - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0`：指定需要使用的 image 名称，您可以通过`docker images`命令查看；/bin/bash 是在 Docker 中要执行的命令 + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1`：指定需要使用的 image 名称，您可以通过`docker images`命令查看；/bin/bash 是在 Docker 中要执行的命令 * 使用 CPU 版本的 PaddlePaddle，且镜像中预装好了 jupyter： @@ -92,7 +92,7 @@ cd ./jupyter_docker ``` ``` - docker run -p 80:80 --rm --env USER_PASSWD="password you set" -v $PWD:/home/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter + docker run -p 80:80 --rm --env USER_PASSWD="password you set" -v $PWD:/home/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter ``` - `--rm`：关闭容器后删除容器； @@ -103,13 +103,13 @@ - `-v $PWD:/home/paddle`：指定将当前路径（PWD 变量会展开为当前路径的绝对路径）挂载到容器内部的 /home/paddle 目录； - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter`：指定需要使用的 image 名称，您可以通过`docker images`命令查看 + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter`：指定需要使用的 image 名称，您可以通过`docker images`命令查看 * 使用 GPU 版本的 PaddlePaddle： ``` - docker run --gpus all --name paddle_docker -v $PWD:/paddle --network=host -it registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6 /bin/bash + docker run --gpus all --name paddle_docker -v $PWD:/paddle --network=host -it registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6 /bin/bash ``` - `--gpus all`: 在 Docker 容器中允许使用 gpu; @@ -121,7 +121,7 @@ - `-it`： 与宿主机保持交互状态; - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6`：使用名为`registry.baidubce.com/paddlepaddle/paddle`, tag 为`3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6`的镜像创建 Docker 容器，/bin/bash 进入容器后启动/bin/bash 命令。 + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6`：使用名为`registry.baidubce.com/paddlepaddle/paddle`, tag 为`3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6`的镜像创建 Docker 容器，/bin/bash 进入容器后启动/bin/bash 命令。 @@ -140,20 +140,20 @@ - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 - 安装了 3.0.0b0 版本 paddle 的 CPU 镜像 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 + 安装了 3.0.0b1 版本 paddle 的 CPU 镜像 - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter - 安装了 3.0.0b0 版本 paddle 的 CPU 镜像，且镜像中预装好了 jupyter，启动 docker 即运行 jupyter 服务 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter + 安装了 3.0.0b1 版本 paddle 的 CPU 镜像，且镜像中预装好了 jupyter，启动 docker 即运行 jupyter 服务 - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda11.8-cudnn8.6-trt8.5 - 安装了 3.0.0b0 版本 paddle 的 GPU 镜像，cuda 版本为 11.8，cudnn 版本为 8.6，trt 版本为 8.5 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda11.8-cudnn8.6-trt8.5 + 安装了 3.0.0b1 版本 paddle 的 GPU 镜像，cuda 版本为 11.8，cudnn 版本为 8.6，trt 版本为 8.5 - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6 - 安装了 3.0.0b0 版本 paddle 的 GPU 镜像，cuda 版本为 12.3，cudnn 版本为 9.0，trt 版本为 8.6 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6 + 安装了 3.0.0b1 版本 paddle 的 GPU 镜像，cuda 版本为 12.3，cudnn 版本为 9.0，trt 版本为 8.6 diff --git a/docs/install/docker/linux-docker_en.md b/docs/install/docker/linux-docker_en.md index 9e6220a91a9..30b83ee6e8f 100644 --- a/docs/install/docker/linux-docker_en.md +++ b/docs/install/docker/linux-docker_en.md @@ -21,40 +21,40 @@ For domestic users, when downloading docker is slow due to network problems, you * CPU version of PaddlePaddle： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 ``` * CPU version of PaddlePaddle, and the image is pre-installed with jupyter： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter ``` * GPU version of PaddlePaddle(**Latest version of gpu image is recommended， and make sure NVIDIA Container Toolkit is installed successfully**)： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6 + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6 ``` ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda11.8-cudnn8.6-trt8.5 + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda11.8-cudnn8.6-trt8.5 ``` If your machine is not in mainland China, you can pull the image directly from DockerHub: * CPU version of PaddlePaddle： ``` - docker pull paddlepaddle/paddle:3.0.0b0 + docker pull paddlepaddle/paddle:3.0.0b1 ``` * CPU version of PaddlePaddle, and the image is pre-installed with jupyter： ``` - docker pull paddlepaddle/paddle:3.0.0b0-jupyter + docker pull paddlepaddle/paddle:3.0.0b1-jupyter ``` * GPU version of PaddlePaddle(**Latest version of gpu image is recommended， and make sure NVIDIA Container Toolkit is installed successfully**)： ``` - docker pull paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6 + docker pull paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6 ``` ``` - docker pull paddlepaddle/paddle:3.0.0b0-gpu-cuda11.8-cudnn8.6-trt8.5 + docker pull paddlepaddle/paddle:3.0.0b1-gpu-cuda11.8-cudnn8.6-trt8.5 ``` You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to get more images. @@ -66,7 +66,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g ``` - docker run --name paddle_docker -it -v $PWD:/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 /bin/bash + docker run --name paddle_docker -it -v $PWD:/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 /bin/bash ``` - `--name paddle_docker`: set name of Docker, `paddle_docker` is name of docker you set; @@ -77,7 +77,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g - `-v $PWD:/paddle`: Specifies to mount the current path of the host (PWD variable in Linux will expand to the absolute path of the current path) to the /paddle directory inside the container; - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0`: Specify the name of the image to be used. You can view it through the 'docker images' command. /bin/Bash is the command to be executed in Docker + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1`: Specify the name of the image to be used. You can view it through the 'docker images' command. /bin/Bash is the command to be executed in Docker * Use GPU version of PaddlePaddle： @@ -85,7 +85,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g ``` - docker run --gpus all --name paddle_docker -v $PWD:/paddle --network=host -it registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6 /bin/bash + docker run --gpus all --name paddle_docker -v $PWD:/paddle --network=host -it registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6 /bin/bash ``` - `--gpus all`: gpu resources can be used in Docker container; @@ -98,7 +98,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g - `-v $PWD:/paddle`: Specifies to mount the current path of the host (PWD variable in Linux will expand to the absolute path of the current path) to the /paddle directory inside the container; - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6`: Specify the name of the image to be used. You can view it through the 'docker images' command. /bin/Bash is the command to be executed in Docker + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6`: Specify the name of the image to be used. You can view it through the 'docker images' command. /bin/Bash is the command to be executed in Docker * Use CPU version of PaddlePaddle with jupyter： @@ -114,7 +114,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g cd ./jupyter_docker ``` ``` - docker run -p 80:80 --rm --env USER_PASSWD="password you set" -v $PWD:/home/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter + docker run -p 80:80 --rm --env USER_PASSWD="password you set" -v $PWD:/home/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter ``` - `--rm`: Delete the container after closing it; @@ -125,7 +125,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g - `-v $PWD:/home/paddle`: Specifies to mount the current path (the PWD variable will be expanded to the absolute path of the current path) to the /home/paddle directory inside the container; - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter`: Specify the name of the image to be used, you can view it through the `docker images` command + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter`: Specify the name of the image to be used, you can view it through the `docker images` command Now you have successfully used Docker to install PaddlePaddle. For more information about using Docker, see[Docker official documents](https://docs.docker.com) @@ -143,20 +143,20 @@ Now you have successfully used Docker to install PaddlePaddle. For more informat - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 - CPU image with 3.0.0b0 version of paddle installed + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 + CPU image with 3.0.0b1 version of paddle installed - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter - CPU image of paddle version 3.0.0b0 is installed, and jupyter is pre-installed in the image. Start the docker to run the jupyter service + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter + CPU image of paddle version 3.0.0b1 is installed, and jupyter is pre-installed in the image. Start the docker to run the jupyter service - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda12.3-cudnn9.0-trt8.6 - GPU image of paddle version 3.0.0b0 is installed, cuda version is 12.3, cudnn version is 9.0, trt version is 8.6 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda12.3-cudnn9.0-trt8.6 + GPU image of paddle version 3.0.0b1 is installed, cuda version is 12.3, cudnn version is 9.0, trt version is 8.6 - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-gpu-cuda11.8-cudnn8.6-trt8.5 - GPU image of paddle version 3.0.0b0 is installed, cuda version is 11.8, cudnn version is 8.6, trt version is 8.5 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-gpu-cuda11.8-cudnn8.6-trt8.5 + GPU image of paddle version 3.0.0b1 is installed, cuda version is 11.8, cudnn version is 8.6, trt version is 8.5 diff --git a/docs/install/docker/macos-docker.md b/docs/install/docker/macos-docker.md index 8b546ebb2f6..d5590b4c859 100644 --- a/docs/install/docker/macos-docker.md +++ b/docs/install/docker/macos-docker.md @@ -19,24 +19,24 @@ * CPU 版的 PaddlePaddle： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 ``` * CPU 版的 PaddlePaddle，且镜像中预装好了 jupyter： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter ``` 如果您的机器不在中国大陆地区，可以直接从 DockerHub 拉取镜像： * CPU 版的 PaddlePaddle： ``` - docker pull paddlepaddle/paddle:3.0.0b0 + docker pull paddlepaddle/paddle:3.0.0b1 ``` * CPU 版的 PaddlePaddle，且镜像中预装好了 jupyter： ``` - docker pull paddlepaddle/paddle:3.0.0b0-jupyter + docker pull paddlepaddle/paddle:3.0.0b1-jupyter ``` 您还可以访问[DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/)获取更多镜像。 @@ -48,7 +48,7 @@ ``` - docker run --name paddle_docker -it -v $PWD:/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 /bin/bash + docker run --name paddle_docker -it -v $PWD:/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 /bin/bash ``` - `--name paddle_docker`：设定 Docker 的名称，`paddle_docker` 是自己设置的名称； @@ -59,7 +59,7 @@ - `-v $PWD:/paddle`：指定将当前路径（PWD 变量会展开为当前路径的绝对路径）挂载到容器内部的 /paddle 目录； - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0`：指定需要使用的 image 名称，您可以通过`docker images`命令查看；/bin/bash 是在 Docker 中要执行的命令 + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1`：指定需要使用的 image 名称，您可以通过`docker images`命令查看；/bin/bash 是在 Docker 中要执行的命令 * 使用 CPU 版本的 PaddlePaddle，且镜像中预装好了 jupyter： @@ -73,7 +73,7 @@ cd ./jupyter_docker ``` ``` - docker run -p 80:80 --rm --env USER_PASSWD="password you set" -v $PWD:/home/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter + docker run -p 80:80 --rm --env USER_PASSWD="password you set" -v $PWD:/home/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter ``` - `--rm`：关闭容器后删除容器； @@ -84,7 +84,7 @@ - `-v $PWD:/home/paddle`：指定将当前路径（PWD 变量会展开为当前路径的绝对路径）挂载到容器内部的 /home/paddle 目录； - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter`：指定需要使用的 image 名称，您可以通过`docker images`命令查看 + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter`：指定需要使用的 image 名称，您可以通过`docker images`命令查看 @@ -104,12 +104,12 @@ - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 - 安装了 3.0.0b0 版本 paddle 的 CPU 镜像 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 + 安装了 3.0.0b1 版本 paddle 的 CPU 镜像 - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter - 安装了 3.0.0b0 版本 paddle 的 CPU 镜像，且镜像中预装好了 jupyter，启动 docker 即运行 jupyter 服务 + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter + 安装了 3.0.0b1 版本 paddle 的 CPU 镜像，且镜像中预装好了 jupyter，启动 docker 即运行 jupyter 服务 diff --git a/docs/install/docker/macos-docker_en.md b/docs/install/docker/macos-docker_en.md index 1f70c8da735..d8d2c4bcd77 100644 --- a/docs/install/docker/macos-docker_en.md +++ b/docs/install/docker/macos-docker_en.md @@ -19,24 +19,24 @@ For domestic users, when downloading docker is slow due to network problems, you * CPU version of PaddlePaddle： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 ``` * CPU version of PaddlePaddle, and the image is pre-installed with jupyter： ``` - docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter + docker pull registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter ``` If your machine is not in mainland China, you can pull the image directly from DockerHub: * CPU version of PaddlePaddle： ``` - docker pull paddlepaddle/paddle:3.0.0b0 + docker pull paddlepaddle/paddle:3.0.0b1 ``` * CPU version of PaddlePaddle, and the image is pre-installed with jupyter： ``` - docker pull paddlepaddle/paddle:3.0.0b0-jupyter + docker pull paddlepaddle/paddle:3.0.0b1-jupyter ``` You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to get more images. @@ -48,7 +48,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g ``` - docker run --name paddle_docker -it -v $PWD:/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 /bin/bash + docker run --name paddle_docker -it -v $PWD:/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 /bin/bash ``` - `--name paddle_docker`: set name of Docker, `paddle_docker` is name of docker you set; @@ -59,7 +59,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g - `-v $PWD:/paddle`: Specifies to mount the current path of the host (PWD variable in Linux will expand to the absolute path of the current path) to the /paddle directory inside the container; - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0`: Specify the name of the image to be used. You can view it through the 'docker images' command. /bin/Bash is the command to be executed in Docker + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1`: Specify the name of the image to be used. You can view it through the 'docker images' command. /bin/Bash is the command to be executed in Docker * Use CPU version of PaddlePaddle with jupyter： @@ -75,7 +75,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g cd ./jupyter_docker ``` ``` - docker run -p 80:80 --rm --env USER_PASSWD="password you set" -v $PWD:/home/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter + docker run -p 80:80 --rm --env USER_PASSWD="password you set" -v $PWD:/home/paddle registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter ``` - `--rm`: Delete the container after closing it; @@ -86,7 +86,7 @@ You can see [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to g - `-v $PWD:/home/paddle`: Specifies to mount the current path (the PWD variable will be expanded to the absolute path of the current path) to the /home/paddle directory inside the container; - - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter`: Specify the name of the image to be used, you can view it through the `docker images` command + - `registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter`: Specify the name of the image to be used, you can view it through the `docker images` command @@ -105,12 +105,12 @@ Now you have successfully used Docker to install PaddlePaddle. For more informat - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0 - CPU image with 3.0.0b0 version of paddle installed + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1 + CPU image with 3.0.0b1 version of paddle installed - registry.baidubce.com/paddlepaddle/paddle:3.0.0b0-jupyter - CPU image of paddle version 3.0.0b0 is installed, and jupyter is pre-installed in the image. Start the docker to run the jupyter service + registry.baidubce.com/paddlepaddle/paddle:3.0.0b1-jupyter + CPU image of paddle version 3.0.0b1 is installed, and jupyter is pre-installed in the image. Start the docker to run the jupyter service diff --git a/docs/install/pip/linux-pip.md b/docs/install/pip/linux-pip.md index 362e40faa6c..9ab6ffd0b09 100644 --- a/docs/install/pip/linux-pip.md +++ b/docs/install/pip/linux-pip.md @@ -78,7 +78,7 @@ ``` - python3 -m pip install paddlepaddle==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ + python3 -m pip install paddlepaddle==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ ``` @@ -89,7 +89,7 @@ ``` - python3 -m pip install paddlepaddle-gpu==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/ + python3 -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/ ``` @@ -97,7 +97,7 @@ ``` - python3 -m pip install paddlepaddle-gpu==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/ + python3 -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/ ``` @@ -113,7 +113,7 @@ * 如果你想安装`avx`、`openblas`的 Paddle 包，可以通过以下命令将 wheel 包下载到本地，再使用`python3 -m pip install [name].whl`本地安装（[name]为 wheel 包名称）： ``` - python3 -m pip install https://paddle-wheel.bj.bcebos.com/3.0.0-beta0/linux/linux-cpu-openblas-avx/paddlepaddle-3.0.0b0-cp38-cp38-linux_x86_64.whl + python3 -m pip install https://paddle-wheel.bj.bcebos.com/3.0.0-beta0/linux/linux-cpu-openblas-avx/paddlepaddle-3.0.0b1-cp38-cp38-linux_x86_64.whl ``` diff --git a/docs/install/pip/linux-pip_en.md b/docs/install/pip/linux-pip_en.md index 76cf858ecc9..a59c7ef1d02 100644 --- a/docs/install/pip/linux-pip_en.md +++ b/docs/install/pip/linux-pip_en.md @@ -83,7 +83,7 @@ You can choose the following version of PaddlePaddle to start installation: ``` - python3 -m pip install paddlepaddle==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ + python3 -m pip install paddlepaddle==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ ``` @@ -94,7 +94,7 @@ You can choose the following version of PaddlePaddle to start installation: ``` - python3 -m pip install paddlepaddle-gpu==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/ + python3 -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/ ``` @@ -102,7 +102,7 @@ You can choose the following version of PaddlePaddle to start installation: ``` - python3 -m pip install paddlepaddle-gpu==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/ + python3 -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/ ``` @@ -118,7 +118,7 @@ Note： * If you want to install the Paddle package with `avx` and `openblas`, you can use the following command to download the wheel package to the local, and then use `python3 -m pip install [name].whl` to install locally ([name] is the name of the wheel package): ``` - python3 -m pip install https://paddle-wheel.bj.bcebos.com/3.0.0-beta0/linux/linux-cpu-openblas-avx/paddlepaddle-3.0.0b0-cp38-cp38-linux_x86_64.whl + python3 -m pip install https://paddle-wheel.bj.bcebos.com/3.0.0-beta0/linux/linux-cpu-openblas-avx/paddlepaddle-3.0.0b1-cp38-cp38-linux_x86_64.whl ``` diff --git a/docs/install/pip/macos-pip.md b/docs/install/pip/macos-pip.md index 5e4f3967235..ab472f326f8 100644 --- a/docs/install/pip/macos-pip.md +++ b/docs/install/pip/macos-pip.md @@ -70,7 +70,7 @@ ``` - python3 -m pip install paddlepaddle==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ + python3 -m pip install paddlepaddle==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ ``` diff --git a/docs/install/pip/macos-pip_en.md b/docs/install/pip/macos-pip_en.md index f24784a2e49..230cf22ba63 100644 --- a/docs/install/pip/macos-pip_en.md +++ b/docs/install/pip/macos-pip_en.md @@ -66,7 +66,7 @@ You can choose the following version of PaddlePaddle to start installation: ``` -python3 -m pip install paddlepaddle==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ +python3 -m pip install paddlepaddle==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ ``` Note： diff --git a/docs/install/pip/windows-pip.md b/docs/install/pip/windows-pip.md index 47d86103306..ebb007b0109 100644 --- a/docs/install/pip/windows-pip.md +++ b/docs/install/pip/windows-pip.md @@ -65,7 +65,7 @@ ``` - python -m pip install paddlepaddle==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ + python -m pip install paddlepaddle==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ ``` @@ -75,13 +75,13 @@ 2.2.4 CUDA11.8 的 PaddlePaddle ``` - python -m pip install paddlepaddle-gpu==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/ + python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/ ``` 2.2.5 CUDA12.3 的 PaddlePaddle ``` - python -m pip install paddlepaddle-gpu==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/ + python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/ ``` 注： @@ -93,7 +93,7 @@ * 如果你想安装`avx`、`openblas`的 Paddle 包，可以通过以下命令将 wheel 包下载到本地，再使用`python -m pip install [name].whl`本地安装（[name]为 wheel 包名称）： ``` - python -m pip install https://paddle-wheel.bj.bcebos.com/3.0.0-beta0/windows/windows-cpu-avx-openblas-vs2017/paddlepaddle-3.0.0b0-cp38-cp38-win_amd64.whl + python -m pip install https://paddle-wheel.bj.bcebos.com/3.0.0-beta0/windows/windows-cpu-avx-openblas-vs2017/paddlepaddle-3.0.0b1-cp38-cp38-win_amd64.whl ``` diff --git a/docs/install/pip/windows-pip_en.md b/docs/install/pip/windows-pip_en.md index 33cb62d33bd..1bf2e8843fe 100644 --- a/docs/install/pip/windows-pip_en.md +++ b/docs/install/pip/windows-pip_en.md @@ -62,7 +62,7 @@ You can choose the following version of PaddlePaddle to start installation: ``` - python -m pip install paddlepaddle==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ + python -m pip install paddlepaddle==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/ ``` @@ -72,13 +72,13 @@ You can choose the following version of PaddlePaddle to start installation: 2.2.4 If you are using CUDA 11.8 ``` - python -m pip install paddlepaddle-gpu==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/ + python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/ ``` 2.2.5 If you are using CUDA 12.3 ``` - python -m pip install paddlepaddle-gpu==3.0.0b0 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/ + python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/ ``` Note： @@ -91,7 +91,7 @@ Note： * If you want to install the Paddle package with `avx` and `openblas`, you can use the following command to download the wheel package to the local, and then use `python -m pip install [name].whl` to install locally ([name] is the name of the wheel package): ``` - python -m pip install https://paddle-wheel.bj.bcebos.com/3.0.0-beta0/windows/windows-cpu-avx-openblas-vs2017/paddlepaddle-3.0.0b0-cp38-cp38-win_amd64.whl + python -m pip install https://paddle-wheel.bj.bcebos.com/3.0.0-beta0/windows/windows-cpu-avx-openblas-vs2017/paddlepaddle-3.0.0b1-cp38-cp38-win_amd64.whl ``` ## Verify installation From 832b355573b8443fb4c187af264029f57d28ffcf Mon Sep 17 00:00:00 2001 From: zachary sun <70642955+sunzhongkai588@users.noreply.github.com> Date: Thu, 11 Jul 2024 11:06:12 +0800 Subject: [PATCH 09/16] fix (#6756) --- docs/release_note_en.md | 3804 +++++---------------------------------- 1 file changed, 423 insertions(+), 3381 deletions(-) diff --git a/docs/release_note_en.md b/docs/release_note_en.md index 3e6ea0cd50f..0052edd54df 100644 --- a/docs/release_note_en.md +++ b/docs/release_note_en.md @@ -1,3495 +1,537 @@ -# 2.6.0 Release Note +# 3.0 Beta Release Note -## 1. Important Updates +# Overview of PaddlePaddle 3.0 Beta -- **Paddle New generation IR(PIR)** : In order to further improve scalability of the PaddlePaddle framework, we have developed a new generation intermediate representaion. It abstracts underlying core concepts of the PaddlePaddle framework, such as Operation, Attribute and Type, providing developers with flexible and efficient basic components. By introducing Dialect mechanism, PIR can comprehensively and hierarchically satisfy needs of each module for intermediate representations to greatly enhancing scalability of the framework. PIR strictly follows Static Single Assignment (SSA) principle, ensuring unity of top-level structure and harmonious coexistence of "operator sequentiality" and "computational graph semantics". In addition, PIR provides a more concise and low-cost Pass development process, with a series of built-in rich and functional Pass optimization strategies. It provides technical support for the ultimate performance optimization of large-scale models. -- **Static graph construction and compiler Optimization Architecture**: In order to further improve performance of the framework, PaddlePaddle's dynamic to static training capability has been comprehensively upgraded to support adaptive graph construction capability. This has been tested on more than 700 PaddlePaddle industry-level models, with 100% success rate of one line code converter to start static training. Meanwhile, Compiler Infrastructure for Neural Networks (CINN) of PaddlePaddle framework is integrated into PaddlePaddle main Repo, making the compiler and PaddlePaddle more integrated. CINN completes architectural optimization and improvement of expansion capability, increasing system stability. Based on PIR framework, it is much more easied to bind dynamic to static, primitive operator, executor and compiler together, to providing more space for boosting overall performance of PaddlePaddle framework. -- **Enhanced dynamic graph distributed capability**: Large models pose higher demands on the distributed training performance of framework. PaddlePaddle has comprehensive optimizations in dimensions of communication library, graph analysis, distributed strategy and task enable/disable, enhancing distributed computing capability of PaddlePaddle's dynamic graph and providing support for efficient training of large models. In terms of performance, training performance is further improved by reducing pipelined GPU memory occupation, adopting TensorFusion technology, implementing communication computation overlap, and reducing non-essential data synchronization copies. Meanwhile, flexibility of hybrid-parallel debugging is improved through environment variable control Optimizer. In addition, stability of system is significantly improved by fixing related Bugs. -- **Auto parallel architecture with dynamic-static unification**: In order to further reduce difficulty of programming and optimizing large models, PaddlePaddle has fully optimized the Semi-Auto Parallel programming paradigm with dynamic-static unification, simplifying programming complexity for developers. Developers do not need to deeply understand complex concepts and APIs under the manual parallel programming paradigm, such as row-parallel, and column-parallel. They only need a small amount of tensor distribution annotations to implement the hybrid parallelism. The distribution specification will be propagated to all tensors and operators automatically, and the framework would handle the communication and synchronization needed by distributed training appropriately. Meanwhile, it supports dynamic-to-static distributed training by adding one extra code only, allowing developers to efficiently implement any mixed parallelism strategy and deeply simplify the development process of hybrid-parallel training paradigm. -- **Hardware Integration Solution (CustomDevice)**: With increased demand for parallel training on new hardware in large model scenarios, PaddlePaddle has added support for distributed advanced policies, custom operators, and custom fusion policies. Distributed communication library is upgraded, with newly added support for many advanced distributed policies such as MP, GroupShared, PP, SP and MOE. Moreover, it supports vendors to flexibly access Transformer operator libraries of different granularities and modify the computation graph through Fusion Pass for performance acceleration. -- **Installation and development experience**: use of modular compilation optimizes logics of CMake codes, and improves efficiency of PaddlePaddle full compilation and incremental compilation. In addition, this can increase efficiency of RD development. It supports Python3.12, CUDA12, Hopper architecture compilation, with introduction of Clang and other tools to fully optimize code formats. In addition, C++ is changed from linking static libraries to linking dynamic libraries to reduce compilation volume. These optimizations provide users with a smoother and more efficient installation and development experience. +The core features of this version mainly include new technologies such as dynamic-static unity auto parallel and automatic optimization of neural network compiler, to aim to address the new challenges in the current deep learning field.PaddlePaddle Framework 3.0 Beta extends the design concepts of 2.x such as dynamic-static unity and integrated training and inference. The development interface is fully compatible with 2.x version. This means that codes developed in version 2.x can run directly on version 3.x without modification in most cases. Several key features are detailed as follows: -## 2. Incompatible Upgrade +- Dynamic-static graph unified auto parallel: To make the parallel training programming of large models easier, PaddlePaddle has also optimized the semi-auto parallel programming paradigm with dynamic-static graph unified. Developers do not need to delve into the complex concepts and APIs need in manual parallel programming; developers only need to perform a small amount of tensor sharding annotation to complete the construction of hybrid parallelism for large models. The framework is able to automatically derive distributed sharding states and add communication operators, and also supports one-key dynamic-to-static distributed training, thus dramatically simplifying the development of hybrid parallel training codes. In terms of dynamic-static unity, PaddlePaddle has comprehensively upgraded its dynamic-to-static training capability by adopting bytecode-based dynamic-static conversion technology, to support adaptive graph construction functions. It has been verified on more than 700 PaddlePaddle industrial-grade models, achieving a 100% success rate of one-key dynamic-to-static training. +- Automatic optimization of neural network compiler: PaddlePaddle Compiler Infrastructure for Neural Networks (CINN) adopts the design of integration with the framework, supporting the efficient training and dynamic shape inference of generative models, scientific computing models and other models. This provides a good balance between computational flexibility and high performance. The inference performance of Llama2 and Stable Diffusion models has been improved by 30% through automatic fusion of operators and code generation technology. +- High-order automatic differentiation: In order to better support scientific computing scenarios, PaddlePaddle Framework designs and implements high-order automatic differentiation technology based on combinatorial operator mechanism, combined with automatic optimization technology of neural network compiler. We have tested more than 40 differential equations in scientific computing scenarios, and its solution speed is 70% ahead of similar products in the industry. +- Highly scalable intermediate representation: In order to improve the scalability of the PaddlePaddle framework, we have developed a highly scalable Paddle Intermediate Representation (PIR).This representation systematically abstracts the underlying core concepts and provides flexible and efficient components. PIR serves as the infrastructure to support a number of technologies such as dynamic-to-static, automatic differentiation, auto parallel, combinatorial operators, and graph optimization; it is widely used in scenarios such as distributed training, model compression, and inference deployment. With the Declarative Rewrite Rule (DRR) mechanism provided by PIR, the development cost of Pass can be reduced by 60%.We have tested over 900 model configurations and the results show that the overall performance of inference improves by more than 10% after using PIR. +- Multi-Hardware adaptation: PaddlePaddle provides a well-functioning and low-cost solution for large model hardware adaptation. The new hardware only needs to be adapted with more than 30 interfaces to support training, compression and inference of large models. Meanwhile, PaddlePaddle provides compiler-based hardware access mode, and hardware vendors only need to implement the compiler's code generation back-end in the form of plug-ins to achieve efficient adaptation with the PaddlePaddle framework.PaddlePaddle hardware access this time has additional support for the daily release of four hardware units: Kunlun XPU, Ascend NPU, Hygon DCU and Cambricon MLU. -- In order to avoid misuse, we removed the 0-dimensional Tensor compatibility state switch, to achieve the same API behaviors as industry's mainstream habits. In the previous version, we already supported 0-dimensional Tensor, but we added a compatibility state switch in order to avoid error reporting of some models, as much as possible. That is, in some scenarios where model suite is used frequently and modification is not completed, we still used 1-dimensional Tensor with only 1 element to replace the 0-dimensional Tensor by default. In this version, compatibility state switch is removed, so the 1-dimensional Tensor with only 1 element will no longer be used, to replace 0-dimensional Tensor in any scenario. Behaviors of 376 APIs that should support the 0-dimensional Tensor have been corrected and unified, to thoroughly complete support for the 0-dimensional Tensor.[#57036](https://github.com/PaddlePaddle/Paddle/pull/57036), [#54581](https://github.com/PaddlePaddle/Paddle/pull/54581), [#54500](https://github.com/PaddlePaddle/Paddle/pull/54500) -- To improve API usability, paddle.nn.functional.diag_embed has been streamlined to paddle.diag_embed, with support of use of Tensor.diag_embed. [#58223](https://github.com/PaddlePaddle/Paddle/pull/58223) -- In order to solve the problem of differential computation error caused by Tensor index writing (e.g., tensor[0] = 10) under static graphs, and to comply with static graph specifications, this version introduces paddle.static.setitem API. In static graph environments, this API is recommended to support indexed write operations for tensor, instead of subscript operators. This change does not affect dynamic graph environments, where index write using subscript operators are still allowed. [#53682](https://github.com/PaddlePaddle/Paddle/pull/53682) -- paddle.fluid API is completely retired in this version. In this update, we completely removed all paddle.fluid APIs and deleted the fluid directory. Meanwhile, a small number of PaddlePaddle underlying public components have been consolidated into the paddle.base directory. It is unnecessary for PaddlePaddle users to pay attention to fluid-related concepts and APIs, further simplifying PaddlePaddle API system and improving readability.[#56576](https://github.com/PaddlePaddle/Paddle/pull/56576), [#54424](https://github.com/PaddlePaddle/Paddle/pull/54424), [#54829](https://github.com/PaddlePaddle/Paddle/pull/54829), [#53992](https://github.com/PaddlePaddle/Paddle/pull/53992), [#54806](https://github.com/PaddlePaddle/Paddle/pull/54806), [#55754](https://github.com/PaddlePaddle/Paddle/pull/55754), [#55986](https://github.com/PaddlePaddle/Paddle/pull/55986), [#55345](https://github.com/PaddlePaddle/Paddle/pull/55345), [#56099](https://github.com/PaddlePaddle/Paddle/pull/56099), [#51717](https://github.com/PaddlePaddle/Paddle/pull/51717), [#54152](https://github.com/PaddlePaddle/Paddle/pull/54152), [#55522](https://github.com/PaddlePaddle/Paddle/pull/55522), [#55757](https://github.com/PaddlePaddle/Paddle/pull/55757), [#58521](https://github.com/PaddlePaddle/Paddle/pull/58521), [#54936](https://github.com/PaddlePaddle/Paddle/pull/54936), [#55007](https://github.com/PaddlePaddle/Paddle/pull/55007), [#55661](https://github.com/PaddlePaddle/Paddle/pull/55661), [#55970](https://github.com/PaddlePaddle/Paddle/pull/55970) +This version includes the continuous improvement of some of the existing features of the framework 2.x. Meanwhile, the new features of this version bring significant improvements in terms of user experience, performance, ease of secondary development and hardware adaptability. In addition to the above core features, this version continues to enrich and enhance the API functions to meet more scenarios at the user experience level, optimizes and improves the distributed parallel strategy optimization and reasoning function enhancement for the large model scenarios, makes thorough improvement in terms of ease-of-use in compilation and installation, makes a new synchronous upgrade to the installation method and version of the dependency packages, strengthens the security of the system comprehensively, and makes comprehensive error-correction checks to the product documentation. We have also carried out a cleanup of some deprecated codes to ensure architectural simplicity. The performance of PaddlePaddle 3.0 Beta is still mature and stable without the use of new features, and each new feature provides a switch for flexible control, which makes it easy for users to quickly understand the related product features and experience comparison. -## 3. Training Framework (including Distributed) +## User Experience Upgrade -### Python API +### Incompatibility Upgrade -#### Upgrade Tensor indexing mechanism +- PaddlePaddle API supports type promotion.In the most common calculations such as addition, subtraction, multiplication, and division, if the two inputs are of different data types, it is necessary to determine the data type of the output. Historically, PaddlePaddle partially supported this and the actual rules were not clear. Objectively, there were dynamic-static inconsistency, inconsistent API and operator overloading, and inconsistent interchange rates, and unexpected problems (hard to fix) especially in the case of large models using a mix of bf16/fp16 and fp32 for a wide range of calculations. Starting from the 3.0 beta, PaddlePaddle has clarified the [type promotion rules](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/advanced/auto_type_promotion_cn.html), and defined in detail the types of Tensor vs Tensor and Tensor vs. 1 number (Scalar) computation results, ensuring that the computation conforms to the exchange law, the operator overloading is consistent with the results of the binary API, and the results of the dynamic graph are consistent with those of the static graph. This is more in line with user understanding and industry practice. [#60638](https://github.com/PaddlePaddle/Paddle/pull/60638), [#63842](https://github.com/PaddlePaddle/Paddle/pull/63842), [#60011](https://github.com/PaddlePaddle/Paddle/pull/60011) -This version comprehensively optimizes basic index, advanced index and joint index functions of Tensor, to better comply with industry standards and user habits. Specifically, we added support for view in basic index, fixed some wrong behaviors in advanced index, and implemented read function of joint index. In addition, we have sunk index parsing to C++ level, improved performance of high-level indexing operators, and removed redundant computations in bool indexing. With these optimizations, performance of Tensor's basic, advanced and joint index has been improved comprehensively. [#56893](https://github.com/PaddlePaddle/Paddle/pull/56893), [#58643](https://github.com/PaddlePaddle/Paddle/pull/58643), [#57986](https://github.com/PaddlePaddle/Paddle/pull/57986), [#56272](https://github.com/PaddlePaddle/Paddle/pull/56272), [#58856](https://github.com/PaddlePaddle/Paddle/pull/58856), [#55211](https://github.com/PaddlePaddle/Paddle/pull/55211), [#57023](https://github.com/PaddlePaddle/Paddle/pull/57023), [#56613](https://github.com/PaddlePaddle/Paddle/pull/56613), [#55602](https://github.com/PaddlePaddle/Paddle/pull/55602), [#59281](https://github.com/PaddlePaddle/Paddle/pull/59281), [#57737](https://github.com/PaddlePaddle/Paddle/pull/57737) +### Deprecated Features -#### Upgrade Inplace mechanism +- There have been two versions stably supporting 0-dimensional Tensor. This version removes the switch `FLAGS_set_to_1d` that converts a 0-dimensional Tensor to a 1-dimensional Tensor with only 1 element in some cases. This switch is for compatibility with the incorrect way of writing a 1-element 1-dimensional Tensor to represent a 0-dimensional Tensor in some kits. That is, the current PaddlePaddle fully distinguish between the semantics of a 0-dimensional Tensor and a 1-dimensional Tensor with only 1 element, both are not equivalent. [#61227](https://github.com/PaddlePaddle/Paddle/pull/61227) -In earlier versions, in order to ensure correctness of inverse differentiation calculations, when reverse calculation of an API depends on its forward input data, PaddlePaddle avoids using Inplace operation method, with possibly overwriting original input data. This mechanism simplifies implementation process, and also limits the ability of many APIs to implement Inplace functionality. As a result, user experience may be affected. -In this version, PaddlePaddle has fully upgraded the Inplace mechanism. It implements automatic detection of the dependency of reverse computation on forward inputs, to save input data when needed. Therefore, more Inplace operations are supported. This improvement not only improves memory usage efficiency, but also enhances functionality of the API. -In addition, we have added 109 new APIs that support Inplace operations, including paddle.abs\_, paddle.sin\_/cos\_/tan\_, comparison operations such as paddle.greater_than\_/less_than\_/equal\_, logical operations such as paddle.logical_and\_/logical_or\_/logical_not\_, paddle.neg\_ and paddle.log\_. While enriching the feature set of PaddlePaddle, it improves users' efficiency and convenience in numerical computation and deep learning tasks. [#54683](https://github.com/PaddlePaddle/Paddle/pull/54683), [#55078](https://github.com/PaddlePaddle/Paddle/pull/55078), [#55576](https://github.com/PaddlePaddle/Paddle/pull/55576), [#56888](https://github.com/PaddlePaddle/Paddle/pull/56888), [#55509](https://github.com/PaddlePaddle/Paddle/pull/55509), [#57093](https://github.com/PaddlePaddle/Paddle/pull/57093) +### New API Features -#### Other new APIs +Compared with the previous version, this version is added with 126 new APIs, richer API functions to better support the needs of large models, and scientific computation. The details are as follows: -- Added paddle.nn.functional.scaled_dot_product_attention. This significantly improves computational efficiency of the attention mechanism in large models, and meets demand for high-performance computation in large-scale deep learning models. [#55242](https://github.com/PaddlePaddle/Paddle/pull/55242) -- Added a series of new scientific computing-related APIs, including paddle.cummax and paddle.cummin for cumulative maximum and minimum computation, paddle.index_fill and paddle.masked_fill for filling tensor by index or mask, paddle.linalg.pca_lowrank for low-rank principal component analysis, paddle.hypot for calculating length of the hypotenuses of right triangles, and paddle.atleast_1d, paddle.atleast_2d, and paddle.atleast_3d to ensure the tensor is at least one, two, or three dimensional. We also provide paddle.select_scatter and paddle.diagonal_scatter for more flexible selection and hashing of tensor data, and paddle.multigammaln for choosing the natural logarithm of multigamma function. In addition, new optimizer-related APIs are added in this version, including: paddle.optimizer.lr.LinearLR and paddle.optimizer.lr.CosineAnnealingWarmRestarts for learning rate scheduling strategies; introduction of paddle.io.SubsetRandomSampler to support random sampling from a subset of data. These added APIs will further enhance flexibility and efficiency of PaddlePaddle in various application scenarios. [#57416](https://github.com/PaddlePaddle/Paddle/pull/57416), [#53546](https://github.com/PaddlePaddle/Paddle/pull/53546), [#53743](https://github.com/PaddlePaddle/Paddle/pull/53743), [#57295](https://github.com/PaddlePaddle/Paddle/pull/57295), [#57726](https://github.com/PaddlePaddle/Paddle/pull/57726), [#58764](https://github.com/PaddlePaddle/Paddle/pull/58764), [#58323](https://github.com/PaddlePaddle/Paddle/pull/58323), [#57720](https://github.com/PaddlePaddle/Paddle/pull/57720), [#58209](https://github.com/PaddlePaddle/Paddle/pull/58209), [#58214](https://github.com/PaddlePaddle/Paddle/pull/58214), [#57792](https://github.com/PaddlePaddle/Paddle/pull/57792), [#51395](https://github.com/PaddlePaddle/Paddle/pull/51395), [#57724](https://github.com/PaddlePaddle/Paddle/pull/57724), [#57355](https://github.com/PaddlePaddle/Paddle/pull/57355), [#57744](https://github.com/PaddlePaddle/Paddle/pull/57744), [#58244](https://github.com/PaddlePaddle/Paddle/pull/58244), [#57599](https://github.com/PaddlePaddle/Paddle/pull/57599), [#59343](https://github.com/PaddlePaddle/Paddle/pull/59343), [#57879](https://github.com/PaddlePaddle/Paddle/pull/57879) +- Add Tensor computation API. `paddle.gammaln`, `paddle.gammainc`, `paddle.gammaincc`, `paddle.sinc`, `paddle.pdist`, `paddle.histogramdd`,`paddle.signbit`, `paddle.copysign`, `paddle.bitwise_right_shift/bitwise_left_shift`, `paddle.isposinf/isneginf/isreal`, `paddle.isin`, `paddle.hsplit/dsplit`, `paddle.column_stack/row_stack/dstack/hstack/vstack`, `paddle.slice_scatter`, `paddle.masked_scatter` [#60553](https://github.com/PaddlePaddle/Paddle/pull/60553), [#59311](https://github.com/PaddlePaddle/Paddle/pull/59311), [#59357](https://github.com/PaddlePaddle/Paddle/pull/59357), [#63521](https://github.com/PaddlePaddle/Paddle/pull/63521), [#57869](https://github.com/PaddlePaddle/Paddle/pull/57869), [#57880](https://github.com/PaddlePaddle/Paddle/pull/57880), [#57882](https://github.com/PaddlePaddle/Paddle/pull/57882), [#60150](https://github.com/PaddlePaddle/Paddle/pull/60150), [#57785](https://github.com/PaddlePaddle/Paddle/pull/57785), [#58092](https://github.com/PaddlePaddle/Paddle/pull/58092), [#63523](https://github.com/PaddlePaddle/Paddle/pull/63523), [#64001](https://github.com/PaddlePaddle/Paddle/pull/64001), [#58917](https://github.com/PaddlePaddle/Paddle/pull/58917), [#59127](https://github.com/PaddlePaddle/Paddle/pull/59127), [#59973](https://github.com/PaddlePaddle/Paddle/pull/59973), [#59383](https://github.com/PaddlePaddle/Paddle/pull/59383) +- Add probability distribution API. `paddle.distribution.ContinuousBernoulli`, `paddle.distribution.MultivariateNormal`, `paddle.distribution.Exponential`, `paddle.distribution.Gamma`, `paddle.distribution.Binomial`, `paddle.distribution.Poisson` [#58004](https://github.com/PaddlePaddle/Paddle/pull/58004), [#57899](https://github.com/PaddlePaddle/Paddle/pull/57899), [#57856](https://github.com/PaddlePaddle/Paddle/pull/57856) +- Add optimizer API. `paddle.optimizer.ASGD`, `paddle.optimizer.NAdam`, `paddle.optimizer.RAdam`, `paddle.optimizer.Rprop` [#58834](https://github.com/PaddlePaddle/Paddle/pull/58834), [#63671](https://github.com/PaddlePaddle/Paddle/pull/63671), [#58851](https://github.com/PaddlePaddle/Paddle/pull/58851) +- Add Linear Algebra API. `paddle.linalg.matrix_exp` [#59715](https://github.com/PaddlePaddle/Paddle/pull/59715) +- Add other APIs. `paddle.bernoulli_`, `paddle.nn.ZeroPad1D/ZeroPad3D`, `paddle.nn.AdaptiveLogSoftmaxWithLoss`, `paddle.Tensor.apply` [#64252](https://github.com/PaddlePaddle/Paddle/pull/64252), [#59690](https://github.com/PaddlePaddle/Paddle/pull/59690), [#63728](https://github.com/PaddlePaddle/Paddle/pull/63728), [#63302](https://github.com/PaddlePaddle/Paddle/pull/63302), [#59374](https://github.com/PaddlePaddle/Paddle/pull/59374),[#63227](https://github.com/PaddlePaddle/Paddle/pull/63227) -### New Generation of Paddle Intermediate Representation (PIR) +### Some API Enhancements -PIR systematically abstracts underlying core concepts such as Operation, Attribute and Type, to build a set of flexible and powerful base components for developers. In addition, PaddlePaddle can comprehensively and hierarchically manage requirements of each module on Intermediate Representation (IR) by introducing the concept of Dialect, and support developers to customize extension of Dialect according to specific needs to significantly improving scalability and adaptability of framework. In terms of designs, PIR strictly follows the Static Single Assignment (SSA) principle, unifies top-level structure, realizes compatibility of "Operator sequentiality" and "computational graph semantics". This provides a clear and consistent view of the complex computation process. In order to further optimize performance of large models, PIR also provides a set of more concise and low-cost Pass development processes, including Declarative Rewrite Rule (DRR) and Pattern Rewriter. In addition, a series of rich and full-featured Pass optimization strategies are built-in, to deeply optimize application according to characteristics of large models, thus providing strong support for ultimate performance of large models. Through these innovative designs and optimization methods, PIR lays a solid foundation for efficient operation and continuous expansion of the PaddlePaddle framework. +- Enhance about 30 APIs to support complex number computation, such as `paddle.log`, `paddle.log1p`, `paddle.square`, and `paddle.reciprocal`, to extend the support for more scientific computing scenarios. [#62448](https://github.com/PaddlePaddle/Paddle/pull/62448), [#60821](https://github.com/PaddlePaddle/Paddle/pull/60821), [#60897](https://github.com/PaddlePaddle/Paddle/pull/60897), [#62764](https://github.com/PaddlePaddle/Paddle/pull/62764), [#59536](https://github.com/PaddlePaddle/Paddle/pull/59536), [#59529](https://github.com/PaddlePaddle/Paddle/pull/59529), [#63207](https://github.com/PaddlePaddle/Paddle/pull/63207), [#62237](https://github.com/PaddlePaddle/Paddle/pull/62237), [#64684](https://github.com/PaddlePaddle/Paddle/pull/64684) +- Enhance 46 APIs, to make existing APIs easier to use and easier to convert to codes,including but not limited to, adding API parameters, extending the data types supported by the APIs, and fixing the existing unreasonable designs. [#59890](https://github.com/PaddlePaddle/Paddle/pull/59890), [#63513](https://github.com/PaddlePaddle/Paddle/pull/63513), [#59674](https://github.com/PaddlePaddle/Paddle/pull/59674), [#62778](https://github.com/PaddlePaddle/Paddle/pull/62778), [#64110](https://github.com/PaddlePaddle/Paddle/pull/64110), [#63222](https://github.com/PaddlePaddle/Paddle/pull/63222), [#64331](https://github.com/PaddlePaddle/Paddle/pull/64331), [#64715](https://github.com/PaddlePaddle/Paddle/pull/64715), [#61155](https://github.com/PaddlePaddle/Paddle/pull/61155), [#60070](https://github.com/PaddlePaddle/Paddle/pull/60070), [#61974](https://github.com/PaddlePaddle/Paddle/pull/61974), [#62407](https://github.com/PaddlePaddle/Paddle/pull/62407), [#62672](https://github.com/PaddlePaddle/Paddle/pull/62672),[#62722](https://github.com/PaddlePaddle/Paddle/pull/62722)， [#62876](https://github.com/PaddlePaddle/Paddle/pull/62876), [#63284](https://github.com/PaddlePaddle/Paddle/pull/63284), [#63860](https://github.com/PaddlePaddle/Paddle/pull/63860), [#60466](https://github.com/PaddlePaddle/Paddle/pull/60466), [#63690](https://github.com/PaddlePaddle/Paddle/pull/63690), [#63953](https://github.com/PaddlePaddle/Paddle/pull/63953), [#63901](https://github.com/PaddlePaddle/Paddle/pull/63901), [#62624](https://github.com/PaddlePaddle/Paddle/pull/62624), [#59857](https://github.com/PaddlePaddle/Paddle/pull/59857), [#60084](https://github.com/PaddlePaddle/Paddle/pull/60084), [#60766](https://github.com/PaddlePaddle/Paddle/pull/60766), [#62788](https://github.com/PaddlePaddle/Paddle/pull/62788), [#62937](https://github.com/PaddlePaddle/Paddle/pull/62937), [#63134](https://github.com/PaddlePaddle/Paddle/pull/63134), [#62966](https://github.com/PaddlePaddle/Paddle/pull/62966), [#63648](https://github.com/PaddlePaddle/Paddle/pull/63648), [#63881](https://github.com/PaddlePaddle/Paddle/pull/63881), [#64358](https://github.com/PaddlePaddle/Paddle/pull/64358), [#60503](https://github.com/PaddlePaddle/Paddle/pull/60503), [#63604](https://github.com/PaddlePaddle/Paddle/pull/63604), [#62338](https://github.com/PaddlePaddle/Paddle/pull/62338) +- Enhance single-test infrastructure for higher-order differentiation, making it easier to add single-test use cases for higher-order differentiation. [#62074](https://github.com/PaddlePaddle/Paddle/pull/62074) -#### New features +### API Performance Improvements -- Abstracted core concepts of IR bottom layer and provided developers with flexible base components, such as Operation, Attribute, Value, Type, Trait, and Interface. [#56354](https://github.com/PaddlePaddle/Paddle/pull/56354),[#57106](https://github.com/PaddlePaddle/Paddle/pull/57106),[#57349](https://github.com/PaddlePaddle/Paddle/pull/57349),[#54844](https://github.com/PaddlePaddle/Paddle/pull/54844),[#54984](https://github.com/PaddlePaddle/Paddle/pull/54984),[#54565](https://github.com/PaddlePaddle/Paddle/pull/54565),[#54562](https://github.com/PaddlePaddle/Paddle/pull/54562),[#57249](https://github.com/PaddlePaddle/Paddle/pull/57249),[#57550](https://github.com/PaddlePaddle/Paddle/pull/57550),[#59278](https://github.com/PaddlePaddle/Paddle/pull/59278),[#54875](https://github.com/PaddlePaddle/Paddle/pull/54875),[#55041](https://github.com/PaddlePaddle/Paddle/pull/55041),[#54987](https://github.com/PaddlePaddle/Paddle/pull/54987),[#55903](https://github.com/PaddlePaddle/Paddle/pull/55903),[#57582](https://github.com/PaddlePaddle/Paddle/pull/57582),[#57580](https://github.com/PaddlePaddle/Paddle/pull/57580),[#58052](https://github.com/PaddlePaddle/Paddle/pull/58052),[#55322](https://github.com/PaddlePaddle/Paddle/pull/55322),[#57418](https://github.com/PaddlePaddle/Paddle/pull/57418),[#57635](https://github.com/PaddlePaddle/Paddle/pull/57635),[#55328](https://github.com/PaddlePaddle/Paddle/pull/55328),[#57463](https://github.com/PaddlePaddle/Paddle/pull/57463),[#59791](https://github.com/PaddlePaddle/Paddle/pull/59791),[#59821](https://github.com/PaddlePaddle/Paddle/pull/59821),[#59115](https://github.com/PaddlePaddle/Paddle/pull/59115),[#57461](https://github.com/PaddlePaddle/Paddle/pull/57461),[#59392](https://github.com/PaddlePaddle/Paddle/pull/59392),[#57373](https://github.com/PaddlePaddle/Paddle/pull/57373),[#59118](https://github.com/PaddlePaddle/Paddle/pull/59118) -- Added Dialect mechanism to support comprehensive and hierarchical management of intermediate representation requirements of each module of framework. Through built-in Builtin Dialect, it supports developers to customize and extend Dialect according to their needs. [#56325](https://github.com/PaddlePaddle/Paddle/pull/56325),[#57539](https://github.com/PaddlePaddle/Paddle/pull/57539),[#54682](https://github.com/PaddlePaddle/Paddle/pull/54682),[#55381](https://github.com/PaddlePaddle/Paddle/pull/55381),[#56156](https://github.com/PaddlePaddle/Paddle/pull/56156),[#56431](https://github.com/PaddlePaddle/Paddle/pull/56431),[#56615](https://github.com/PaddlePaddle/Paddle/pull/56615),[#57103](https://github.com/PaddlePaddle/Paddle/pull/57103),[#57209](https://github.com/PaddlePaddle/Paddle/pull/57209) -- Normalized PaddlePaddle static graph operator system. Added OperatorDialect and KernelDialect. Managed conceptual differences of operators in the form of Dialect during compilation and execution, making Architecture clearer. [#56284](https://github.com/PaddlePaddle/Paddle/pull/56284),[#54469](https://github.com/PaddlePaddle/Paddle/pull/54469),[#58660](https://github.com/PaddlePaddle/Paddle/pull/58660),[#58975](https://github.com/PaddlePaddle/Paddle/pull/58975),[#56680](https://github.com/PaddlePaddle/Paddle/pull/56680),[#54790](https://github.com/PaddlePaddle/Paddle/pull/54790),[#54826](https://github.com/PaddlePaddle/Paddle/pull/54826),[#54840](https://github.com/PaddlePaddle/Paddle/pull/54840),[#55699](https://github.com/PaddlePaddle/Paddle/pull/55699),[#55648](https://github.com/PaddlePaddle/Paddle/pull/55648),[#55880](https://github.com/PaddlePaddle/Paddle/pull/55880),[#56101](https://github.com/PaddlePaddle/Paddle/pull/56101),[#56754](https://github.com/PaddlePaddle/Paddle/pull/56754),[#54944](https://github.com/PaddlePaddle/Paddle/pull/54944),[#56836](https://github.com/PaddlePaddle/Paddle/pull/56836),[#57185](https://github.com/PaddlePaddle/Paddle/pull/57185),[#58757](https://github.com/PaddlePaddle/Paddle/pull/58757),[#56243](https://github.com/PaddlePaddle/Paddle/pull/56243),[#56436](https://github.com/PaddlePaddle/Paddle/pull/56436),[#57741](https://github.com/PaddlePaddle/Paddle/pull/57741),[#59124](https://github.com/PaddlePaddle/Paddle/pull/59124),[#57054](https://github.com/PaddlePaddle/Paddle/pull/57054),[#56984](https://github.com/PaddlePaddle/Paddle/pull/56984),[#57403](https://github.com/PaddlePaddle/Paddle/pull/57403),[#57904](https://github.com/PaddlePaddle/Paddle/pull/57904),[#58031](https://github.com/PaddlePaddle/Paddle/pull/58031),[#56924](https://github.com/PaddlePaddle/Paddle/pull/56924),[#59270](https://github.com/PaddlePaddle/Paddle/pull/59270),[#55343](https://github.com/PaddlePaddle/Paddle/pull/55343),[#56557](https://github.com/PaddlePaddle/Paddle/pull/56557),[#55693](https://github.com/PaddlePaddle/Paddle/pull/55693),[#54428](https://github.com/PaddlePaddle/Paddle/pull/54428) -- Added ShapeDialect with built-in rich shape operation operators for constructing dynamic shape constraints and expressions for AI compilers. [#56727](https://github.com/PaddlePaddle/Paddle/pull/56727),[#59254](https://github.com/PaddlePaddle/Paddle/pull/59254),[#58368](https://github.com/PaddlePaddle/Paddle/pull/58368),[#57069](https://github.com/PaddlePaddle/Paddle/pull/57069),[#57337](https://github.com/PaddlePaddle/Paddle/pull/57337),[#56351](https://github.com/PaddlePaddle/Paddle/pull/56351),[#57029](https://github.com/PaddlePaddle/Paddle/pull/57029),[#58036](https://github.com/PaddlePaddle/Paddle/pull/58036),[#59032](https://github.com/PaddlePaddle/Paddle/pull/59032),[#57961](https://github.com/PaddlePaddle/Paddle/pull/57961),[#56427](https://github.com/PaddlePaddle/Paddle/pull/56427),[#57459](https://github.com/PaddlePaddle/Paddle/pull/57459) -- Unified top-level structure of Framework Program, supporting compatible representation of "operator sequentiality" and "computational graph semantics", decoupling dependency on ir::Graph, and strictly following the principle of Static Single Assignment (SSA). [#59369](https://github.com/PaddlePaddle/Paddle/pull/59369),[#54563](https://github.com/PaddlePaddle/Paddle/pull/54563),[#57051](https://github.com/PaddlePaddle/Paddle/pull/57051),[#57306](https://github.com/PaddlePaddle/Paddle/pull/57306),[#57857](https://github.com/PaddlePaddle/Paddle/pull/57857) -- Added IrPrinter and IrPaser components to support serialization and deserialization of PIR Programs, providing a friendly debugging experience for PIR development. [#55695](https://github.com/PaddlePaddle/Paddle/pull/55695),[#59449](https://github.com/PaddlePaddle/Paddle/pull/59449),[#54369](https://github.com/PaddlePaddle/Paddle/pull/54369),[#54499](https://github.com/PaddlePaddle/Paddle/pull/54499),[#55518](https://github.com/PaddlePaddle/Paddle/pull/55518),[#55784](https://github.com/PaddlePaddle/Paddle/pull/55784),[#57180](https://github.com/PaddlePaddle/Paddle/pull/57180),[#57471](https://github.com/PaddlePaddle/Paddle/pull/57471),[#54859](https://github.com/PaddlePaddle/Paddle/pull/54859),[#54968](https://github.com/PaddlePaddle/Paddle/pull/54968),[#55209](https://github.com/PaddlePaddle/Paddle/pull/55209),[#57314](https://github.com/PaddlePaddle/Paddle/pull/57314),[#57969](https://github.com/PaddlePaddle/Paddle/pull/57969) -- Built a new, simple and low-cost Pass development system based on Declarative Rewrite Rule (DDR) and Pattern Rewriter, with built-in a series of rich and full-featured Pass Optimization strategies, to accelerate training and inference execution process. [#54385](https://github.com/PaddlePaddle/Paddle/pull/54385),[#54738](https://github.com/PaddlePaddle/Paddle/pull/54738),[#55859](https://github.com/PaddlePaddle/Paddle/pull/55859),[#56638](https://github.com/PaddlePaddle/Paddle/pull/56638),[#57090](https://github.com/PaddlePaddle/Paddle/pull/57090),[#58673](https://github.com/PaddlePaddle/Paddle/pull/58673),[#59415](https://github.com/PaddlePaddle/Paddle/pull/59415),[#56729](https://github.com/PaddlePaddle/Paddle/pull/56729),[#58655](https://github.com/PaddlePaddle/Paddle/pull/58655) -- Added ProgramTranslator component, to support conversion from ProgramDesc to new generation of IR representations of PaddlePaddle by pressing one key, with provision of easy-to-use C++ and Python interfaces. [#55433](https://github.com/PaddlePaddle/Paddle/pull/55433),[#54470](https://github.com/PaddlePaddle/Paddle/pull/54470),[#58044](https://github.com/PaddlePaddle/Paddle/pull/58044),[#58390](https://github.com/PaddlePaddle/Paddle/pull/58390),[#58100](https://github.com/PaddlePaddle/Paddle/pull/58100),[#55403](https://github.com/PaddlePaddle/Paddle/pull/55403),[#55406](https://github.com/PaddlePaddle/Paddle/pull/55406),[#54719](https://github.com/PaddlePaddle/Paddle/pull/54719),[#56550](https://github.com/PaddlePaddle/Paddle/pull/56550),[#55448](https://github.com/PaddlePaddle/Paddle/pull/55448),[#55453](https://github.com/PaddlePaddle/Paddle/pull/55453),[#56294](https://github.com/PaddlePaddle/Paddle/pull/56294),[#56308](https://github.com/PaddlePaddle/Paddle/pull/56308),[#56842](https://github.com/PaddlePaddle/Paddle/pull/56842),[#58517](https://github.com/PaddlePaddle/Paddle/pull/58517) -- With help of automatic code generation technology, it can generate the full amount of static graph operator representations for PaddlePaddle framework by pressing one key. Sank static graph networking logic to C++ side and bind it to \_C_ops module. This can greatly streamline API code on Python side, realize ultimate dynamic-static unification of APIs of PaddlePaddle Framework, and upgrade a lot of Python APIs to support static graph networking of the new IR. [#56570](https://github.com/PaddlePaddle/Paddle/pull/56570),[#55745](https://github.com/PaddlePaddle/Paddle/pull/55745),[#56955](https://github.com/PaddlePaddle/Paddle/pull/56955),[#57298](https://github.com/PaddlePaddle/Paddle/pull/57298),[#57946](https://github.com/PaddlePaddle/Paddle/pull/57946),[#57248](https://github.com/PaddlePaddle/Paddle/pull/57248),[#56080](https://github.com/PaddlePaddle/Paddle/pull/56080),[#54396](https://github.com/PaddlePaddle/Paddle/pull/54396),[#54551](https://github.com/PaddlePaddle/Paddle/pull/54551),[#56520](https://github.com/PaddlePaddle/Paddle/pull/56520),[#55002](https://github.com/PaddlePaddle/Paddle/pull/55002),[#57067](https://github.com/PaddlePaddle/Paddle/pull/57067),[#59320](https://github.com/PaddlePaddle/Paddle/pull/59320),[#59348](https://github.com/PaddlePaddle/Paddle/pull/59348),[#57164](https://github.com/PaddlePaddle/Paddle/pull/57164),[#57267](https://github.com/PaddlePaddle/Paddle/pull/57267),[#59064](https://github.com/PaddlePaddle/Paddle/pull/59064),[#54340](https://github.com/PaddlePaddle/Paddle/pull/54340),[#54895](https://github.com/PaddlePaddle/Paddle/pull/54895),[#55004](https://github.com/PaddlePaddle/Paddle/pull/55004),[#56196](https://github.com/PaddlePaddle/Paddle/pull/56196),[#56862](https://github.com/PaddlePaddle/Paddle/pull/56862),[#58991](https://github.com/PaddlePaddle/Paddle/pull/58991),[#55428](https://github.com/PaddlePaddle/Paddle/pull/55428),[#55909](https://github.com/PaddlePaddle/Paddle/pull/55909),[#56241](https://github.com/PaddlePaddle/Paddle/pull/56241),[#56526](https://github.com/PaddlePaddle/Paddle/pull/56526),[#56571](https://github.com/PaddlePaddle/Paddle/pull/56571),[#56518](https://github.com/PaddlePaddle/Paddle/pull/56518),[#57016](https://github.com/PaddlePaddle/Paddle/pull/57016),[#56653](https://github.com/PaddlePaddle/Paddle/pull/56653),[#56809](https://github.com/PaddlePaddle/Paddle/pull/56809),[#57158](https://github.com/PaddlePaddle/Paddle/pull/57158),[#55422](https://github.com/PaddlePaddle/Paddle/pull/55422),[#55458](https://github.com/PaddlePaddle/Paddle/pull/55458),[#55432](https://github.com/PaddlePaddle/Paddle/pull/55432),[#55467](https://github.com/PaddlePaddle/Paddle/pull/55467),[#55483](https://github.com/PaddlePaddle/Paddle/pull/55483),[#55419](https://github.com/PaddlePaddle/Paddle/pull/55419),[#55517](https://github.com/PaddlePaddle/Paddle/pull/55517),[#55500](https://github.com/PaddlePaddle/Paddle/pull/55500),[#56674](https://github.com/PaddlePaddle/Paddle/pull/56674),[#57693](https://github.com/PaddlePaddle/Paddle/pull/57693),[#55008](https://github.com/PaddlePaddle/Paddle/pull/55008),[#57166](https://github.com/PaddlePaddle/Paddle/pull/57166),[#57157](https://github.com/PaddlePaddle/Paddle/pull/57157),[#57159](https://github.com/PaddlePaddle/Paddle/pull/57159),[#57175](https://github.com/PaddlePaddle/Paddle/pull/57175),[#57325](https://github.com/PaddlePaddle/Paddle/pull/57325),[#57330](https://github.com/PaddlePaddle/Paddle/pull/57330),[#57415](https://github.com/PaddlePaddle/Paddle/pull/57415),[#57122](https://github.com/PaddlePaddle/Paddle/pull/57122),[#57393](https://github.com/PaddlePaddle/Paddle/pull/57393),[#57344](https://github.com/PaddlePaddle/Paddle/pull/57344),[#57667](https://github.com/PaddlePaddle/Paddle/pull/57667),[#57348](https://github.com/PaddlePaddle/Paddle/pull/57348),[#57700](https://github.com/PaddlePaddle/Paddle/pull/57700),[#58093](https://github.com/PaddlePaddle/Paddle/pull/58093),[#58005](https://github.com/PaddlePaddle/Paddle/pull/58005),[#58081](https://github.com/PaddlePaddle/Paddle/pull/58081),[#58094](https://github.com/PaddlePaddle/Paddle/pull/58094),[#58137](https://github.com/PaddlePaddle/Paddle/pull/58137),[#58287](https://github.com/PaddlePaddle/Paddle/pull/58287),[#58352](https://github.com/PaddlePaddle/Paddle/pull/58352),[#58340](https://github.com/PaddlePaddle/Paddle/pull/58340),[#58363](https://github.com/PaddlePaddle/Paddle/pull/58363),[#58331](https://github.com/PaddlePaddle/Paddle/pull/58331),[#58343](https://github.com/PaddlePaddle/Paddle/pull/58343),[#58317](https://github.com/PaddlePaddle/Paddle/pull/58317),[#58450](https://github.com/PaddlePaddle/Paddle/pull/58450),[#58377](https://github.com/PaddlePaddle/Paddle/pull/58377),[#58466](https://github.com/PaddlePaddle/Paddle/pull/58466),[#58470](https://github.com/PaddlePaddle/Paddle/pull/58470),[#58491](https://github.com/PaddlePaddle/Paddle/pull/58491),[#58546](https://github.com/PaddlePaddle/Paddle/pull/58546),[#58587](https://github.com/PaddlePaddle/Paddle/pull/58587),[#58453](https://github.com/PaddlePaddle/Paddle/pull/58453),[#58634](https://github.com/PaddlePaddle/Paddle/pull/58634),[#58604](https://github.com/PaddlePaddle/Paddle/pull/58604),[#58605](https://github.com/PaddlePaddle/Paddle/pull/58605),[#58593](https://github.com/PaddlePaddle/Paddle/pull/58593),[#58675](https://github.com/PaddlePaddle/Paddle/pull/58675),[#58699](https://github.com/PaddlePaddle/Paddle/pull/58699),[#58384](https://github.com/PaddlePaddle/Paddle/pull/58384),[#58629](https://github.com/PaddlePaddle/Paddle/pull/58629),[#58579](https://github.com/PaddlePaddle/Paddle/pull/58579),[#58695](https://github.com/PaddlePaddle/Paddle/pull/58695),[#58548](https://github.com/PaddlePaddle/Paddle/pull/58548),[#58688](https://github.com/PaddlePaddle/Paddle/pull/58688),[#58792](https://github.com/PaddlePaddle/Paddle/pull/58792),[#58843](https://github.com/PaddlePaddle/Paddle/pull/58843),[#58840](https://github.com/PaddlePaddle/Paddle/pull/58840),[#58718](https://github.com/PaddlePaddle/Paddle/pull/58718),[#58883](https://github.com/PaddlePaddle/Paddle/pull/58883),[#58785](https://github.com/PaddlePaddle/Paddle/pull/58785),[#58608](https://github.com/PaddlePaddle/Paddle/pull/58608),[#58781](https://github.com/PaddlePaddle/Paddle/pull/58781),[#58783](https://github.com/PaddlePaddle/Paddle/pull/58783),[#58429](https://github.com/PaddlePaddle/Paddle/pull/58429),[#58685](https://github.com/PaddlePaddle/Paddle/pull/58685),[#58696](https://github.com/PaddlePaddle/Paddle/pull/58696),[#58690](https://github.com/PaddlePaddle/Paddle/pull/58690),[#58831](https://github.com/PaddlePaddle/Paddle/pull/58831),[#58929](https://github.com/PaddlePaddle/Paddle/pull/58929),[#58740](https://github.com/PaddlePaddle/Paddle/pull/58740),[#58937](https://github.com/PaddlePaddle/Paddle/pull/58937),[#58782](https://github.com/PaddlePaddle/Paddle/pull/58782),[#58833](https://github.com/PaddlePaddle/Paddle/pull/58833),[#58882](https://github.com/PaddlePaddle/Paddle/pull/58882),[#58935](https://github.com/PaddlePaddle/Paddle/pull/58935),[#58931](https://github.com/PaddlePaddle/Paddle/pull/58931),[#59041](https://github.com/PaddlePaddle/Paddle/pull/59041),[#59040](https://github.com/PaddlePaddle/Paddle/pull/59040),[#58877](https://github.com/PaddlePaddle/Paddle/pull/58877),[#58888](https://github.com/PaddlePaddle/Paddle/pull/58888),[#59042](https://github.com/PaddlePaddle/Paddle/pull/59042),[#58780](https://github.com/PaddlePaddle/Paddle/pull/58780),[#58682](https://github.com/PaddlePaddle/Paddle/pull/58682),[#58815](https://github.com/PaddlePaddle/Paddle/pull/58815),[#58676](https://github.com/PaddlePaddle/Paddle/pull/58676),[#58678](https://github.com/PaddlePaddle/Paddle/pull/58678),[#58446](https://github.com/PaddlePaddle/Paddle/pull/58446),[#59077](https://github.com/PaddlePaddle/Paddle/pull/59077),[#59091](https://github.com/PaddlePaddle/Paddle/pull/59091),[#58661](https://github.com/PaddlePaddle/Paddle/pull/58661),[#58832](https://github.com/PaddlePaddle/Paddle/pull/58832),[#58642](https://github.com/PaddlePaddle/Paddle/pull/58642),[#58698](https://github.com/PaddlePaddle/Paddle/pull/58698),[#59313](https://github.com/PaddlePaddle/Paddle/pull/59313),[#59371](https://github.com/PaddlePaddle/Paddle/pull/59371),[#58700](https://github.com/PaddlePaddle/Paddle/pull/58700),[#58953](https://github.com/PaddlePaddle/Paddle/pull/58953),[#58879](https://github.com/PaddlePaddle/Paddle/pull/58879),[#59469](https://github.com/PaddlePaddle/Paddle/pull/59469),[#59573](https://github.com/PaddlePaddle/Paddle/pull/59573),[#59481](https://github.com/PaddlePaddle/Paddle/pull/59481),[#59419](https://github.com/PaddlePaddle/Paddle/pull/59419),[#59509](https://github.com/PaddlePaddle/Paddle/pull/59509),[#58735](https://github.com/PaddlePaddle/Paddle/pull/58735),[#59616](https://github.com/PaddlePaddle/Paddle/pull/59616),[#59582](https://github.com/PaddlePaddle/Paddle/pull/59582),[#59420](https://github.com/PaddlePaddle/Paddle/pull/59420),[#59500](https://github.com/PaddlePaddle/Paddle/pull/59500),[#58911](https://github.com/PaddlePaddle/Paddle/pull/58911),[#59535](https://github.com/PaddlePaddle/Paddle/pull/59535),[#54891](https://github.com/PaddlePaddle/Paddle/pull/54891),[#56794](https://github.com/PaddlePaddle/Paddle/pull/56794),[#57477](https://github.com/PaddlePaddle/Paddle/pull/57477),[#57929](https://github.com/PaddlePaddle/Paddle/pull/57929),[#57765](https://github.com/PaddlePaddle/Paddle/pull/57765),[#58693](https://github.com/PaddlePaddle/Paddle/pull/58693),[#58603](https://github.com/PaddlePaddle/Paddle/pull/58603),[#56291](https://github.com/PaddlePaddle/Paddle/pull/56291),[#57123](https://github.com/PaddlePaddle/Paddle/pull/57123),[#57317](https://github.com/PaddlePaddle/Paddle/pull/57317),[#57341](https://github.com/PaddlePaddle/Paddle/pull/57341),[#57020](https://github.com/PaddlePaddle/Paddle/pull/57020),[#57324](https://github.com/PaddlePaddle/Paddle/pull/57324),[#57761](https://github.com/PaddlePaddle/Paddle/pull/57761),[#57762](https://github.com/PaddlePaddle/Paddle/pull/57762),[#57907](https://github.com/PaddlePaddle/Paddle/pull/57907),[#57909](https://github.com/PaddlePaddle/Paddle/pull/57909),[#58099](https://github.com/PaddlePaddle/Paddle/pull/58099),[#58110](https://github.com/PaddlePaddle/Paddle/pull/58110),[#58114](https://github.com/PaddlePaddle/Paddle/pull/58114),[#58139](https://github.com/PaddlePaddle/Paddle/pull/58139),[#58144](https://github.com/PaddlePaddle/Paddle/pull/58144),[#58165](https://github.com/PaddlePaddle/Paddle/pull/58165),[#58194](https://github.com/PaddlePaddle/Paddle/pull/58194),[#58138](https://github.com/PaddlePaddle/Paddle/pull/58138),[#58113](https://github.com/PaddlePaddle/Paddle/pull/58113),[#58245](https://github.com/PaddlePaddle/Paddle/pull/58245),[#58318](https://github.com/PaddlePaddle/Paddle/pull/58318),[#58105](https://github.com/PaddlePaddle/Paddle/pull/58105),[#58348](https://github.com/PaddlePaddle/Paddle/pull/58348),[#58235](https://github.com/PaddlePaddle/Paddle/pull/58235),[#58354](https://github.com/PaddlePaddle/Paddle/pull/58354),[#58341](https://github.com/PaddlePaddle/Paddle/pull/58341),[#58445](https://github.com/PaddlePaddle/Paddle/pull/58445),[#58418](https://github.com/PaddlePaddle/Paddle/pull/58418),[#58239](https://github.com/PaddlePaddle/Paddle/pull/58239),[#58473](https://github.com/PaddlePaddle/Paddle/pull/58473),[#58239](https://github.com/PaddlePaddle/Paddle/pull/58239),[#58391](https://github.com/PaddlePaddle/Paddle/pull/58391),[#58501](https://github.com/PaddlePaddle/Paddle/pull/58501),[#58519](https://github.com/PaddlePaddle/Paddle/pull/58519),[#58416](https://github.com/PaddlePaddle/Paddle/pull/58416),[#58588](https://github.com/PaddlePaddle/Paddle/pull/58588),[#58531](https://github.com/PaddlePaddle/Paddle/pull/58531),[#58730](https://github.com/PaddlePaddle/Paddle/pull/58730),[#58773](https://github.com/PaddlePaddle/Paddle/pull/58773),[#58862](https://github.com/PaddlePaddle/Paddle/pull/58862),[#58946](https://github.com/PaddlePaddle/Paddle/pull/58946),[#58500](https://github.com/PaddlePaddle/Paddle/pull/58500),[#56585](https://github.com/PaddlePaddle/Paddle/pull/56585),[#57480](https://github.com/PaddlePaddle/Paddle/pull/57480),[#57433](https://github.com/PaddlePaddle/Paddle/pull/57433),[#58498](https://github.com/PaddlePaddle/Paddle/pull/58498) +- Focus on optimizing the performance of Tensor basic index, advanced index, and combined index, improving computational performance by 2X to 31X on GPUs and 1.8X to 1004X on CPUs. [#60254](https://github.com/PaddlePaddle/Paddle/pull/60254), [#60276](https://github.com/PaddlePaddle/Paddle/pull/60276), [#60452](https://github.com/PaddlePaddle/Paddle/pull/60452), [#60771](https://github.com/PaddlePaddle/Paddle/pull/60771), [#61021](https://github.com/PaddlePaddle/Paddle/pull/61021), [#60983](https://github.com/PaddlePaddle/Paddle/pull/60983), [#61060](https://github.com/PaddlePaddle/Paddle/pull/61060), [#60618](https://github.com/PaddlePaddle/Paddle/pull/60618) -#### Function optimization +### Bug Fixing -- Upgraded static graph executor to extend more Kernel Instruction types, and supported loading of PIR with efficiently scheduling execution. This has significant video memory and performance gains in training and inference. [#54570](https://github.com/PaddlePaddle/Paddle/pull/54570),[#58665](https://github.com/PaddlePaddle/Paddle/pull/58665),[#57291](https://github.com/PaddlePaddle/Paddle/pull/57291),[#54452](https://github.com/PaddlePaddle/Paddle/pull/54452),[#57431](https://github.com/PaddlePaddle/Paddle/pull/57431),[#54692](https://github.com/PaddlePaddle/Paddle/pull/54692),[#55112](https://github.com/PaddlePaddle/Paddle/pull/55112),[#55210](https://github.com/PaddlePaddle/Paddle/pull/55210),[#55401](https://github.com/PaddlePaddle/Paddle/pull/55401),[#55772](https://github.com/PaddlePaddle/Paddle/pull/55772),[#55828](https://github.com/PaddlePaddle/Paddle/pull/55828),[#56148](https://github.com/PaddlePaddle/Paddle/pull/56148),[#54763](https://github.com/PaddlePaddle/Paddle/pull/54763),[#56886](https://github.com/PaddlePaddle/Paddle/pull/56886),[#57284](https://github.com/PaddlePaddle/Paddle/pull/57284),[#57268](https://github.com/PaddlePaddle/Paddle/pull/57268),[#57791](https://github.com/PaddlePaddle/Paddle/pull/57791),[#56789](https://github.com/PaddlePaddle/Paddle/pull/56789),[#56704](https://github.com/PaddlePaddle/Paddle/pull/56704),[#57594](https://github.com/PaddlePaddle/Paddle/pull/57594),[#58397](https://github.com/PaddlePaddle/Paddle/pull/58397),[#58337](https://github.com/PaddlePaddle/Paddle/pull/58337),[#58756](https://github.com/PaddlePaddle/Paddle/pull/58756),[#58371](https://github.com/PaddlePaddle/Paddle/pull/58371) -- Reconstructed auto-differentiation module for PIR, migrate and adapted the high-order auto-differentiation function. Optimized Stop Gradient transfer mechanism, so logic is clearer and function is more robust. [#55660](https://github.com/PaddlePaddle/Paddle/pull/55660),[#57084](https://github.com/PaddlePaddle/Paddle/pull/57084),[#56890](https://github.com/PaddlePaddle/Paddle/pull/56890),[#58942](https://github.com/PaddlePaddle/Paddle/pull/58942),[#59373](https://github.com/PaddlePaddle/Paddle/pull/59373),[#57206](https://github.com/PaddlePaddle/Paddle/pull/57206),[#58145](https://github.com/PaddlePaddle/Paddle/pull/58145),[#55235](https://github.com/PaddlePaddle/Paddle/pull/55235),[#57255](https://github.com/PaddlePaddle/Paddle/pull/57255),[#56925](https://github.com/PaddlePaddle/Paddle/pull/56925),[#55957](https://github.com/PaddlePaddle/Paddle/pull/55957),[#56163](https://github.com/PaddlePaddle/Paddle/pull/56163),[#56316](https://github.com/PaddlePaddle/Paddle/pull/56316),[#57294](https://github.com/PaddlePaddle/Paddle/pull/57294),[#57449](https://github.com/PaddlePaddle/Paddle/pull/57449),[#59520](https://github.com/PaddlePaddle/Paddle/pull/59520),[#59565](https://github.com/PaddlePaddle/Paddle/pull/59565),[#56265](https://github.com/PaddlePaddle/Paddle/pull/56265),[#56512](https://github.com/PaddlePaddle/Paddle/pull/56512),[#56650](https://github.com/PaddlePaddle/Paddle/pull/56650),[#57183](https://github.com/PaddlePaddle/Paddle/pull/57183),[#57956](https://github.com/PaddlePaddle/Paddle/pull/57956),[#59100](https://github.com/PaddlePaddle/Paddle/pull/59100) -- Optimized design and representation of control flow forward and reverse operators, introduced ControlFlow Dialect, and supported conversion and execution from control flow operators to PIR under ProgramDesc. [#58729](https://github.com/PaddlePaddle/Paddle/pull/58729),[#57364](https://github.com/PaddlePaddle/Paddle/pull/57364),[#58625](https://github.com/PaddlePaddle/Paddle/pull/58625),[#57475](https://github.com/PaddlePaddle/Paddle/pull/57475),[#57265](https://github.com/PaddlePaddle/Paddle/pull/57265),[#56799](https://github.com/PaddlePaddle/Paddle/pull/56799),[#59033](https://github.com/PaddlePaddle/Paddle/pull/59033),[#57342](https://github.com/PaddlePaddle/Paddle/pull/57342),[#57801](https://github.com/PaddlePaddle/Paddle/pull/57801),[#57958](https://github.com/PaddlePaddle/Paddle/pull/57958),[#57949](https://github.com/PaddlePaddle/Paddle/pull/57949),[#57937](https://github.com/PaddlePaddle/Paddle/pull/57937),[#59231](https://github.com/PaddlePaddle/Paddle/pull/59231),[#59496](https://github.com/PaddlePaddle/Paddle/pull/59496),[#59321](https://github.com/PaddlePaddle/Paddle/pull/59321),[#58088](https://github.com/PaddlePaddle/Paddle/pull/58088),[#58198](https://github.com/PaddlePaddle/Paddle/pull/58198),[#58024](https://github.com/PaddlePaddle/Paddle/pull/58024),[#58089](https://github.com/PaddlePaddle/Paddle/pull/58089),[#58086](https://github.com/PaddlePaddle/Paddle/pull/58086),[#59175](https://github.com/PaddlePaddle/Paddle/pull/59175),[#59423](https://github.com/PaddlePaddle/Paddle/pull/59423),[#59567](https://github.com/PaddlePaddle/Paddle/pull/59567),[#58098](https://github.com/PaddlePaddle/Paddle/pull/58098),[#58163](https://github.com/PaddlePaddle/Paddle/pull/58163),[#58250](https://github.com/PaddlePaddle/Paddle/pull/58250),[#58277](https://github.com/PaddlePaddle/Paddle/pull/58277),[#58355](https://github.com/PaddlePaddle/Paddle/pull/58355),[#59020](https://github.com/PaddlePaddle/Paddle/pull/59020),[#59200](https://github.com/PaddlePaddle/Paddle/pull/59200),[#59585](https://github.com/PaddlePaddle/Paddle/pull/59585),[#58109](https://github.com/PaddlePaddle/Paddle/pull/58109) -- Upgraded dynamic to static execution flow to support PIR, optimized dynamic to static subgraph Pass mechanism, and supported users to try and use functions in the PIR system under the @to_static function. [#57566](https://github.com/PaddlePaddle/Paddle/pull/57566),[#55620](https://github.com/PaddlePaddle/Paddle/pull/55620),[#56791](https://github.com/PaddlePaddle/Paddle/pull/56791),[#57357](https://github.com/PaddlePaddle/Paddle/pull/57357),[#59152](https://github.com/PaddlePaddle/Paddle/pull/59152),[#59312](https://github.com/PaddlePaddle/Paddle/pull/59312),[#58630](https://github.com/PaddlePaddle/Paddle/pull/58630),[#56035](https://github.com/PaddlePaddle/Paddle/pull/56035),[#59447](https://github.com/PaddlePaddle/Paddle/pull/59447),[#57361](https://github.com/PaddlePaddle/Paddle/pull/57361),[#59261](https://github.com/PaddlePaddle/Paddle/pull/59261),[#59774](https://github.com/PaddlePaddle/Paddle/pull/59774) -- Upgraded combination operator function with introducing the concept of Backend to manage logic of combination operator module of dynamic and static graphs in a hierarchical way. Sank necessary components and operator splitting rules into C++, to dramatically reduce maintenance costs. [#58153](https://github.com/PaddlePaddle/Paddle/pull/58153),[#56391](https://github.com/PaddlePaddle/Paddle/pull/56391),[#56614](https://github.com/PaddlePaddle/Paddle/pull/56614),[#57030](https://github.com/PaddlePaddle/Paddle/pull/57030),[#57554](https://github.com/PaddlePaddle/Paddle/pull/57554),[#58018](https://github.com/PaddlePaddle/Paddle/pull/58018),[#58130](https://github.com/PaddlePaddle/Paddle/pull/58130),[#58581](https://github.com/PaddlePaddle/Paddle/pull/58581),[#58679](https://github.com/PaddlePaddle/Paddle/pull/58679),[#59054](https://github.com/PaddlePaddle/Paddle/pull/59054),[#55480](https://github.com/PaddlePaddle/Paddle/pull/55480),[#58451](https://github.com/PaddlePaddle/Paddle/pull/58451),[#55647](https://github.com/PaddlePaddle/Paddle/pull/55647),[#56342](https://github.com/PaddlePaddle/Paddle/pull/56342),[#56798](https://github.com/PaddlePaddle/Paddle/pull/56798),[#57561](https://github.com/PaddlePaddle/Paddle/pull/57561),[#58023](https://github.com/PaddlePaddle/Paddle/pull/58023),[#57722](https://github.com/PaddlePaddle/Paddle/pull/57722) +- Fix errors in `paddle.optimizer.LBFGS` caused by using non-Tensor computations [#60219](https://github.com/PaddlePaddle/Paddle/pull/60219) +- Fix the problem of random numbers not being fixed in `paddle.optimizer.LBFGS` [#60591](https://github.com/PaddlePaddle/Paddle/pull/60591) +- Fix the incorrect calculation of gradient of `set_value` operator [#59034](https://github.com/PaddlePaddle/Paddle/pull/59034) +- Fix the problem of Tensor basic index adapting to PIR [#60259](https://github.com/PaddlePaddle/Paddle/pull/60259), [#61103](https://github.com/PaddlePaddle/Paddle/pull/61103) +- Fix the problem of Tensor combined index assignment [problem](https://github.com/PaddlePaddle/Paddle/issues/60376) [#60447](https://github.com/PaddlePaddle/Paddle/pull/60447) +- Fix the problem when Tensor combined index takes values [problem] [#61922](https://github.com/PaddlePaddle/Paddle/pull/61922) +- Fix `paddle.flatten` stride calculation error issue, with being able to add `paddle.flatten_` [#63084](https://github.com/PaddlePaddle/Paddle/pull/63084) +- Fix the result inconsistency problem between `paddle.index_fill` and `paddle.index_fill_` [#59863](https://github.com/PaddlePaddle/Paddle/pull/59863) +- Fix the `paddle.masked_scatter` error report issue [#60835](https://github.com/PaddlePaddle/Paddle/pull/60835) +- Fix the `paddle.histogramdd` cpu error report issue [#61891](https://github.com/PaddlePaddle/Paddle/pull/61891) +- Fix the bug that `paddle.cast_` continuous use on cpu leads to incorrect result [#60054](https://github.com/PaddlePaddle/Paddle/pull/60054) +- Fix `paddle.put_along_axis` bug when input size is very large [#60551](https://github.com/PaddlePaddle/Paddle/pull/60551) +- Fix `paddle.nanmedian` cpu error report issue [#63221](https://github.com/PaddlePaddle/Paddle/pull/63221) +- Fix the bug that `paddle.median` does not support inputs other than floating-point types in the min branch. [#64444](https://github.com/PaddlePaddle/Paddle/pull/64444) +- Fix the dataloader issue in distributed scenarios. [#62696](https://github.com/PaddlePaddle/Paddle/pull/62696), [#63378](https://github.com/PaddlePaddle/Paddle/pull/63378) +- Fix the formatting issue in error prompt [#63106](https://github.com/PaddlePaddle/Paddle/pull/63106), [#63144](https://github.com/PaddlePaddle/Paddle/pull/63144) +- Fix the format issue under GLOG_v>=6. [#63345](https://github.com/PaddlePaddle/Paddle/pull/63345) -#### Performance optimization +### Security Improvements -- Added PIR Program operators such as DCE and constant_folding_pass, and structure-optimized Pass. [#54935](https://github.com/PaddlePaddle/Paddle/pull/54935),[#59430](https://github.com/PaddlePaddle/Paddle/pull/59430),[#58753](https://github.com/PaddlePaddle/Paddle/pull/58753),[#58732](https://github.com/PaddlePaddle/Paddle/pull/58732) +- Enhance the checking of parent_ids [#62826](https://github.com/PaddlePaddle/Paddle/pull/62826) -2. Added optimization operators fusing class Pass, such as fused_attention, fused_dropout_add, fused_gemm_epilogue_pass, fused_linear_param_grad_add_pass, fused_weight_only_linear_pass, and fused_softmax_mask_upper_triangle, to improve training and inference performance. [#57557](https://github.com/PaddlePaddle/Paddle/pull/57557),[#58272](https://github.com/PaddlePaddle/Paddle/pull/58272),[#58188](https://github.com/PaddlePaddle/Paddle/pull/58188),[#58401](https://github.com/PaddlePaddle/Paddle/pull/58401),[#59366](https://github.com/PaddlePaddle/Paddle/pull/59366),[#57655](https://github.com/PaddlePaddle/Paddle/pull/57655),[#57360](https://github.com/PaddlePaddle/Paddle/pull/57360),[#56672](https://github.com/PaddlePaddle/Paddle/pull/56672),[#58537](https://github.com/PaddlePaddle/Paddle/pull/58537),[#56247](https://github.com/PaddlePaddle/Paddle/pull/56247),[#59391](https://github.com/PaddlePaddle/Paddle/pull/59391),[#58897](https://github.com/PaddlePaddle/Paddle/pull/58897),[#54933](https://github.com/PaddlePaddle/Paddle/pull/54933) +## Basic Execution Architecture -### Dynamic to static capability enhancement +PIR basic functions have been upgraded and improved comprehensively, and the maturity level has been greatly improved. Based on PIR, the design of the PaddlePaddle infrastructure is more reasonable, ensuring the excellent performance and good scalability of the framework. In this version, we have completed the inference verification of PIR in multiple scenarios: For the single-machine scenario, complete the PIR back-end switching in the dynamic-to-static scenarios; For inference scenario, complete the verification of all the stock models, and 84.2% of the models have a gain of 10%+; we have completed the verification of distributed scenarios based on PIR. Meanwhile, based on PIR, we have completed the development and validation of core modules such as control flow, backward logic, save/load, and OneDNN adaptation, which lays a solid foundation for the switching of the PaddlePaddle PIR to the default mode. The functional completeness, execution efficiency and stability of the PaddlePaddle framework operator system are further improved, bringing better use and development experience to the developers. -Dynamic to static graph conversion is a key technology in deep learning frameworks. It allows developers to find the best balance between flexibility and training efficiency. This version of PaddlePaddle has fully upgraded core Dynamic to Static functionality. Success rate of dynamic to static training is up to 100% among 700+ models in PaddlePaddle industry-grade model library. +### Function Optimization -#### New features +- Improve the basic functions of PIR, including basic type system enhancement, debugging, printing, Pass development, and AMP support, to enhance the development efficiency of PIR. [#60723](https://github.com/PaddlePaddle/Paddle/pull/60723), [#60677](https://github.com/PaddlePaddle/Paddle/pull/60677), [#60783](https://github.com/PaddlePaddle/Paddle/pull/60783), [#60798](https://github.com/PaddlePaddle/Paddle/pull/60798), [#61053](https://github.com/PaddlePaddle/Paddle/pull/61053), [#61366](https://github.com/PaddlePaddle/Paddle/pull/61366), [#61446](https://github.com/PaddlePaddle/Paddle/pull/61446), [#60024](https://github.com/PaddlePaddle/Paddle/pull/60024), [#59939](https://github.com/PaddlePaddle/Paddle/pull/59939), [#63376](https://github.com/PaddlePaddle/Paddle/pull/63376), [#61853](https://github.com/PaddlePaddle/Paddle/pull/61853), [#63914](https://github.com/PaddlePaddle/Paddle/pull/63914), [#60170](https://github.com/PaddlePaddle/Paddle/pull/60170), [#60678](https://github.com/PaddlePaddle/Paddle/pull/60678), [#64093](https://github.com/PaddlePaddle/Paddle/pull/64093), [#64065](https://github.com/PaddlePaddle/Paddle/pull/64065), [#62451](https://github.com/PaddlePaddle/Paddle/pull/62451), [#59784](https://github.com/PaddlePaddle/Paddle/pull/59784), [#60136](https://github.com/PaddlePaddle/Paddle/pull/60136), [#63336](https://github.com/PaddlePaddle/Paddle/pull/63336), [#62108](https://github.com/PaddlePaddle/Paddle/pull/62108), [#60860](https://github.com/PaddlePaddle/Paddle/pull/60860), [#60536](https://github.com/PaddlePaddle/Paddle/pull/60536), [#60590](https://github.com/PaddlePaddle/Paddle/pull/60590), [#60752](https://github.com/PaddlePaddle/Paddle/pull/60752), [#61435](https://github.com/PaddlePaddle/Paddle/pull/61435), [#62977](https://github.com/PaddlePaddle/Paddle/pull/62977), [#62139](https://github.com/PaddlePaddle/Paddle/pull/62139), [#60432](https://github.com/PaddlePaddle/Paddle/pull/60432), [#61452](https://github.com/PaddlePaddle/Paddle/pull/61452), [#61978](https://github.com/PaddlePaddle/Paddle/pull/61978), [#62262](https://github.com/PaddlePaddle/Paddle/pull/62262), [#62422](https://github.com/PaddlePaddle/Paddle/pull/62422), [#60359](https://github.com/PaddlePaddle/Paddle/pull/60359), [#62989](https://github.com/PaddlePaddle/Paddle/pull/62989), [#61297](https://github.com/PaddlePaddle/Paddle/pull/61297), [#61399](https://github.com/PaddlePaddle/Paddle/pull/61399), [#61871](https://github.com/PaddlePaddle/Paddle/pull/61871), [#61496](https://github.com/PaddlePaddle/Paddle/pull/61496), [#62413](https://github.com/PaddlePaddle/Paddle/pull/62413) +- Optimize the execution logic of the PaddlePaddle actuator, improve the Pass system, enhance the performance of training and inference, to better support distributed parallel logic operation. [#60182](https://github.com/PaddlePaddle/Paddle/pull/60182), [#60516](https://github.com/PaddlePaddle/Paddle/pull/60516), [#63573](https://github.com/PaddlePaddle/Paddle/pull/63573), [#60181](https://github.com/PaddlePaddle/Paddle/pull/60181), [#59792](https://github.com/PaddlePaddle/Paddle/pull/59792), [#62025](https://github.com/PaddlePaddle/Paddle/pull/62025), [#61160](https://github.com/PaddlePaddle/Paddle/pull/61160), [#61188](https://github.com/PaddlePaddle/Paddle/pull/61188), [#61277](https://github.com/PaddlePaddle/Paddle/pull/61277), [#61669](https://github.com/PaddlePaddle/Paddle/pull/61669), [#60823](https://github.com/PaddlePaddle/Paddle/pull/60823), [#61310](https://github.com/PaddlePaddle/Paddle/pull/61310), [#60892](https://github.com/PaddlePaddle/Paddle/pull/60892), [#60578](https://github.com/PaddlePaddle/Paddle/pull/60578), [#61657](https://github.com/PaddlePaddle/Paddle/pull/61657), [#62638](https://github.com/PaddlePaddle/Paddle/pull/62638), [#63960](https://github.com/PaddlePaddle/Paddle/pull/63960), [#64234](https://github.com/PaddlePaddle/Paddle/pull/64234) -- Adopted Python Eval Frame and VM simulation execution technology to innovatively implement an adaptive Graph Break mechanism. This mechanism is especially designed for control flow scenarios. By introducing the CallLayer mechanism, it makes full use of the advantage of PaddlePaddle dynamic-static unification motion. Support hybrid mode of Abstract Syntax Tree (AST) and bytecode simulation. Efficiently captures control flow operators, thus dramatically improving ability of computational graph to be static. At cache optimization level, fuse advanced optimization technologies such as common sub-expression elimination, to significantly improve execution efficiency of Guard. These optimizations not only reduce redundant computations, but also improve overall system operation speed. To enhance robustness of the system, a simple and efficient data intermediate layer structure is designed. Structure supports correctness recovery of SideEffects, ensuring stability and reliability of system in complex environments. In addition, it is widely compatible with mainstream interpreter versions from Python 3.8 to 3.11, providing users with a wide range of applicability. [#57824](https://github.com/PaddlePaddle/Paddle/pull/57824),[#55887](https://github.com/PaddlePaddle/Paddle/pull/55887),[#58155](https://github.com/PaddlePaddle/Paddle/pull/58155),[#56107](https://github.com/PaddlePaddle/Paddle/pull/56107),[#57490](https://github.com/PaddlePaddle/Paddle/pull/57490),[#58829](https://github.com/PaddlePaddle/Paddle/pull/58829),[#57240](https://github.com/PaddlePaddle/Paddle/pull/57240),[#57588](https://github.com/PaddlePaddle/Paddle/pull/57588),[#58117](https://github.com/PaddlePaddle/Paddle/pull/58117),[#59823](https://github.com/PaddlePaddle/Paddle/pull/59823),[#56077](https://github.com/PaddlePaddle/Paddle/pull/56077),[#58956](https://github.com/PaddlePaddle/Paddle/pull/58956),[#57653](https://github.com/PaddlePaddle/Paddle/pull/57653),[#59855](https://github.com/PaddlePaddle/Paddle/pull/59855),[#59017](https://github.com/PaddlePaddle/Paddle/pull/59017),[#58424](https://github.com/PaddlePaddle/Paddle/pull/58424),[#58187](https://github.com/PaddlePaddle/Paddle/pull/58187),[#57793](https://github.com/PaddlePaddle/Paddle/pull/57793),[#59698](https://github.com/PaddlePaddle/Paddle/pull/59698),[#59747](https://github.com/PaddlePaddle/Paddle/pull/59747),[#59710](https://github.com/PaddlePaddle/Paddle/pull/59710),[#59297](https://github.com/PaddlePaddle/Paddle/pull/59297),[#58423](https://github.com/PaddlePaddle/Paddle/pull/58423),[#56262](https://github.com/PaddlePaddle/Paddle/pull/56262),[#58103](https://github.com/PaddlePaddle/Paddle/pull/58103),[#58538](https://github.com/PaddlePaddle/Paddle/pull/58538),[#58771](https://github.com/PaddlePaddle/Paddle/pull/58771),[#59191](https://github.com/PaddlePaddle/Paddle/pull/59191),[#57754](https://github.com/PaddlePaddle/Paddle/pull/57754),[#59439](https://github.com/PaddlePaddle/Paddle/pull/59439),[#59816](https://github.com/PaddlePaddle/Paddle/pull/59816),[#59035](https://github.com/PaddlePaddle/Paddle/pull/59035) -- Added dynamic to static syntax transcription parsing for PyLayer functions, making PyLayer's conversion between dynamic and static graphs smoother. Users can now seamlessly carry out dynamic to static training on PyLayer, to easily export inference models. [#56108](https://github.com/PaddlePaddle/Paddle/pull/56108),[#56531](https://github.com/PaddlePaddle/Paddle/pull/56531),[#57066](https://github.com/PaddlePaddle/Paddle/pull/57066),[#57633](https://github.com/PaddlePaddle/Paddle/pull/57633) +### PIR New Features -#### Bug Fix +- Realize reverse logic based on PIR, generate reverse computation graph directly, and support higher-order differentiation at the same time. [#60174](https://github.com/PaddlePaddle/Paddle/pull/60174), [#60328](https://github.com/PaddlePaddle/Paddle/pull/60328), [#60818](https://github.com/PaddlePaddle/Paddle/pull/60818), [#61352](https://github.com/PaddlePaddle/Paddle/pull/61352), [#61661](https://github.com/PaddlePaddle/Paddle/pull/61661), [#61927](https://github.com/PaddlePaddle/Paddle/pull/61927), [#62772](https://github.com/PaddlePaddle/Paddle/pull/62772), [#60360](https://github.com/PaddlePaddle/Paddle/pull/60360), [#60866](https://github.com/PaddlePaddle/Paddle/pull/60866), [#60970](https://github.com/PaddlePaddle/Paddle/pull/60970), [#60810](https://github.com/PaddlePaddle/Paddle/pull/60810), [#64696](https://github.com/PaddlePaddle/Paddle/pull/64696), [#59844](https://github.com/PaddlePaddle/Paddle/pull/59844), [#59999](https://github.com/PaddlePaddle/Paddle/pull/59999), [#60262](https://github.com/PaddlePaddle/Paddle/pull/60262), [#60338](https://github.com/PaddlePaddle/Paddle/pull/60338), [#59935](https://github.com/PaddlePaddle/Paddle/pull/59935), [#59982](https://github.com/PaddlePaddle/Paddle/pull/59982), [#60221](https://github.com/PaddlePaddle/Paddle/pull/60221), [#62621](https://github.com/PaddlePaddle/Paddle/pull/62621), [#60044](https://github.com/PaddlePaddle/Paddle/pull/60044), [#59790](https://github.com/PaddlePaddle/Paddle/pull/59790), [#60529](https://github.com/PaddlePaddle/Paddle/pull/60529), [#61378](https://github.com/PaddlePaddle/Paddle/pull/61378), [#61584](https://github.com/PaddlePaddle/Paddle/pull/61584) +- Implement control flow logic based on PIR to improve the expressive ability of PIR and better support multi-scenario services such as training and inference. [#61396](https://github.com/PaddlePaddle/Paddle/pull/61396), [#64045](https://github.com/PaddlePaddle/Paddle/pull/64045), [#60953](https://github.com/PaddlePaddle/Paddle/pull/60953), [#61091](https://github.com/PaddlePaddle/Paddle/pull/61091), [#61304](https://github.com/PaddlePaddle/Paddle/pull/61304), [#62093](https://github.com/PaddlePaddle/Paddle/pull/62093), [#64710](https://github.com/PaddlePaddle/Paddle/pull/64710), [#60668](https://github.com/PaddlePaddle/Paddle/pull/60668), [#60433](https://github.com/PaddlePaddle/Paddle/pull/60433), [#60963](https://github.com/PaddlePaddle/Paddle/pull/60963), [#61192](https://github.com/PaddlePaddle/Paddle/pull/61192), [#60895](https://github.com/PaddlePaddle/Paddle/pull/60895), [#60017](https://github.com/PaddlePaddle/Paddle/pull/60017), [#60369](https://github.com/PaddlePaddle/Paddle/pull/60369), [#60330](https://github.com/PaddlePaddle/Paddle/pull/60330), [#60364](https://github.com/PaddlePaddle/Paddle/pull/60364), [#61416](https://github.com/PaddlePaddle/Paddle/pull/61416), [#60460](https://github.com/PaddlePaddle/Paddle/pull/60460), [#60703](https://github.com/PaddlePaddle/Paddle/pull/60703), [#61027](https://github.com/PaddlePaddle/Paddle/pull/61027) +- Realize save/load logic based on PIR, to carry out the process of PIR and upstream/downstream training and inference services. [#63438](https://github.com/PaddlePaddle/Paddle/pull/63438), [#63574](https://github.com/PaddlePaddle/Paddle/pull/63574), [#64281](https://github.com/PaddlePaddle/Paddle/pull/64281), [#64327](https://github.com/PaddlePaddle/Paddle/pull/64327), [#63622](https://github.com/PaddlePaddle/Paddle/pull/63622), [#64507](https://github.com/PaddlePaddle/Paddle/pull/64507), [#63389](https://github.com/PaddlePaddle/Paddle/pull/63389), [#63539](https://github.com/PaddlePaddle/Paddle/pull/63539), [#63749](https://github.com/PaddlePaddle/Paddle/pull/63749), [#63957](https://github.com/PaddlePaddle/Paddle/pull/63957), [#64044](https://github.com/PaddlePaddle/Paddle/pull/64044), [#64121](https://github.com/PaddlePaddle/Paddle/pull/64121), [#64239](https://github.com/PaddlePaddle/Paddle/pull/64239), [#63818](https://github.com/PaddlePaddle/Paddle/pull/63818), [#63910](https://github.com/PaddlePaddle/Paddle/pull/63910),[#63380](https://github.com/PaddlePaddle/Paddle/pull/63380)[#63380](https://github.com/PaddlePaddle/Paddle/pull/63380),[#63275](https://github.com/PaddlePaddle/Paddle/pull/63275),[#63663](https://github.com/PaddlePaddle/Paddle/pull/63663),[#64692](https://github.com/PaddlePaddle/Paddle/pull/64692),[#63958](https://github.com/PaddlePaddle/Paddle/pull/63958) +- Completed the development and validation of OneDNN related basic functions to prepare for the full-scale switch of OneDNN. [#60680](https://github.com/PaddlePaddle/Paddle/pull/60680), [#60665](https://github.com/PaddlePaddle/Paddle/pull/60665), [#63162](https://github.com/PaddlePaddle/Paddle/pull/63162), [#59917](https://github.com/PaddlePaddle/Paddle/pull/59917), [#62901](https://github.com/PaddlePaddle/Paddle/pull/62901), [#59918](https://github.com/PaddlePaddle/Paddle/pull/59918), [#60257](https://github.com/PaddlePaddle/Paddle/pull/60257), [#60502](https://github.com/PaddlePaddle/Paddle/pull/60502), [#61062](https://github.com/PaddlePaddle/Paddle/pull/61062), [#61170](https://github.com/PaddlePaddle/Paddle/pull/61170), [#61474](https://github.com/PaddlePaddle/Paddle/pull/61474), [#60874](https://github.com/PaddlePaddle/Paddle/pull/60874), [#61495](https://github.com/PaddlePaddle/Paddle/pull/61495), [#61664](https://github.com/PaddlePaddle/Paddle/pull/61664), [#61649](https://github.com/PaddlePaddle/Paddle/pull/61649), [#61592](https://github.com/PaddlePaddle/Paddle/pull/61592), [#61667](https://github.com/PaddlePaddle/Paddle/pull/61667), [#61137](https://github.com/PaddlePaddle/Paddle/pull/61137), [#60952](https://github.com/PaddlePaddle/Paddle/pull/60952), [#61651](https://github.com/PaddlePaddle/Paddle/pull/61651), [#62126](https://github.com/PaddlePaddle/Paddle/pull/62126), [#62187](https://github.com/PaddlePaddle/Paddle/pull/62187), [#61307](https://github.com/PaddlePaddle/Paddle/pull/61307), [#62734](https://github.com/PaddlePaddle/Paddle/pull/62734), [#60974](https://github.com/PaddlePaddle/Paddle/pull/60974), [#61451](https://github.com/PaddlePaddle/Paddle/pull/61451), [#61011](https://github.com/PaddlePaddle/Paddle/pull/61011), [#61218](https://github.com/PaddlePaddle/Paddle/pull/61218), [#61623](https://github.com/PaddlePaddle/Paddle/pull/61623), [#61893](https://github.com/PaddlePaddle/Paddle/pull/61893), [#61876](https://github.com/PaddlePaddle/Paddle/pull/61876), [#61892](https://github.com/PaddlePaddle/Paddle/pull/61892), [#62085](https://github.com/PaddlePaddle/Paddle/pull/62085), [#62220](https://github.com/PaddlePaddle/Paddle/pull/62220), [#62244](https://github.com/PaddlePaddle/Paddle/pull/62244), [#62265](https://github.com/PaddlePaddle/Paddle/pull/62265), [#60754](https://github.com/PaddlePaddle/Paddle/pull/60754), [#60896](https://github.com/PaddlePaddle/Paddle/pull/60896), [#61868](https://github.com/PaddlePaddle/Paddle/pull/61868), [#61659](https://github.com/PaddlePaddle/Paddle/pull/61659), [#62241](https://github.com/PaddlePaddle/Paddle/pull/62241), [#62471](https://github.com/PaddlePaddle/Paddle/pull/62471), [#61165](https://github.com/PaddlePaddle/Paddle/pull/61165),[#64441](https://github.com/PaddlePaddle/Paddle/pull/64441),[#63141](https://github.com/PaddlePaddle/Paddle/pull/63141),[#63145](https://github.com/PaddlePaddle/Paddle/pull/63145),[#63592](https://github.com/PaddlePaddle/Paddle/pull/63592),[#63617](https://github.com/PaddlePaddle/Paddle/pull/63617),[#63518](https://github.com/PaddlePaddle/Paddle/pull/63518),[#63726](https://github.com/PaddlePaddle/Paddle/pull/63726),[#63853](https://github.com/PaddlePaddle/Paddle/pull/63853),[#63812](https://github.com/PaddlePaddle/Paddle/pull/63812),[#63811](https://github.com/PaddlePaddle/Paddle/pull/63811),[#64524](https://github.com/PaddlePaddle/Paddle/pull/64524),[#62993](https://github.com/PaddlePaddle/Paddle/pull/62993),[#63516](https://github.com/PaddlePaddle/Paddle/pull/63516),[#62998](https://github.com/PaddlePaddle/Paddle/pull/62998),[#63151](https://github.com/PaddlePaddle/Paddle/pull/63151),[#64661](https://github.com/PaddlePaddle/Paddle/pull/64661),[#64433](https://github.com/PaddlePaddle/Paddle/pull/64433),[#64448](https://github.com/PaddlePaddle/Paddle/pull/64448),[#63201](https://github.com/PaddlePaddle/Paddle/pull/63201),[#63230](https://github.com/PaddlePaddle/Paddle/pull/63230),[#63233](https://github.com/PaddlePaddle/Paddle/pull/63233),[#63281](https://github.com/PaddlePaddle/Paddle/pull/63281),[#64671](https://github.com/PaddlePaddle/Paddle/pull/64671),[#63274](https://github.com/PaddlePaddle/Paddle/pull/63274) +- Implement Sparse related logic based on PIR, including basic Type and operator expression, and complete the verification of Sparse key functions. [#62868](https://github.com/PaddlePaddle/Paddle/pull/62868), [#63015](https://github.com/PaddlePaddle/Paddle/pull/63015), [#62894](https://github.com/PaddlePaddle/Paddle/pull/62894) -- Fixed the issue that video memory is abnormal in some scenarios of dynamic to static in is_test=True mode. [#58350](https://github.com/PaddlePaddle/Paddle/pull/58350) -- Fixed the issue that function decorated by @to_static is exported to jit.save model in scenarios like foo(x,x,y). [#55963](https://github.com/PaddlePaddle/Paddle/pull/55963) -- Fixed the issue that dynamic and static logic of some API behaviors is not uniform. This improves success rate and user experience of dynamic to static graph conversion. [#56092](https://github.com/PaddlePaddle/Paddle/pull/56092) +### Dynamic-to-static Function Optimization -#### Fixed vulnerability +Optimize the dynamic-to-static basic capability, adapt to the dynamic dimension in SOT training scenarios, and support Python 3.12. -- Fixed a potential security vulnerability in use of eval() in dynamic to static syntax transcription module. [#60100](https://github.com/PaddlePaddle/Paddle/pull/60100) +- Complete the PIR adaptation in dynamic-to-static scenarios. [#60988](https://github.com/PaddlePaddle/Paddle/pull/60988), [#61936](https://github.com/PaddlePaddle/Paddle/pull/61936), [#59929](https://github.com/PaddlePaddle/Paddle/pull/59929), [#61790](https://github.com/PaddlePaddle/Paddle/pull/61790), [#64323](https://github.com/PaddlePaddle/Paddle/pull/64323), [#62030](https://github.com/PaddlePaddle/Paddle/pull/62030), [#61143](https://github.com/PaddlePaddle/Paddle/pull/61143), [#62680](https://github.com/PaddlePaddle/Paddle/pull/62680), [#63309](https://github.com/PaddlePaddle/Paddle/pull/63309), [#63311](https://github.com/PaddlePaddle/Paddle/pull/63311), [#62199](https://github.com/PaddlePaddle/Paddle/pull/62199) +- SOT adapts to Python 3.12 bytecode, and the dynamic-to-static SOT function can be used in Python 3.12. [#61414](https://github.com/PaddlePaddle/Paddle/pull/61414), [#59562](https://github.com/PaddlePaddle/Paddle/pull/59562), [#61031](https://github.com/PaddlePaddle/Paddle/pull/61031), [#61272](https://github.com/PaddlePaddle/Paddle/pull/61272), [#61412](https://github.com/PaddlePaddle/Paddle/pull/61412), [#61305](https://github.com/PaddlePaddle/Paddle/pull/61305), [#61964](https://github.com/PaddlePaddle/Paddle/pull/61964), [#62008](https://github.com/PaddlePaddle/Paddle/pull/62008), [#62028](https://github.com/PaddlePaddle/Paddle/pull/62028), [#61995](https://github.com/PaddlePaddle/Paddle/pull/61995), [#62073](https://github.com/PaddlePaddle/Paddle/pull/62073), [#62120](https://github.com/PaddlePaddle/Paddle/pull/62120), [#62218](https://github.com/PaddlePaddle/Paddle/pull/62218), [#62155](https://github.com/PaddlePaddle/Paddle/pull/62155) +- SOT completes the adaptation of the dynamic dimension of the training scenario, avoiding triggering duplicate graph compositions in dimension changes, and improving the operation efficiency. [#64278](https://github.com/PaddlePaddle/Paddle/pull/64278), [#64435](https://github.com/PaddlePaddle/Paddle/pull/64435), [#64499](https://github.com/PaddlePaddle/Paddle/pull/64499), [#64500](https://github.com/PaddlePaddle/Paddle/pull/64500), [#62080](https://github.com/PaddlePaddle/Paddle/pull/62080) -### Enhanced distributed dynamic graph capability +### Operator Mechanisms -In order to meet the needs of large models, this version focuses on improving the distributed computing capability of the dynamic graph of the PaddlePaddle. Various improvements have been made in communication library, graph analysis, distributed policies and task enable/disable, to provide comprehensive support for large model training. In terms of performance, we further improved training performance by reducing streaming parallel GPU memory occupation, adopting TensorFusion technology, implementing communication computation overlap, and reducing non-essential data synchronization copies. Meanwhile, flexibility of hybrid-parallel debugging is improved through environment variable control Optimizer. In addition, stability of system is further improved by fixing related Bugs. +For the problems of incomplete implementation of some kernels and inefficient calculation logic, we have improved and optimized some of the operator implementation and internal mechanisms of framework, fixed some known problems, and supported some new features. -#### New features +- For XPU kernel, we have optimized the data type support of `numel`, `concat`, and `slice`, and the mixed-precision training support for `AdamW` optimizer. [#63715](https://github.com/PaddlePaddle/Paddle/pull/63715), [#61617](https://github.com/PaddlePaddle/Paddle/pull/61617), [#61694](https://github.com/PaddlePaddle/Paddle/pull/61694), [#64542](https://github.com/PaddlePaddle/Paddle/pull/64542), [#63644](https://github.com/PaddlePaddle/Paddle/pull/63644), [#61340](https://github.com/PaddlePaddle/Paddle/pull/61340), [#63108](https://github.com/PaddlePaddle/Paddle/pull/63108) +- Improve the function and performance of some operators. [#59413](https://github.com/PaddlePaddle/Paddle/pull/59413), [#60295](https://github.com/PaddlePaddle/Paddle/pull/60295), [#64304](https://github.com/PaddlePaddle/Paddle/pull/64304), [#60979](https://github.com/PaddlePaddle/Paddle/pull/60979), [#63556](https://github.com/PaddlePaddle/Paddle/pull/63556), [#63061](https://github.com/PaddlePaddle/Paddle/pull/63061), [#62533](https://github.com/PaddlePaddle/Paddle/pull/62533) +- Improve the mechanism of composite operators, and optimize composite logic for some operators. [#59448](https://github.com/PaddlePaddle/Paddle/pull/59448), [#60505](https://github.com/PaddlePaddle/Paddle/pull/60505), [#59891](https://github.com/PaddlePaddle/Paddle/pull/59891), [#63161](https://github.com/PaddlePaddle/Paddle/pull/63161), [#63245](https://github.com/PaddlePaddle/Paddle/pull/63245), [#63782](https://github.com/PaddlePaddle/Paddle/pull/63782), [#64346](https://github.com/PaddlePaddle/Paddle/pull/64346), [#63156](https://github.com/PaddlePaddle/Paddle/pull/63156), [#63171](https://github.com/PaddlePaddle/Paddle/pull/63171), [#61315](https://github.com/PaddlePaddle/Paddle/pull/61315), [#61701](https://github.com/PaddlePaddle/Paddle/pull/61701), [#61874](https://github.com/PaddlePaddle/Paddle/pull/61874), [#61873](https://github.com/PaddlePaddle/Paddle/pull/61873), [#62059](https://github.com/PaddlePaddle/Paddle/pull/62059), [#61912](https://github.com/PaddlePaddle/Paddle/pull/61912), [#62112](https://github.com/PaddlePaddle/Paddle/pull/62112), [#63011](https://github.com/PaddlePaddle/Paddle/pull/63011), [#63009](https://github.com/PaddlePaddle/Paddle/pull/63009), [#64714](https://github.com/PaddlePaddle/Paddle/pull/64714) -- Added TraceHang function in communication library, to quickly locate the faulty node when cluster training has Hang problem. [#59217](https://github.com/PaddlePaddle/Paddle/pull/59217) -- In order to improve training efficiency and reduce memory, dynamic graph supports stride mechanism. [#55156](https://github.com/PaddlePaddle/Paddle/pull/55156),[#54762](https://github.com/PaddlePaddle/Paddle/pull/54762),[#55850](https://github.com/PaddlePaddle/Paddle/pull/55850),[#59190](https://github.com/PaddlePaddle/Paddle/pull/59190),[#57005](https://github.com/PaddlePaddle/Paddle/pull/57005),[#57005](https://github.com/PaddlePaddle/Paddle/pull/57005),[#57331](https://github.com/PaddlePaddle/Paddle/pull/57331),[#58033](https://github.com/PaddlePaddle/Paddle/pull/58033),[#58033](https://github.com/PaddlePaddle/Paddle/pull/58033),[#58303](https://github.com/PaddlePaddle/Paddle/pull/58303),[#57835](https://github.com/PaddlePaddle/Paddle/pull/57835),[#57189](https://github.com/PaddlePaddle/Paddle/pull/57189) -- Enhanced paddleviz function to facilitate analysis of computational graphs. [#56837](https://github.com/PaddlePaddle/Paddle/pull/56837),[#57626](https://github.com/PaddlePaddle/Paddle/pull/57626) -- In distributed Sharding strategies (Stage1,2,3), added main_grad function to support higher precision gradient accumulation, and reduce precision loss caused by low precision accumulation. [#57972](https://github.com/PaddlePaddle/Paddle/pull/57972),[#57934](https://github.com/PaddlePaddle/Paddle/pull/57934),[#57473](https://github.com/PaddlePaddle/Paddle/pull/57473),[#57537](https://github.com/PaddlePaddle/Paddle/pull/57537),[#59611](https://github.com/PaddlePaddle/Paddle/pull/59611),[#57960](https://github.com/PaddlePaddle/Paddle/pull/57960) -- In Sharding Stage1 strategy, added a switch variable to control whether to perform fusion calculation on Optimizer. [#58790](https://github.com/PaddlePaddle/Paddle/pull/58790) -- In Recompute function, added support for Tuple input parameters, enhancing calling ability of Recompute interface. [#56793](https://github.com/PaddlePaddle/Paddle/pull/56793) -- Enhanced Launch function, allowing distributed training without specifying endpoints in dynamic graphs. [#54636](https://github.com/PaddlePaddle/Paddle/pull/54636) +### Bug Fixing -#### Function optimization +- Fix the bugs related to PIR, actuator, and dynamic-to-static. [#64442](https://github.com/PaddlePaddle/Paddle/pull/64442), [#60443](https://github.com/PaddlePaddle/Paddle/pull/60443), [#60122](https://github.com/PaddlePaddle/Paddle/pull/60122), [#60625](https://github.com/PaddlePaddle/Paddle/pull/60625), [#60607](https://github.com/PaddlePaddle/Paddle/pull/60607), [#60705](https://github.com/PaddlePaddle/Paddle/pull/60705), [#61110](https://github.com/PaddlePaddle/Paddle/pull/61110), [#61278](https://github.com/PaddlePaddle/Paddle/pull/61278), [#61448](https://github.com/PaddlePaddle/Paddle/pull/61448), [#61491](https://github.com/PaddlePaddle/Paddle/pull/61491), [#61692](https://github.com/PaddlePaddle/Paddle/pull/61692), [#62100](https://github.com/PaddlePaddle/Paddle/pull/62100), [#62239](https://github.com/PaddlePaddle/Paddle/pull/62239), [#62365](https://github.com/PaddlePaddle/Paddle/pull/62365), [#62758](https://github.com/PaddlePaddle/Paddle/pull/62758), [#63395](https://github.com/PaddlePaddle/Paddle/pull/63395), [#64272](https://github.com/PaddlePaddle/Paddle/pull/64272), [#62165](https://github.com/PaddlePaddle/Paddle/pull/62165), [#64151](https://github.com/PaddlePaddle/Paddle/pull/64151), [#64204](https://github.com/PaddlePaddle/Paddle/pull/64204), [#64815](https://github.com/PaddlePaddle/Paddle/pull/64815), [#63757](https://github.com/PaddlePaddle/Paddle/pull/63757), [#61972](https://github.com/PaddlePaddle/Paddle/pull/61972), [#64806](https://github.com/PaddlePaddle/Paddle/pull/64806), [#60010](https://github.com/PaddlePaddle/Paddle/pull/60010), [#60461](https://github.com/PaddlePaddle/Paddle/pull/60461), [#60310](https://github.com/PaddlePaddle/Paddle/pull/60310), [#62006](https://github.com/PaddlePaddle/Paddle/pull/62006), [#61591](https://github.com/PaddlePaddle/Paddle/pull/61591), [#60327](https://github.com/PaddlePaddle/Paddle/pull/60327), [#60720](https://github.com/PaddlePaddle/Paddle/pull/60720), [#64656](https://github.com/PaddlePaddle/Paddle/pull/64656), [#60236](https://github.com/PaddlePaddle/Paddle/pull/60236), [#60684](https://github.com/PaddlePaddle/Paddle/pull/60684), [#60790](https://github.com/PaddlePaddle/Paddle/pull/60790), [#60944](https://github.com/PaddlePaddle/Paddle/pull/60944), [#62056](https://github.com/PaddlePaddle/Paddle/pull/62056), [#62891](https://github.com/PaddlePaddle/Paddle/pull/62891), [#64676](https://github.com/PaddlePaddle/Paddle/pull/64676), [#60271](https://github.com/PaddlePaddle/Paddle/pull/60271), [#60634](https://github.com/PaddlePaddle/Paddle/pull/60634), [#60663](https://github.com/PaddlePaddle/Paddle/pull/60663), [#60827](https://github.com/PaddlePaddle/Paddle/pull/60827), [#60845](https://github.com/PaddlePaddle/Paddle/pull/60845), [#60905](https://github.com/PaddlePaddle/Paddle/pull/60905), [#60945](https://github.com/PaddlePaddle/Paddle/pull/60945), [#60949](https://github.com/PaddlePaddle/Paddle/pull/60949), [#61107](https://github.com/PaddlePaddle/Paddle/pull/61107), [#61111](https://github.com/PaddlePaddle/Paddle/pull/61111), [#61117](https://github.com/PaddlePaddle/Paddle/pull/61117), [#61158](https://github.com/PaddlePaddle/Paddle/pull/61158), [#61177](https://github.com/PaddlePaddle/Paddle/pull/61177), [#61355](https://github.com/PaddlePaddle/Paddle/pull/61355), [#61593](https://github.com/PaddlePaddle/Paddle/pull/61593), [#61666](https://github.com/PaddlePaddle/Paddle/pull/61666), [#61934](https://github.com/PaddlePaddle/Paddle/pull/61934), [#62216](https://github.com/PaddlePaddle/Paddle/pull/62216), [#62491](https://github.com/PaddlePaddle/Paddle/pull/62491), [#62515](https://github.com/PaddlePaddle/Paddle/pull/62515), [#62594](https://github.com/PaddlePaddle/Paddle/pull/62594), [#62605](https://github.com/PaddlePaddle/Paddle/pull/62605), [#62895](https://github.com/PaddlePaddle/Paddle/pull/62895), [#62913](https://github.com/PaddlePaddle/Paddle/pull/62913), [#64413](https://github.com/PaddlePaddle/Paddle/pull/64413), [#59947](https://github.com/PaddlePaddle/Paddle/pull/59947), [#60264](https://github.com/PaddlePaddle/Paddle/pull/60264), [#60721](https://github.com/PaddlePaddle/Paddle/pull/60721), [#63113](https://github.com/PaddlePaddle/Paddle/pull/63113), [#63629](https://github.com/PaddlePaddle/Paddle/pull/63629), [#64300](https://github.com/PaddlePaddle/Paddle/pull/64300), [#64450](https://github.com/PaddlePaddle/Paddle/pull/64450), [#64532](https://github.com/PaddlePaddle/Paddle/pull/64532), [#64561](https://github.com/PaddlePaddle/Paddle/pull/64561), [#64625](https://github.com/PaddlePaddle/Paddle/pull/64625), [#64731](https://github.com/PaddlePaddle/Paddle/pull/64731), [#60059](https://github.com/PaddlePaddle/Paddle/pull/60059), [#60487](https://github.com/PaddlePaddle/Paddle/pull/60487), [#60423](https://github.com/PaddlePaddle/Paddle/pull/60423), [#61599](https://github.com/PaddlePaddle/Paddle/pull/61599), [#62032](https://github.com/PaddlePaddle/Paddle/pull/62032), [#62686](https://github.com/PaddlePaddle/Paddle/pull/62686), [#64055](https://github.com/PaddlePaddle/Paddle/pull/64055), [#60751](https://github.com/PaddlePaddle/Paddle/pull/60751), [#61646](https://github.com/PaddlePaddle/Paddle/pull/61646), [#60454](https://github.com/PaddlePaddle/Paddle/pull/60454), [#62530](https://github.com/PaddlePaddle/Paddle/pull/62530), [#62821](https://github.com/PaddlePaddle/Paddle/pull/62821), [#64454](https://github.com/PaddlePaddle/Paddle/pull/64454), [#64754](https://github.com/PaddlePaddle/Paddle/pull/64754), [#59860](https://github.com/PaddlePaddle/Paddle/pull/59860), [#60280](https://github.com/PaddlePaddle/Paddle/pull/60280), [#60357](https://github.com/PaddlePaddle/Paddle/pull/60357), [#60363](https://github.com/PaddlePaddle/Paddle/pull/60363), [#60900](https://github.com/PaddlePaddle/Paddle/pull/60900), [#61185](https://github.com/PaddlePaddle/Paddle/pull/61185), [#61505](https://github.com/PaddlePaddle/Paddle/pull/61505), [#61644](https://github.com/PaddlePaddle/Paddle/pull/61644), [#62256](https://github.com/PaddlePaddle/Paddle/pull/62256), [#62396](https://github.com/PaddlePaddle/Paddle/pull/62396), [#63040](https://github.com/PaddlePaddle/Paddle/pull/63040), [#63409](https://github.com/PaddlePaddle/Paddle/pull/63409), [#63764](https://github.com/PaddlePaddle/Paddle/pull/63764), [#59571](https://github.com/PaddlePaddle/Paddle/pull/59571), [#59894](https://github.com/PaddlePaddle/Paddle/pull/59894), [#59569](https://github.com/PaddlePaddle/Paddle/pull/59569), [#59896](https://github.com/PaddlePaddle/Paddle/pull/59896), [#60015](https://github.com/PaddlePaddle/Paddle/pull/60015), [#60081](https://github.com/PaddlePaddle/Paddle/pull/60081), [#60164](https://github.com/PaddlePaddle/Paddle/pull/60164), [#60200](https://github.com/PaddlePaddle/Paddle/pull/60200), [#60211](https://github.com/PaddlePaddle/Paddle/pull/60211), [#60267](https://github.com/PaddlePaddle/Paddle/pull/60267), [#60458](https://github.com/PaddlePaddle/Paddle/pull/60458), [#60395](https://github.com/PaddlePaddle/Paddle/pull/60395), [#60907](https://github.com/PaddlePaddle/Paddle/pull/60907), [#60707](https://github.com/PaddlePaddle/Paddle/pull/60707), [#60993](https://github.com/PaddlePaddle/Paddle/pull/60993), [#61401](https://github.com/PaddlePaddle/Paddle/pull/61401), [#61433](https://github.com/PaddlePaddle/Paddle/pull/61433), [#61450](https://github.com/PaddlePaddle/Paddle/pull/61450), [#61577](https://github.com/PaddlePaddle/Paddle/pull/61577), [#61575](https://github.com/PaddlePaddle/Paddle/pull/61575), [#61703](https://github.com/PaddlePaddle/Paddle/pull/61703), [#61711](https://github.com/PaddlePaddle/Paddle/pull/61711), [#61883](https://github.com/PaddlePaddle/Paddle/pull/61883), [#61822](https://github.com/PaddlePaddle/Paddle/pull/61822), [#62012](https://github.com/PaddlePaddle/Paddle/pull/62012), [#61858](https://github.com/PaddlePaddle/Paddle/pull/61858), [#62176](https://github.com/PaddlePaddle/Paddle/pull/62176), [#62257](https://github.com/PaddlePaddle/Paddle/pull/62257), [#62470](https://github.com/PaddlePaddle/Paddle/pull/62470), [#62536](https://github.com/PaddlePaddle/Paddle/pull/62536), [#62606](https://github.com/PaddlePaddle/Paddle/pull/62606), [#62808](https://github.com/PaddlePaddle/Paddle/pull/62808), [#62854](https://github.com/PaddlePaddle/Paddle/pull/62854), [#62879](https://github.com/PaddlePaddle/Paddle/pull/62879), [#62864](https://github.com/PaddlePaddle/Paddle/pull/62864), [#63063](https://github.com/PaddlePaddle/Paddle/pull/63063), [#62958](https://github.com/PaddlePaddle/Paddle/pull/62958), [#63397](https://github.com/PaddlePaddle/Paddle/pull/63397), [#63805](https://github.com/PaddlePaddle/Paddle/pull/63805), [#63694](https://github.com/PaddlePaddle/Paddle/pull/63694), [#64168](https://github.com/PaddlePaddle/Paddle/pull/64168), [#64184](https://github.com/PaddlePaddle/Paddle/pull/64184), [#64174](https://github.com/PaddlePaddle/Paddle/pull/64174), [#64315](https://github.com/PaddlePaddle/Paddle/pull/64315), [#64362](https://github.com/PaddlePaddle/Paddle/pull/64362), [#64400](https://github.com/PaddlePaddle/Paddle/pull/64400), [#64475](https://github.com/PaddlePaddle/Paddle/pull/64475), [#64458](https://github.com/PaddlePaddle/Paddle/pull/64458), [#64548](https://github.com/PaddlePaddle/Paddle/pull/64548), [#59858](https://github.com/PaddlePaddle/Paddle/pull/59858), [#61132](https://github.com/PaddlePaddle/Paddle/pull/61132), [#62010](https://github.com/PaddlePaddle/Paddle/pull/62010), [#62069](https://github.com/PaddlePaddle/Paddle/pull/62069), [#62707](https://github.com/PaddlePaddle/Paddle/pull/62707), [#62921](https://github.com/PaddlePaddle/Paddle/pull/62921), [#63085](https://github.com/PaddlePaddle/Paddle/pull/63085), [#63321](https://github.com/PaddlePaddle/Paddle/pull/63321), [#63351](https://github.com/PaddlePaddle/Paddle/pull/63351), [#63549](https://github.com/PaddlePaddle/Paddle/pull/63549), [#64567](https://github.com/PaddlePaddle/Paddle/pull/64567), [#59936](https://github.com/PaddlePaddle/Paddle/pull/59936), [#60269](https://github.com/PaddlePaddle/Paddle/pull/60269), [#60879](https://github.com/PaddlePaddle/Paddle/pull/60879), [#61314](https://github.com/PaddlePaddle/Paddle/pull/61314), [#61391](https://github.com/PaddlePaddle/Paddle/pull/61391), [#61479](https://github.com/PaddlePaddle/Paddle/pull/61479), [#61789](https://github.com/PaddlePaddle/Paddle/pull/61789), [#61832](https://github.com/PaddlePaddle/Paddle/pull/61832), [#61864](https://github.com/PaddlePaddle/Paddle/pull/61864), [#61917](https://github.com/PaddlePaddle/Paddle/pull/61917), [#62052](https://github.com/PaddlePaddle/Paddle/pull/62052), [#62068](https://github.com/PaddlePaddle/Paddle/pull/62068), [#62293](https://github.com/PaddlePaddle/Paddle/pull/62293), [#62479](https://github.com/PaddlePaddle/Paddle/pull/62479), [#62506](https://github.com/PaddlePaddle/Paddle/pull/62506), [#59948](https://github.com/PaddlePaddle/Paddle/pull/59948), [#64118](https://github.com/PaddlePaddle/Paddle/pull/64118), [#64126](https://github.com/PaddlePaddle/Paddle/pull/64126), [#64195](https://github.com/PaddlePaddle/Paddle/pull/64195), [#64307](https://github.com/PaddlePaddle/Paddle/pull/64307), [#64314](https://github.com/PaddlePaddle/Paddle/pull/64314), [#64276](https://github.com/PaddlePaddle/Paddle/pull/64276), [#64312](https://github.com/PaddlePaddle/Paddle/pull/64312), [#64350](https://github.com/PaddlePaddle/Paddle/pull/64350), [#64319](https://github.com/PaddlePaddle/Paddle/pull/64319), [#64463](https://github.com/PaddlePaddle/Paddle/pull/64463), [#64457](https://github.com/PaddlePaddle/Paddle/pull/64457), [#64455](https://github.com/PaddlePaddle/Paddle/pull/64455), [#64487](https://github.com/PaddlePaddle/Paddle/pull/64487), [#64645](https://github.com/PaddlePaddle/Paddle/pull/64645), [#63155](https://github.com/PaddlePaddle/Paddle/pull/63155), [#59893](https://github.com/PaddlePaddle/Paddle/pull/59893), [#63332](https://github.com/PaddlePaddle/Paddle/pull/63332), [#63332](https://github.com/PaddlePaddle/Paddle/pull/63332), [#64786](https://github.com/PaddlePaddle/Paddle/pull/64786), [#60515](https://github.com/PaddlePaddle/Paddle/pull/60515), [#60627](https://github.com/PaddlePaddle/Paddle/pull/60627), [#60863](https://github.com/PaddlePaddle/Paddle/pull/60863), [#60854](https://github.com/PaddlePaddle/Paddle/pull/60854), [#61447](https://github.com/PaddlePaddle/Paddle/pull/61447), [#61440](https://github.com/PaddlePaddle/Paddle/pull/61440), [#61932](https://github.com/PaddlePaddle/Paddle/pull/61932), [#62131](https://github.com/PaddlePaddle/Paddle/pull/62131), [#62252](https://github.com/PaddlePaddle/Paddle/pull/62252), [#62283](https://github.com/PaddlePaddle/Paddle/pull/62283), [#62358](https://github.com/PaddlePaddle/Paddle/pull/62358), [#62411](https://github.com/PaddlePaddle/Paddle/pull/62411), [#62424](https://github.com/PaddlePaddle/Paddle/pull/62424), [#62810](https://github.com/PaddlePaddle/Paddle/pull/62810), [#62811](https://github.com/PaddlePaddle/Paddle/pull/62811), [#62896](https://github.com/PaddlePaddle/Paddle/pull/62896), [#62947](https://github.com/PaddlePaddle/Paddle/pull/62947), [#63182](https://github.com/PaddlePaddle/Paddle/pull/63182), [#63190](https://github.com/PaddlePaddle/Paddle/pull/63190), [#63294](https://github.com/PaddlePaddle/Paddle/pull/63294), [#63306](https://github.com/PaddlePaddle/Paddle/pull/63306), [#63352](https://github.com/PaddlePaddle/Paddle/pull/63352), [#63404](https://github.com/PaddlePaddle/Paddle/pull/63404), [#63474](https://github.com/PaddlePaddle/Paddle/pull/63474), [#64013](https://github.com/PaddlePaddle/Paddle/pull/64013), [#64674](https://github.com/PaddlePaddle/Paddle/pull/64674),[#60055](https://github.com/PaddlePaddle/Paddle/pull/60055),[#62050](https://github.com/PaddlePaddle/Paddle/pull/62050),[#62770](https://github.com/PaddlePaddle/Paddle/pull/62770),[#63234](https://github.com/PaddlePaddle/Paddle/pull/63234),[#63374](https://github.com/PaddlePaddle/Paddle/pull/63374),[#64277](https://github.com/PaddlePaddle/Paddle/pull/64277), [#63420](https://github.com/PaddlePaddle/Paddle/pull/63420), [#60312](https://github.com/PaddlePaddle/Paddle/pull/60312), [#63810](https://github.com/PaddlePaddle/Paddle/pull/63810), [#64631](https://github.com/PaddlePaddle/Paddle/pull/64631), [#63970](https://github.com/PaddlePaddle/Paddle/pull/63970), [#63708](https://github.com/PaddlePaddle/Paddle/pull/63708), [#62062](https://github.com/PaddlePaddle/Paddle/pull/62062), [#60898](https://github.com/PaddlePaddle/Paddle/pull/60898), [#62373](https://github.com/PaddlePaddle/Paddle/pull/62373), [#59878](https://github.com/PaddlePaddle/Paddle/pull/59878) +- Fix some bugs in operator mechanism, operator implementation logic and related unit tests. [#63792](https://github.com/PaddlePaddle/Paddle/pull/63792), [#60570](https://github.com/PaddlePaddle/Paddle/pull/60570), [#61572](https://github.com/PaddlePaddle/Paddle/pull/61572), [#59971](https://github.com/PaddlePaddle/Paddle/pull/59971), [#61336](https://github.com/PaddlePaddle/Paddle/pull/61336), [#63276](https://github.com/PaddlePaddle/Paddle/pull/63276), [#63251](https://github.com/PaddlePaddle/Paddle/pull/63251), [#63697](https://github.com/PaddlePaddle/Paddle/pull/63697), [#63706](https://github.com/PaddlePaddle/Paddle/pull/63706), [#64685](https://github.com/PaddlePaddle/Paddle/pull/64685), [#64009](https://github.com/PaddlePaddle/Paddle/pull/64009), [#62461](https://github.com/PaddlePaddle/Paddle/pull/62461), [#61568](https://github.com/PaddlePaddle/Paddle/pull/61568), [#63912](https://github.com/PaddlePaddle/Paddle/pull/63912), [#60475](https://github.com/PaddlePaddle/Paddle/pull/60475), [#60222](https://github.com/PaddlePaddle/Paddle/pull/60222), [#63961](https://github.com/PaddlePaddle/Paddle/pull/63961), [#63593](https://github.com/PaddlePaddle/Paddle/pull/63593) -- Implemented new communication library with dynamic-static unification. Communication operators are fully adapted to PHI operator system, reducing development and maintenance costs to better support dynamic graphs and auto parallel architecture upgrade. [#54417](https://github.com/PaddlePaddle/Paddle/pull/54417),[#57768](https://github.com/PaddlePaddle/Paddle/pull/57768),[#57897](https://github.com/PaddlePaddle/Paddle/pull/57897),[#55537](https://github.com/PaddlePaddle/Paddle/pull/55537),[#56604](https://github.com/PaddlePaddle/Paddle/pull/56604),[#57519](https://github.com/PaddlePaddle/Paddle/pull/57519),[#56088](https://github.com/PaddlePaddle/Paddle/pull/56088),[#57153](https://github.com/PaddlePaddle/Paddle/pull/57153),[#57161](https://github.com/PaddlePaddle/Paddle/pull/57161),[#57252](https://github.com/PaddlePaddle/Paddle/pull/57252),[#57251](https://github.com/PaddlePaddle/Paddle/pull/57251),[#57208](https://github.com/PaddlePaddle/Paddle/pull/57208),[#57305](https://github.com/PaddlePaddle/Paddle/pull/57305),[#57424](https://github.com/PaddlePaddle/Paddle/pull/57424),[#57548](https://github.com/PaddlePaddle/Paddle/pull/57548),[#57560](https://github.com/PaddlePaddle/Paddle/pull/57560),[#57564](https://github.com/PaddlePaddle/Paddle/pull/57564),[#57233](https://github.com/PaddlePaddle/Paddle/pull/57233),[#55726](https://github.com/PaddlePaddle/Paddle/pull/55726),[#58073](https://github.com/PaddlePaddle/Paddle/pull/58073) -- TCPStore is changed to a single instance to support dynamic graphs and auto parallel more flexibly. [#55956](https://github.com/PaddlePaddle/Paddle/pull/55956) -- Improved maintainability and flexibility of distributed policies such as MP/PP/SP, including addition of printing warning and error messages, structural cleanup of code files, and optimization of PP restrictions on inputs. [#54448](https://github.com/PaddlePaddle/Paddle/pull/54448),[#59762](https://github.com/PaddlePaddle/Paddle/pull/59762),[#55462](https://github.com/PaddlePaddle/Paddle/pull/55462),[#54788](https://github.com/PaddlePaddle/Paddle/pull/54788),[#54664](https://github.com/PaddlePaddle/Paddle/pull/54664),[#56456](https://github.com/PaddlePaddle/Paddle/pull/56456),[#55540](https://github.com/PaddlePaddle/Paddle/pull/55540) -- In PP strategy, added support for P2P communication in computation flow, making communication mode more flexible. [#54747](https://github.com/PaddlePaddle/Paddle/pull/54747) -- Sharding strategy supports reduce Operation on gradient. [#58842](https://github.com/PaddlePaddle/Paddle/pull/58842),[#57967](https://github.com/PaddlePaddle/Paddle/pull/57967),[#55495](https://github.com/PaddlePaddle/Paddle/pull/55495) +### Developer Content -#### Performance optimization +- Developer related contents include PIR switching, unit test start, function verification and other PR. [#60621](https://github.com/PaddlePaddle/Paddle/pull/60621), [#59703](https://github.com/PaddlePaddle/Paddle/pull/59703), [#59694](https://github.com/PaddlePaddle/Paddle/pull/59694), [#59717](https://github.com/PaddlePaddle/Paddle/pull/59717), [#59729](https://github.com/PaddlePaddle/Paddle/pull/59729), [#59730](https://github.com/PaddlePaddle/Paddle/pull/59730), [#60216](https://github.com/PaddlePaddle/Paddle/pull/60216), [#60238](https://github.com/PaddlePaddle/Paddle/pull/60238), [#60246](https://github.com/PaddlePaddle/Paddle/pull/60246), [#60343](https://github.com/PaddlePaddle/Paddle/pull/60343), [#60302](https://github.com/PaddlePaddle/Paddle/pull/60302), [#60870](https://github.com/PaddlePaddle/Paddle/pull/60870), [#59956](https://github.com/PaddlePaddle/Paddle/pull/59956), [#60795](https://github.com/PaddlePaddle/Paddle/pull/60795), [#62528](https://github.com/PaddlePaddle/Paddle/pull/62528), [#59932](https://github.com/PaddlePaddle/Paddle/pull/59932), [#59636](https://github.com/PaddlePaddle/Paddle/pull/59636), [#59959](https://github.com/PaddlePaddle/Paddle/pull/59959), [#59734](https://github.com/PaddlePaddle/Paddle/pull/59734), [#60287](https://github.com/PaddlePaddle/Paddle/pull/60287), [#60347](https://github.com/PaddlePaddle/Paddle/pull/60347), [#60335](https://github.com/PaddlePaddle/Paddle/pull/60335), [#60332](https://github.com/PaddlePaddle/Paddle/pull/60332), [#59631](https://github.com/PaddlePaddle/Paddle/pull/59631), [#60255](https://github.com/PaddlePaddle/Paddle/pull/60255), [#60329](https://github.com/PaddlePaddle/Paddle/pull/60329), [#60401](https://github.com/PaddlePaddle/Paddle/pull/60401), [#60522](https://github.com/PaddlePaddle/Paddle/pull/60522), [#60792](https://github.com/PaddlePaddle/Paddle/pull/60792), [#59617](https://github.com/PaddlePaddle/Paddle/pull/59617), [#60277](https://github.com/PaddlePaddle/Paddle/pull/60277), [#60584](https://github.com/PaddlePaddle/Paddle/pull/60584), [#60911](https://github.com/PaddlePaddle/Paddle/pull/60911), [#61322](https://github.com/PaddlePaddle/Paddle/pull/61322), [#60838](https://github.com/PaddlePaddle/Paddle/pull/60838), [#60602](https://github.com/PaddlePaddle/Paddle/pull/60602), [#61458](https://github.com/PaddlePaddle/Paddle/pull/61458), [#61607](https://github.com/PaddlePaddle/Paddle/pull/61607), [#61960](https://github.com/PaddlePaddle/Paddle/pull/61960), [#60484](https://github.com/PaddlePaddle/Paddle/pull/60484), [#61662](https://github.com/PaddlePaddle/Paddle/pull/61662), [#62263](https://github.com/PaddlePaddle/Paddle/pull/62263), [#62270](https://github.com/PaddlePaddle/Paddle/pull/62270), [#62469](https://github.com/PaddlePaddle/Paddle/pull/62469), [#62416](https://github.com/PaddlePaddle/Paddle/pull/62416), [#62443](https://github.com/PaddlePaddle/Paddle/pull/62443), [#62412](https://github.com/PaddlePaddle/Paddle/pull/62412), [#62541](https://github.com/PaddlePaddle/Paddle/pull/62541), [#62634](https://github.com/PaddlePaddle/Paddle/pull/62634), [#62369](https://github.com/PaddlePaddle/Paddle/pull/62369), [#60805](https://github.com/PaddlePaddle/Paddle/pull/60805), [#62644](https://github.com/PaddlePaddle/Paddle/pull/62644), [#62494](https://github.com/PaddlePaddle/Paddle/pull/62494), [#62767](https://github.com/PaddlePaddle/Paddle/pull/62767), [#62735](https://github.com/PaddlePaddle/Paddle/pull/62735), [#62802](https://github.com/PaddlePaddle/Paddle/pull/62802), [#62801](https://github.com/PaddlePaddle/Paddle/pull/62801), [#62783](https://github.com/PaddlePaddle/Paddle/pull/62783), [#62579](https://github.com/PaddlePaddle/Paddle/pull/62579), [#62833](https://github.com/PaddlePaddle/Paddle/pull/62833), [#62668](https://github.com/PaddlePaddle/Paddle/pull/62668), [#62972](https://github.com/PaddlePaddle/Paddle/pull/62972), [#62505](https://github.com/PaddlePaddle/Paddle/pull/62505), [#63005](https://github.com/PaddlePaddle/Paddle/pull/63005), [#62900](https://github.com/PaddlePaddle/Paddle/pull/62900), [#60577](https://github.com/PaddlePaddle/Paddle/pull/60577), [#60877](https://github.com/PaddlePaddle/Paddle/pull/60877), [#61076](https://github.com/PaddlePaddle/Paddle/pull/61076), [#61038](https://github.com/PaddlePaddle/Paddle/pull/61038), [#61112](https://github.com/PaddlePaddle/Paddle/pull/61112), [#61120](https://github.com/PaddlePaddle/Paddle/pull/61120), [#61582](https://github.com/PaddlePaddle/Paddle/pull/61582), [#61119](https://github.com/PaddlePaddle/Paddle/pull/61119), [#61036](https://github.com/PaddlePaddle/Paddle/pull/61036), [#61289](https://github.com/PaddlePaddle/Paddle/pull/61289), [#60695](https://github.com/PaddlePaddle/Paddle/pull/60695), [#61039](https://github.com/PaddlePaddle/Paddle/pull/61039), [#61963](https://github.com/PaddlePaddle/Paddle/pull/61963), [#62118](https://github.com/PaddlePaddle/Paddle/pull/62118), [#62797](https://github.com/PaddlePaddle/Paddle/pull/62797), [#62807](https://github.com/PaddlePaddle/Paddle/pull/62807), [#62887](https://github.com/PaddlePaddle/Paddle/pull/62887), [#62830](https://github.com/PaddlePaddle/Paddle/pull/62830), [#62849](https://github.com/PaddlePaddle/Paddle/pull/62849), [#62750](https://github.com/PaddlePaddle/Paddle/pull/62750), [#62965](https://github.com/PaddlePaddle/Paddle/pull/62965), [#59742](https://github.com/PaddlePaddle/Paddle/pull/59742), [#59867](https://github.com/PaddlePaddle/Paddle/pull/59867), [#60836](https://github.com/PaddlePaddle/Paddle/pull/60836), [#60902](https://github.com/PaddlePaddle/Paddle/pull/60902), [#61228](https://github.com/PaddlePaddle/Paddle/pull/61228), [#60037](https://github.com/PaddlePaddle/Paddle/pull/60037), [#60079](https://github.com/PaddlePaddle/Paddle/pull/60079), [#60173](https://github.com/PaddlePaddle/Paddle/pull/60173), [#60373](https://github.com/PaddlePaddle/Paddle/pull/60373), [#60380](https://github.com/PaddlePaddle/Paddle/pull/60380), [#60381](https://github.com/PaddlePaddle/Paddle/pull/60381), [#60750](https://github.com/PaddlePaddle/Paddle/pull/60750), [#61065](https://github.com/PaddlePaddle/Paddle/pull/61065), [#61122](https://github.com/PaddlePaddle/Paddle/pull/61122), [#61074](https://github.com/PaddlePaddle/Paddle/pull/61074), [#61204](https://github.com/PaddlePaddle/Paddle/pull/61204), [#61191](https://github.com/PaddlePaddle/Paddle/pull/61191), [#61182](https://github.com/PaddlePaddle/Paddle/pull/61182), [#61219](https://github.com/PaddlePaddle/Paddle/pull/61219), [#61296](https://github.com/PaddlePaddle/Paddle/pull/61296), [#61503](https://github.com/PaddlePaddle/Paddle/pull/61503), [#61484](https://github.com/PaddlePaddle/Paddle/pull/61484), [#61513](https://github.com/PaddlePaddle/Paddle/pull/61513), [#61476](https://github.com/PaddlePaddle/Paddle/pull/61476), [#61510](https://github.com/PaddlePaddle/Paddle/pull/61510), [#61511](https://github.com/PaddlePaddle/Paddle/pull/61511), [#61526](https://github.com/PaddlePaddle/Paddle/pull/61526), [#61524](https://github.com/PaddlePaddle/Paddle/pull/61524), [#61525](https://github.com/PaddlePaddle/Paddle/pull/61525), [#61466](https://github.com/PaddlePaddle/Paddle/pull/61466), [#61497](https://github.com/PaddlePaddle/Paddle/pull/61497), [#61538](https://github.com/PaddlePaddle/Paddle/pull/61538), [#61533](https://github.com/PaddlePaddle/Paddle/pull/61533), [#61530](https://github.com/PaddlePaddle/Paddle/pull/61530), [#61468](https://github.com/PaddlePaddle/Paddle/pull/61468), [#61527](https://github.com/PaddlePaddle/Paddle/pull/61527), [#61535](https://github.com/PaddlePaddle/Paddle/pull/61535), [#61512](https://github.com/PaddlePaddle/Paddle/pull/61512), [#61531](https://github.com/PaddlePaddle/Paddle/pull/61531), [#61539](https://github.com/PaddlePaddle/Paddle/pull/61539), [#61532](https://github.com/PaddlePaddle/Paddle/pull/61532), [#61521](https://github.com/PaddlePaddle/Paddle/pull/61521), [#61517](https://github.com/PaddlePaddle/Paddle/pull/61517), [#61518](https://github.com/PaddlePaddle/Paddle/pull/61518), [#61550](https://github.com/PaddlePaddle/Paddle/pull/61550), [#61545](https://github.com/PaddlePaddle/Paddle/pull/61545), [#61548](https://github.com/PaddlePaddle/Paddle/pull/61548), [#61519](https://github.com/PaddlePaddle/Paddle/pull/61519), [#61549](https://github.com/PaddlePaddle/Paddle/pull/61549), [#61574](https://github.com/PaddlePaddle/Paddle/pull/61574), [#61585](https://github.com/PaddlePaddle/Paddle/pull/61585), [#61581](https://github.com/PaddlePaddle/Paddle/pull/61581), [#61553](https://github.com/PaddlePaddle/Paddle/pull/61553), [#61504](https://github.com/PaddlePaddle/Paddle/pull/61504), [#61603](https://github.com/PaddlePaddle/Paddle/pull/61603), [#61534](https://github.com/PaddlePaddle/Paddle/pull/61534), [#61567](https://github.com/PaddlePaddle/Paddle/pull/61567), [#61523](https://github.com/PaddlePaddle/Paddle/pull/61523), [#61565](https://github.com/PaddlePaddle/Paddle/pull/61565), [#61564](https://github.com/PaddlePaddle/Paddle/pull/61564), [#61707](https://github.com/PaddlePaddle/Paddle/pull/61707), [#61560](https://github.com/PaddlePaddle/Paddle/pull/61560), [#61684](https://github.com/PaddlePaddle/Paddle/pull/61684), [#61706](https://github.com/PaddlePaddle/Paddle/pull/61706), [#61724](https://github.com/PaddlePaddle/Paddle/pull/61724), [#61719](https://github.com/PaddlePaddle/Paddle/pull/61719), [#61729](https://github.com/PaddlePaddle/Paddle/pull/61729), [#61763](https://github.com/PaddlePaddle/Paddle/pull/61763), [#61755](https://github.com/PaddlePaddle/Paddle/pull/61755), [#61737](https://github.com/PaddlePaddle/Paddle/pull/61737), [#61750](https://github.com/PaddlePaddle/Paddle/pull/61750), [#61753](https://github.com/PaddlePaddle/Paddle/pull/61753), [#61756](https://github.com/PaddlePaddle/Paddle/pull/61756), [#61777](https://github.com/PaddlePaddle/Paddle/pull/61777), [#61758](https://github.com/PaddlePaddle/Paddle/pull/61758), [#61731](https://github.com/PaddlePaddle/Paddle/pull/61731), [#61771](https://github.com/PaddlePaddle/Paddle/pull/61771), [#61739](https://github.com/PaddlePaddle/Paddle/pull/61739), [#61559](https://github.com/PaddlePaddle/Paddle/pull/61559), [#61717](https://github.com/PaddlePaddle/Paddle/pull/61717), [#61733](https://github.com/PaddlePaddle/Paddle/pull/61733), [#61563](https://github.com/PaddlePaddle/Paddle/pull/61563), [#61546](https://github.com/PaddlePaddle/Paddle/pull/61546), [#61566](https://github.com/PaddlePaddle/Paddle/pull/61566), [#61562](https://github.com/PaddlePaddle/Paddle/pull/61562), [#61793](https://github.com/PaddlePaddle/Paddle/pull/61793), [#61902](https://github.com/PaddlePaddle/Paddle/pull/61902), [#61905](https://github.com/PaddlePaddle/Paddle/pull/61905), [#61904](https://github.com/PaddlePaddle/Paddle/pull/61904), [#62227](https://github.com/PaddlePaddle/Paddle/pull/62227), [#62332](https://github.com/PaddlePaddle/Paddle/pull/62332), [#62653](https://github.com/PaddlePaddle/Paddle/pull/62653), [#62681](https://github.com/PaddlePaddle/Paddle/pull/62681), [#62709](https://github.com/PaddlePaddle/Paddle/pull/62709), [#62794](https://github.com/PaddlePaddle/Paddle/pull/62794), [#62938](https://github.com/PaddlePaddle/Paddle/pull/62938), [#63185](https://github.com/PaddlePaddle/Paddle/pull/63185), [#63754](https://github.com/PaddlePaddle/Paddle/pull/63754), [#63769](https://github.com/PaddlePaddle/Paddle/pull/63769), [#63793](https://github.com/PaddlePaddle/Paddle/pull/63793), [#63830](https://github.com/PaddlePaddle/Paddle/pull/63830), [#63939](https://github.com/PaddlePaddle/Paddle/pull/63939), [#64340](https://github.com/PaddlePaddle/Paddle/pull/64340), [#64657](https://github.com/PaddlePaddle/Paddle/pull/64657), [#62527](https://github.com/PaddlePaddle/Paddle/pull/62527), [#64088](https://github.com/PaddlePaddle/Paddle/pull/64088), [#60203](https://github.com/PaddlePaddle/Paddle/pull/60203), [#60372](https://github.com/PaddlePaddle/Paddle/pull/60372), [#60685](https://github.com/PaddlePaddle/Paddle/pull/60685), [#60815](https://github.com/PaddlePaddle/Paddle/pull/60815), [#60791](https://github.com/PaddlePaddle/Paddle/pull/60791), [#60864](https://github.com/PaddlePaddle/Paddle/pull/60864), [#60851](https://github.com/PaddlePaddle/Paddle/pull/60851), [#60844](https://github.com/PaddlePaddle/Paddle/pull/60844), [#60694](https://github.com/PaddlePaddle/Paddle/pull/60694), [#60855](https://github.com/PaddlePaddle/Paddle/pull/60855), [#60869](https://github.com/PaddlePaddle/Paddle/pull/60869), [#60948](https://github.com/PaddlePaddle/Paddle/pull/60948), [#61042](https://github.com/PaddlePaddle/Paddle/pull/61042), [#61455](https://github.com/PaddlePaddle/Paddle/pull/61455), [#61580](https://github.com/PaddlePaddle/Paddle/pull/61580), [#61589](https://github.com/PaddlePaddle/Paddle/pull/61589), [#61609](https://github.com/PaddlePaddle/Paddle/pull/61609), [#61616](https://github.com/PaddlePaddle/Paddle/pull/61616), [#61715](https://github.com/PaddlePaddle/Paddle/pull/61715), [#61716](https://github.com/PaddlePaddle/Paddle/pull/61716), [#61759](https://github.com/PaddlePaddle/Paddle/pull/61759), [#61555](https://github.com/PaddlePaddle/Paddle/pull/61555), [#61492](https://github.com/PaddlePaddle/Paddle/pull/61492), [#61805](https://github.com/PaddlePaddle/Paddle/pull/61805), [#61712](https://github.com/PaddlePaddle/Paddle/pull/61712), [#61615](https://github.com/PaddlePaddle/Paddle/pull/61615), [#61713](https://github.com/PaddlePaddle/Paddle/pull/61713), [#62129](https://github.com/PaddlePaddle/Paddle/pull/62129), [#59294](https://github.com/PaddlePaddle/Paddle/pull/59294), [#59865](https://github.com/PaddlePaddle/Paddle/pull/59865), [#60270](https://github.com/PaddlePaddle/Paddle/pull/60270), [#60547](https://github.com/PaddlePaddle/Paddle/pull/60547), [#60698](https://github.com/PaddlePaddle/Paddle/pull/60698), [#60762](https://github.com/PaddlePaddle/Paddle/pull/60762), [#60753](https://github.com/PaddlePaddle/Paddle/pull/60753), [#60966](https://github.com/PaddlePaddle/Paddle/pull/60966), [#60976](https://github.com/PaddlePaddle/Paddle/pull/60976), [#61100](https://github.com/PaddlePaddle/Paddle/pull/61100), [#61203](https://github.com/PaddlePaddle/Paddle/pull/61203), [#61210](https://github.com/PaddlePaddle/Paddle/pull/61210), [#61424](https://github.com/PaddlePaddle/Paddle/pull/61424), [#61213](https://github.com/PaddlePaddle/Paddle/pull/61213), [#61275](https://github.com/PaddlePaddle/Paddle/pull/61275), [#61276](https://github.com/PaddlePaddle/Paddle/pull/61276), [#61279](https://github.com/PaddlePaddle/Paddle/pull/61279), [#61292](https://github.com/PaddlePaddle/Paddle/pull/61292), [#61295](https://github.com/PaddlePaddle/Paddle/pull/61295), [#61298](https://github.com/PaddlePaddle/Paddle/pull/61298), [#61299](https://github.com/PaddlePaddle/Paddle/pull/61299), [#61301](https://github.com/PaddlePaddle/Paddle/pull/61301), [#61302](https://github.com/PaddlePaddle/Paddle/pull/61302), [#61329](https://github.com/PaddlePaddle/Paddle/pull/61329), [#61804](https://github.com/PaddlePaddle/Paddle/pull/61804), [#62745](https://github.com/PaddlePaddle/Paddle/pull/62745), [#62909](https://github.com/PaddlePaddle/Paddle/pull/62909), [#64247](https://github.com/PaddlePaddle/Paddle/pull/64247), [#64308](https://github.com/PaddlePaddle/Paddle/pull/64308), [#60690](https://github.com/PaddlePaddle/Paddle/pull/60690), [#61149](https://github.com/PaddlePaddle/Paddle/pull/61149), [#61145](https://github.com/PaddlePaddle/Paddle/pull/61145), [#61193](https://github.com/PaddlePaddle/Paddle/pull/61193), [#61207](https://github.com/PaddlePaddle/Paddle/pull/61207), [#61229](https://github.com/PaddlePaddle/Paddle/pull/61229), [#61236](https://github.com/PaddlePaddle/Paddle/pull/61236), [#61244](https://github.com/PaddlePaddle/Paddle/pull/61244), [#61242](https://github.com/PaddlePaddle/Paddle/pull/61242), [#61263](https://github.com/PaddlePaddle/Paddle/pull/61263), [#61370](https://github.com/PaddlePaddle/Paddle/pull/61370), [#61410](https://github.com/PaddlePaddle/Paddle/pull/61410), [#61480](https://github.com/PaddlePaddle/Paddle/pull/61480), [#61522](https://github.com/PaddlePaddle/Paddle/pull/61522), [#61540](https://github.com/PaddlePaddle/Paddle/pull/61540), [#61520](https://github.com/PaddlePaddle/Paddle/pull/61520), [#61625](https://github.com/PaddlePaddle/Paddle/pull/61625), [#61700](https://github.com/PaddlePaddle/Paddle/pull/61700), [#61708](https://github.com/PaddlePaddle/Paddle/pull/61708), [#61736](https://github.com/PaddlePaddle/Paddle/pull/61736), [#61889](https://github.com/PaddlePaddle/Paddle/pull/61889), [#61952](https://github.com/PaddlePaddle/Paddle/pull/61952), [#62033](https://github.com/PaddlePaddle/Paddle/pull/62033), [#62637](https://github.com/PaddlePaddle/Paddle/pull/62637), [#62777](https://github.com/PaddlePaddle/Paddle/pull/62777), [#62779](https://github.com/PaddlePaddle/Paddle/pull/62779), [#63226](https://github.com/PaddlePaddle/Paddle/pull/63226), [#63287](https://github.com/PaddlePaddle/Paddle/pull/63287), [#63398](https://github.com/PaddlePaddle/Paddle/pull/63398), [#63431](https://github.com/PaddlePaddle/Paddle/pull/63431), [#64000](https://github.com/PaddlePaddle/Paddle/pull/64000), [#64058](https://github.com/PaddlePaddle/Paddle/pull/64058), [#64059](https://github.com/PaddlePaddle/Paddle/pull/64059), [#64063](https://github.com/PaddlePaddle/Paddle/pull/64063), [#64066](https://github.com/PaddlePaddle/Paddle/pull/64066), [#64089](https://github.com/PaddlePaddle/Paddle/pull/64089), [#64170](https://github.com/PaddlePaddle/Paddle/pull/64170), [#64235](https://github.com/PaddlePaddle/Paddle/pull/64235), [#64237](https://github.com/PaddlePaddle/Paddle/pull/64237), [#64243](https://github.com/PaddlePaddle/Paddle/pull/64243), [#64242](https://github.com/PaddlePaddle/Paddle/pull/64242), [#64286](https://github.com/PaddlePaddle/Paddle/pull/64286), [#64322](https://github.com/PaddlePaddle/Paddle/pull/64322), [#64317](https://github.com/PaddlePaddle/Paddle/pull/64317), [#64490](https://github.com/PaddlePaddle/Paddle/pull/64490), [#60138](https://github.com/PaddlePaddle/Paddle/pull/60138), [#62384](https://github.com/PaddlePaddle/Paddle/pull/62384), [#59702](https://github.com/PaddlePaddle/Paddle/pull/59702), [#60341](https://github.com/PaddlePaddle/Paddle/pull/60341), [#60636](https://github.com/PaddlePaddle/Paddle/pull/60636), [#60714](https://github.com/PaddlePaddle/Paddle/pull/60714), [#60716](https://github.com/PaddlePaddle/Paddle/pull/60716), [#60700](https://github.com/PaddlePaddle/Paddle/pull/60700), [#60702](https://github.com/PaddlePaddle/Paddle/pull/60702), [#60704](https://github.com/PaddlePaddle/Paddle/pull/60704), [#60715](https://github.com/PaddlePaddle/Paddle/pull/60715), [#60713](https://github.com/PaddlePaddle/Paddle/pull/60713), [#60711](https://github.com/PaddlePaddle/Paddle/pull/60711), [#60724](https://github.com/PaddlePaddle/Paddle/pull/60724), [#60803](https://github.com/PaddlePaddle/Paddle/pull/60803), [#61331](https://github.com/PaddlePaddle/Paddle/pull/61331), [#63286](https://github.com/PaddlePaddle/Paddle/pull/63286), [#60473](https://github.com/PaddlePaddle/Paddle/pull/60473), [#61046](https://github.com/PaddlePaddle/Paddle/pull/61046), [#61859](https://github.com/PaddlePaddle/Paddle/pull/61859), [#60675](https://github.com/PaddlePaddle/Paddle/pull/60675), [#60719](https://github.com/PaddlePaddle/Paddle/pull/60719), [#62863](https://github.com/PaddlePaddle/Paddle/pull/62863), [#63013](https://github.com/PaddlePaddle/Paddle/pull/63013), [#61293](https://github.com/PaddlePaddle/Paddle/pull/61293), [#62781](https://github.com/PaddlePaddle/Paddle/pull/62781), [#62935](https://github.com/PaddlePaddle/Paddle/pull/62935), [#63014](https://github.com/PaddlePaddle/Paddle/pull/63014), [#64203](https://github.com/PaddlePaddle/Paddle/pull/64203), [#63349](https://github.com/PaddlePaddle/Paddle/pull/63349), [#59572](https://github.com/PaddlePaddle/Paddle/pull/59572), [#59911](https://github.com/PaddlePaddle/Paddle/pull/59911), [#59861](https://github.com/PaddlePaddle/Paddle/pull/59861), [#60014](https://github.com/PaddlePaddle/Paddle/pull/60014), [#59913](https://github.com/PaddlePaddle/Paddle/pull/59913), [#58889](https://github.com/PaddlePaddle/Paddle/pull/58889), [#60114](https://github.com/PaddlePaddle/Paddle/pull/60114), [#59928](https://github.com/PaddlePaddle/Paddle/pull/59928), [#60180](https://github.com/PaddlePaddle/Paddle/pull/60180), [#60168](https://github.com/PaddlePaddle/Paddle/pull/60168), [#60166](https://github.com/PaddlePaddle/Paddle/pull/60166), [#60250](https://github.com/PaddlePaddle/Paddle/pull/60250), [#60247](https://github.com/PaddlePaddle/Paddle/pull/60247), [#60172](https://github.com/PaddlePaddle/Paddle/pull/60172), [#59661](https://github.com/PaddlePaddle/Paddle/pull/59661), [#58880](https://github.com/PaddlePaddle/Paddle/pull/58880), [#60291](https://github.com/PaddlePaddle/Paddle/pull/60291), [#58881](https://github.com/PaddlePaddle/Paddle/pull/58881), [#58955](https://github.com/PaddlePaddle/Paddle/pull/58955), [#58684](https://github.com/PaddlePaddle/Paddle/pull/58684), [#58708](https://github.com/PaddlePaddle/Paddle/pull/58708), [#60323](https://github.com/PaddlePaddle/Paddle/pull/60323), [#58762](https://github.com/PaddlePaddle/Paddle/pull/58762), [#60048](https://github.com/PaddlePaddle/Paddle/pull/60048), [#60345](https://github.com/PaddlePaddle/Paddle/pull/60345), [#60325](https://github.com/PaddlePaddle/Paddle/pull/60325), [#59627](https://github.com/PaddlePaddle/Paddle/pull/59627), [#60416](https://github.com/PaddlePaddle/Paddle/pull/60416), [#60434](https://github.com/PaddlePaddle/Paddle/pull/60434), [#59801](https://github.com/PaddlePaddle/Paddle/pull/59801), [#60619](https://github.com/PaddlePaddle/Paddle/pull/60619), [#60445](https://github.com/PaddlePaddle/Paddle/pull/60445), [#60666](https://github.com/PaddlePaddle/Paddle/pull/60666), [#60353](https://github.com/PaddlePaddle/Paddle/pull/60353), [#60733](https://github.com/PaddlePaddle/Paddle/pull/60733), [#60693](https://github.com/PaddlePaddle/Paddle/pull/60693), [#60350](https://github.com/PaddlePaddle/Paddle/pull/60350), [#61096](https://github.com/PaddlePaddle/Paddle/pull/61096), [#61121](https://github.com/PaddlePaddle/Paddle/pull/61121), [#61164](https://github.com/PaddlePaddle/Paddle/pull/61164), [#62054](https://github.com/PaddlePaddle/Paddle/pull/62054), [#62136](https://github.com/PaddlePaddle/Paddle/pull/62136), [#62508](https://github.com/PaddlePaddle/Paddle/pull/62508), [#62988](https://github.com/PaddlePaddle/Paddle/pull/62988), [#63472](https://github.com/PaddlePaddle/Paddle/pull/63472), [#60193](https://github.com/PaddlePaddle/Paddle/pull/60193), [#60197](https://github.com/PaddlePaddle/Paddle/pull/60197), [#60198](https://github.com/PaddlePaddle/Paddle/pull/60198), [#60346](https://github.com/PaddlePaddle/Paddle/pull/60346), [#60318](https://github.com/PaddlePaddle/Paddle/pull/60318), [#60645](https://github.com/PaddlePaddle/Paddle/pull/60645), [#60650](https://github.com/PaddlePaddle/Paddle/pull/60650), [#60660](https://github.com/PaddlePaddle/Paddle/pull/60660), [#60706](https://github.com/PaddlePaddle/Paddle/pull/60706), [#60799](https://github.com/PaddlePaddle/Paddle/pull/60799), [#60837](https://github.com/PaddlePaddle/Paddle/pull/60837), [#60817](https://github.com/PaddlePaddle/Paddle/pull/60817), [#60820](https://github.com/PaddlePaddle/Paddle/pull/60820), [#60894](https://github.com/PaddlePaddle/Paddle/pull/60894), [#61079](https://github.com/PaddlePaddle/Paddle/pull/61079), [#61087](https://github.com/PaddlePaddle/Paddle/pull/61087), [#61073](https://github.com/PaddlePaddle/Paddle/pull/61073), [#61072](https://github.com/PaddlePaddle/Paddle/pull/61072), [#61127](https://github.com/PaddlePaddle/Paddle/pull/61127), [#61097](https://github.com/PaddlePaddle/Paddle/pull/61097), [#61365](https://github.com/PaddlePaddle/Paddle/pull/61365), [#61456](https://github.com/PaddlePaddle/Paddle/pull/61456), [#61846](https://github.com/PaddlePaddle/Paddle/pull/61846), [#62217](https://github.com/PaddlePaddle/Paddle/pull/62217), [#62519](https://github.com/PaddlePaddle/Paddle/pull/62519), [#62881](https://github.com/PaddlePaddle/Paddle/pull/62881), [#62880](https://github.com/PaddlePaddle/Paddle/pull/62880), [#59723](https://github.com/PaddlePaddle/Paddle/pull/59723), [#59722](https://github.com/PaddlePaddle/Paddle/pull/59722), [#59797](https://github.com/PaddlePaddle/Paddle/pull/59797), [#59960](https://github.com/PaddlePaddle/Paddle/pull/59960), [#59761](https://github.com/PaddlePaddle/Paddle/pull/59761), [#59996](https://github.com/PaddlePaddle/Paddle/pull/59996), [#60009](https://github.com/PaddlePaddle/Paddle/pull/60009), [#58896](https://github.com/PaddlePaddle/Paddle/pull/58896), [#60051](https://github.com/PaddlePaddle/Paddle/pull/60051), [#60410](https://github.com/PaddlePaddle/Paddle/pull/60410), [#60420](https://github.com/PaddlePaddle/Paddle/pull/60420), [#60548](https://github.com/PaddlePaddle/Paddle/pull/60548), [#60575](https://github.com/PaddlePaddle/Paddle/pull/60575), [#60726](https://github.com/PaddlePaddle/Paddle/pull/60726), [#60809](https://github.com/PaddlePaddle/Paddle/pull/60809), [#61346](https://github.com/PaddlePaddle/Paddle/pull/61346), [#61222](https://github.com/PaddlePaddle/Paddle/pull/61222), [#61099](https://github.com/PaddlePaddle/Paddle/pull/61099), [#62254](https://github.com/PaddlePaddle/Paddle/pull/62254), [#62269](https://github.com/PaddlePaddle/Paddle/pull/62269), [#62362](https://github.com/PaddlePaddle/Paddle/pull/62362) +- Improve the underlying error checking mechanism of PaddlePaddle to facilitate developers' debugging. [#62571](https://github.com/PaddlePaddle/Paddle/pull/62571), [#62602](https://github.com/PaddlePaddle/Paddle/pull/62602), [#60903](https://github.com/PaddlePaddle/Paddle/pull/60903), [#64695](https://github.com/PaddlePaddle/Paddle/pull/64695), [#59907](https://github.com/PaddlePaddle/Paddle/pull/59907), [#62018](https://github.com/PaddlePaddle/Paddle/pull/62018), [#62839](https://github.com/PaddlePaddle/Paddle/pull/62839), [#60651](https://github.com/PaddlePaddle/Paddle/pull/60651), [#61488](https://github.com/PaddlePaddle/Paddle/pull/61488), [#64064](https://github.com/PaddlePaddle/Paddle/pull/64064), [#63192](https://github.com/PaddlePaddle/Paddle/pull/63192), [#63525](https://github.com/PaddlePaddle/Paddle/pull/63525)。 -- Implemented timely release of last layer of PP strategy, to save video memory. [#54505](https://github.com/PaddlePaddle/Paddle/pull/54505) -- In MP strategy Tensor fusion, supported incoming params group to enhance Tensor fusion function. Improved allreduce asynchronous communication performance, and enhanced training performance through overlap of computation and communication. [#57690](https://github.com/PaddlePaddle/Paddle/pull/57690),[#55662](https://github.com/PaddlePaddle/Paddle/pull/55662) -- In Sharding strategy, carried out overlap for reverse computation and gradient communication, to improve training performance. For Sharding stage1, added Tensor fusion and fuse grad clip, and optimizer, to improve computational efficiency. Supported overlap between VPP and DP/Sharding Stage1, to improve communication and computation parallelism. Optimized performance of Sharding Stage1 under FP16. Check only gradient responsible for this sharding rank in the check finite stage, to reduce computation overhead; added environment variables to control whether Optimize is performed to save video memory, to achieve use of fewer resources for model training debugging. [#55598](https://github.com/PaddlePaddle/Paddle/pull/55598),[#55427](https://github.com/PaddlePaddle/Paddle/pull/55427),[#56063](https://github.com/PaddlePaddle/Paddle/pull/56063),[#55766](https://github.com/PaddlePaddle/Paddle/pull/55766),[#59848](https://github.com/PaddlePaddle/Paddle/pull/59848) -- In Hybrid Parallel strategy, arranged Tensor fusion under PP/VPP to pre-run, to solve the problem of extra overhead of runtime fuse on video memory. Improved model training performance by reducing non-essential synchronous memcpy. [#54403](https://github.com/PaddlePaddle/Paddle/pull/54403),[#57215](https://github.com/PaddlePaddle/Paddle/pull/57215) +### Vulnerability Fixing -#### Bug Fix +- Fix potential security vulnerabilities. [#59957](https://github.com/PaddlePaddle/Paddle/pull/59957), [#61032](https://github.com/PaddlePaddle/Paddle/pull/61032), [#61356](https://github.com/PaddlePaddle/Paddle/pull/61356), [#61573](https://github.com/PaddlePaddle/Paddle/pull/61573), [#61671](https://github.com/PaddlePaddle/Paddle/pull/61671), [#62345](https://github.com/PaddlePaddle/Paddle/pull/62345), [#60097](https://github.com/PaddlePaddle/Paddle/pull/60097), [#61161](https://github.com/PaddlePaddle/Paddle/pull/61161), [#61294](https://github.com/PaddlePaddle/Paddle/pull/61294), [#61349](https://github.com/PaddlePaddle/Paddle/pull/61349), [#61344](https://github.com/PaddlePaddle/Paddle/pull/61344), [#61162](https://github.com/PaddlePaddle/Paddle/pull/61162), [#61285](https://github.com/PaddlePaddle/Paddle/pull/61285), [#61826](https://github.com/PaddlePaddle/Paddle/pull/61826), [#59967](https://github.com/PaddlePaddle/Paddle/pull/59967), [#59976](https://github.com/PaddlePaddle/Paddle/pull/59976), [#59979](https://github.com/PaddlePaddle/Paddle/pull/59979)[#60527](https://github.com/PaddlePaddle/Paddle/pull/60527),[#60646](https://github.com/PaddlePaddle/Paddle/pull/60646),[#61827](https://github.com/PaddlePaddle/Paddle/pull/61827) -- Fixed 13 bugs in PP, Launch function, MP strategy, and fuse_rope, to enhance stability of distributed strategies. At mechanism level, fixed errors of inplace and tensor reference to improve stability. [#55116](https://github.com/PaddlePaddle/Paddle/pull/55116),[#55782](https://github.com/PaddlePaddle/Paddle/pull/55782),[#59609](https://github.com/PaddlePaddle/Paddle/pull/59609),[#57394](https://github.com/PaddlePaddle/Paddle/pull/57394),[#55864](https://github.com/PaddlePaddle/Paddle/pull/55864),[#58482](https://github.com/PaddlePaddle/Paddle/pull/58482),[#54571](https://github.com/PaddlePaddle/Paddle/pull/54571),[#55896](https://github.com/PaddlePaddle/Paddle/pull/55896),[#54648](https://github.com/PaddlePaddle/Paddle/pull/54648),[#58307](https://github.com/PaddlePaddle/Paddle/pull/58307),[#55679](https://github.com/PaddlePaddle/Paddle/pull/55679),[#58133](https://github.com/PaddlePaddle/Paddle/pull/58133),[#58408](https://github.com/PaddlePaddle/Paddle/pull/58408),[#59707](https://github.com/PaddlePaddle/Paddle/pull/59707),[#55342](https://github.com/PaddlePaddle/Paddle/pull/55342),[#54703](https://github.com/PaddlePaddle/Paddle/pull/54703),[#54869](https://github.com/PaddlePaddle/Paddle/pull/54869),[#55568](https://github.com/PaddlePaddle/Paddle/pull/55568),[#55233](https://github.com/PaddlePaddle/Paddle/pull/55233),[#56418](https://github.com/PaddlePaddle/Paddle/pull/56418),[#56428](https://github.com/PaddlePaddle/Paddle/pull/56428),[#56892](https://github.com/PaddlePaddle/Paddle/pull/56892),[#57192](https://github.com/PaddlePaddle/Paddle/pull/57192),[#59161](https://github.com/PaddlePaddle/Paddle/pull/59161),[#59340](https://github.com/PaddlePaddle/Paddle/pull/59340),[#57006](https://github.com/PaddlePaddle/Paddle/pull/57006),[#57353](https://github.com/PaddlePaddle/Paddle/pull/57353),[#57352](https://github.com/PaddlePaddle/Paddle/pull/57352),[#59088](https://github.com/PaddlePaddle/Paddle/pull/59088) -- Fixed bug that PP strategy can't release single-layer output in time. Fixed the bug that initialization process may Hang. [#54624](https://github.com/PaddlePaddle/Paddle/pull/54624),[#58844](https://github.com/PaddlePaddle/Paddle/pull/58844),[#54673](https://github.com/PaddlePaddle/Paddle/pull/54673),[#58376](https://github.com/PaddlePaddle/Paddle/pull/58376) -- Fixed the bug calculation is wrong when input data type is not uniform under MP strategy. Fixed the bug of parameter synchronization under MP strategy. Fixed the bug user input config is not used correctly. [#58858](https://github.com/PaddlePaddle/Paddle/pull/58858),[#57918](https://github.com/PaddlePaddle/Paddle/pull/57918),[#58037](https://github.com/PaddlePaddle/Paddle/pull/58037) -- Unified judgment method of dygraph and dynamic mode. [#54633](https://github.com/PaddlePaddle/Paddle/pull/54633) -- Fixed the bug shape of sin and cos in fuse_rope is not correct. [#56132](https://github.com/PaddlePaddle/Paddle/pull/56132) -- Fixed the bug task fails to due to long endpoints in Luanch distributed scenarios. Fixed the bug endpoints may be out of order. [#55011](https://github.com/PaddlePaddle/Paddle/pull/55011),[#55478](https://github.com/PaddlePaddle/Paddle/pull/55478) -- Fixed the bug MEA function may cause segmentation fault error. [#55408](https://github.com/PaddlePaddle/Paddle/pull/55408) +### Deprecated Features -### Auto parallel +- Clean up deprecated actuators and other logic to reduce redundant codes. [#64822](https://github.com/PaddlePaddle/Paddle/pull/64822), [#60941](https://github.com/PaddlePaddle/Paddle/pull/60941) -This release fully optimizes Auto Parallel programming paradigm with dynamic-static unification to simplify programming complexity for developers. Developers do not need to understand complex concepts and APIs in manual parallel programming paradigm, such as row-parallel, column-parallel, and so on. A small amount of tensor distribution annotations is required to build a hybrid parallel model. Framework will handle the derivation of distribution states of all tensors and operators, and adding appropriate communication operators. Meanwhile, it supports the dynamic to static distributed training by just one extra code changed, enabling developers to efficiently and easily implement any hybrid parallel strategy. This can significantly reduce development costs of hybrid parallel training codes. +## Compiler Infrastructure for Neural Networks (CINN) -#### Improved auto parallel core functions +In version 3.0, the compiler architecture has been significantly upgraded. Based on Shape Dialect, build a symbolic automatic derivation and simplification system, support symbolic expression and constraint construction, and support end-to-end execution under the dynamic shape of the compiler. Meanwhile, CINN has upgraded the automatic fusion of subgraphs and Pass Pipline mechanism, merged the core modules of dynamic and static shapes, and merged the iteration paths, so that the architecture is clear and unified. In this version, the compiler has been refactored in important back-end modules such as AST Compute, Schedule strategy, and Tiling, improving the general optimization capability of the compiler, and verifies the training, inference correctness and speedup performance of the dynamic shapes on the subgraphs of PaddlePaddle Industry Suite models and typical large models Llama2-7B and Stable Diffusion models. -- Implemented auto parallel core APIs such as process_mesh, placement, shard_tensor, reshard, dtensor_from_fn, unshard_dtensor, shard_layer, to_static, and so on. [#55494](https://github.com/PaddlePaddle/Paddle/pull/55494),[#59059](https://github.com/PaddlePaddle/Paddle/pull/59059),[#56561](https://github.com/PaddlePaddle/Paddle/pull/56561),[#54425](https://github.com/PaddlePaddle/Paddle/pull/54425),[#59557](https://github.com/PaddlePaddle/Paddle/pull/59557),[#59682](https://github.com/PaddlePaddle/Paddle/pull/59682),[#56565](https://github.com/PaddlePaddle/Paddle/pull/56565),[#59862](https://github.com/PaddlePaddle/Paddle/pull/59862),[#59856](https://github.com/PaddlePaddle/Paddle/pull/59856),[#59342](https://github.com/PaddlePaddle/Paddle/pull/59342),[#59575](https://github.com/PaddlePaddle/Paddle/pull/59575),[#57604](https://github.com/PaddlePaddle/Paddle/pull/57604),[#57293](https://github.com/PaddlePaddle/Paddle/pull/57293),[#57278](https://github.com/PaddlePaddle/Paddle/pull/57278) -- Implemented Sharding derivation rules based on Enisum expressions, and completed 20+ classes of operator Sharding derivation rules, which covers LLaMA, GPT and other transformer-like large language models. [#55196](https://github.com/PaddlePaddle/Paddle/pull/55196),[#53863](https://github.com/PaddlePaddle/Paddle/pull/53863),[#56257](https://github.com/PaddlePaddle/Paddle/pull/56257),[#55394](https://github.com/PaddlePaddle/Paddle/pull/55394),[#54810](https://github.com/PaddlePaddle/Paddle/pull/54810),[#55508](https://github.com/PaddlePaddle/Paddle/pull/55508),[#56257](https://github.com/PaddlePaddle/Paddle/pull/56257),[#57813](https://github.com/PaddlePaddle/Paddle/pull/57813),[#58149](https://github.com/PaddlePaddle/Paddle/pull/58149),[#58506](https://github.com/PaddlePaddle/Paddle/pull/58506),[#58563](https://github.com/PaddlePaddle/Paddle/pull/58563),[#58360](https://github.com/PaddlePaddle/Paddle/pull/58360),[#58920](https://github.com/PaddlePaddle/Paddle/pull/58920),[#59050](https://github.com/PaddlePaddle/Paddle/pull/59050),[#58760](https://github.com/PaddlePaddle/Paddle/pull/58760),[#59083](https://github.com/PaddlePaddle/Paddle/pull/59083),[#59236](https://github.com/PaddlePaddle/Paddle/pull/59236),[#59350](https://github.com/PaddlePaddle/Paddle/pull/59350),[#59411](https://github.com/PaddlePaddle/Paddle/pull/59411),[#59260](https://github.com/PaddlePaddle/Paddle/pull/59260),[#54373](https://github.com/PaddlePaddle/Paddle/pull/54373),[#54991](https://github.com/PaddlePaddle/Paddle/pull/54991),[#55397](https://github.com/PaddlePaddle/Paddle/pull/55397),[#55350](https://github.com/PaddlePaddle/Paddle/pull/55350),[#55177](https://github.com/PaddlePaddle/Paddle/pull/55177),[#56443](https://github.com/PaddlePaddle/Paddle/pull/56443),[#58097](https://github.com/PaddlePaddle/Paddle/pull/58097),[#56509](https://github.com/PaddlePaddle/Paddle/pull/56509),[#56502](https://github.com/PaddlePaddle/Paddle/pull/56502),[#56504](https://github.com/PaddlePaddle/Paddle/pull/56504),[#56506](https://github.com/PaddlePaddle/Paddle/pull/56506),[#56507](https://github.com/PaddlePaddle/Paddle/pull/56507),[#56505](https://github.com/PaddlePaddle/Paddle/pull/56505),[#57176](https://github.com/PaddlePaddle/Paddle/pull/57176),[#57374](https://github.com/PaddlePaddle/Paddle/pull/57374),[#57573](https://github.com/PaddlePaddle/Paddle/pull/57573),[#57545](https://github.com/PaddlePaddle/Paddle/pull/57545),[#57875](https://github.com/PaddlePaddle/Paddle/pull/57875),[#57866](https://github.com/PaddlePaddle/Paddle/pull/57866),[#58854](https://github.com/PaddlePaddle/Paddle/pull/58854),[#59109](https://github.com/PaddlePaddle/Paddle/pull/59109),[#59185](https://github.com/PaddlePaddle/Paddle/pull/59185),[#58913](https://github.com/PaddlePaddle/Paddle/pull/58913),[#59547](https://github.com/PaddlePaddle/Paddle/pull/59547),[#58296](https://github.com/PaddlePaddle/Paddle/pull/58296),[#59545](https://github.com/PaddlePaddle/Paddle/pull/59545),[#59039](https://github.com/PaddlePaddle/Paddle/pull/59039),[#59002](https://github.com/PaddlePaddle/Paddle/pull/59002),[#58087](https://github.com/PaddlePaddle/Paddle/pull/58087),[#56367](https://github.com/PaddlePaddle/Paddle/pull/56367),[#57877](https://github.com/PaddlePaddle/Paddle/pull/57877),[#56839](https://github.com/PaddlePaddle/Paddle/pull/56839),[#59003](https://github.com/PaddlePaddle/Paddle/pull/59003),[#57269](https://github.com/PaddlePaddle/Paddle/pull/57269),[#55130](https://github.com/PaddlePaddle/Paddle/pull/55130),[#58474](https://github.com/PaddlePaddle/Paddle/pull/58474),[#57197](https://github.com/PaddlePaddle/Paddle/pull/57197),[#57467](https://github.com/PaddlePaddle/Paddle/pull/57467),[#57259](https://github.com/PaddlePaddle/Paddle/pull/57259),[#57280](https://github.com/PaddlePaddle/Paddle/pull/57280),[#56508](https://github.com/PaddlePaddle/Paddle/pull/56508) -- Implemented distributed checkpoint storage and loading with dynamic-static unification. Supports ReShard upon arbitrary Sharding of storage and loading in a Sharding state. [#59659](https://github.com/PaddlePaddle/Paddle/pull/59659),[#59843](https://github.com/PaddlePaddle/Paddle/pull/59843),[#60033](https://github.com/PaddlePaddle/Paddle/pull/60033),[#60034](https://github.com/PaddlePaddle/Paddle/pull/60034) +### New Features -#### Enhanced semi-auto parallel capability of dynamic graph +1. Upgrade the new automatic subgraph fusion mechanism, and innovatively propose the TrivialOp and ReduceOp fusion theory, supporting a wider range of vertical fusion and horizontal fusion, ensuring the correctness and robustness of subgraph fusion, and giving full play to the fusion potential of the neural network compiler.（[#63340](https://github.com/PaddlePaddle/Paddle/pull/63340)、[#63913](https://github.com/PaddlePaddle/Paddle/pull/63913)、[#63579](https://github.com/PaddlePaddle/Paddle/pull/63579)、[#63605](https://github.com/PaddlePaddle/Paddle/pull/63605)、[#60769](https://github.com/PaddlePaddle/Paddle/pull/60769)、[#62088](https://github.com/PaddlePaddle/Paddle/pull/62088)、[#63124](https://github.com/PaddlePaddle/Paddle/pull/63124)、[#63658](https://github.com/PaddlePaddle/Paddle/pull/63658)、[#64557](https://github.com/PaddlePaddle/Paddle/pull/64557)、[#63318](https://github.com/PaddlePaddle/Paddle/pull/63318)、[#62545](https://github.com/PaddlePaddle/Paddle/pull/62545)） +2. Add the symbol derivation function of dynamic shapes. Based on the Shape Dialect, realize the dynamic symbol construction, automatic derivation, constraint expression, symbol simplification and other mechanisms, introduce the DimExpr concept, upgrade the support for the PaddlePaddle framework of the InferSymbolicShape logic of the 150 + typical primitive operators, and provide more information for training and inference with compiler support for dynamic shapes.（[#60843](https://github.com/PaddlePaddle/Paddle/pull/60843)、[#62662](https://github.com/PaddlePaddle/Paddle/pull/62662)、[#63790](https://github.com/PaddlePaddle/Paddle/pull/63790)、[#60098](https://github.com/PaddlePaddle/Paddle/pull/60098)、[#60511](https://github.com/PaddlePaddle/Paddle/pull/60511)、[#61232](https://github.com/PaddlePaddle/Paddle/pull/61232)、[#61939](https://github.com/PaddlePaddle/Paddle/pull/61939)、[#62798](https://github.com/PaddlePaddle/Paddle/pull/62798)、[#62955](https://github.com/PaddlePaddle/Paddle/pull/62955)、[#63029](https://github.com/PaddlePaddle/Paddle/pull/63029)、[#60572](https://github.com/PaddlePaddle/Paddle/pull/60572)、[#61035](https://github.com/PaddlePaddle/Paddle/pull/61035)、[#61224](https://github.com/PaddlePaddle/Paddle/pull/61224)、[#61587](https://github.com/PaddlePaddle/Paddle/pull/61587)、[#61937](https://github.com/PaddlePaddle/Paddle/pull/61937)、[#62314](https://github.com/PaddlePaddle/Paddle/pull/62314)、[#62394](https://github.com/PaddlePaddle/Paddle/pull/62394)、[#62569](https://github.com/PaddlePaddle/Paddle/pull/62569)、[#62495](https://github.com/PaddlePaddle/Paddle/pull/62495)、[#62844](https://github.com/PaddlePaddle/Paddle/pull/62844)、[#63000](https://github.com/PaddlePaddle/Paddle/pull/63000)、[#63016](https://github.com/PaddlePaddle/Paddle/pull/63016)、[#64222](https://github.com/PaddlePaddle/Paddle/pull/64222)、[#60129](https://github.com/PaddlePaddle/Paddle/pull/60129)、[#60899](https://github.com/PaddlePaddle/Paddle/pull/60899)、[#61342](https://github.com/PaddlePaddle/Paddle/pull/61342)、[#61439](https://github.com/PaddlePaddle/Paddle/pull/61439)、[#62766](https://github.com/PaddlePaddle/Paddle/pull/62766)、[#61133](https://github.com/PaddlePaddle/Paddle/pull/61133)、[#61430](https://github.com/PaddlePaddle/Paddle/pull/61430)、[#61498](https://github.com/PaddlePaddle/Paddle/pull/61498)、[#61680](https://github.com/PaddlePaddle/Paddle/pull/61680)、[#63367](https://github.com/PaddlePaddle/Paddle/pull/63367)、[#62151](https://github.com/PaddlePaddle/Paddle/pull/62151)、[#62665](https://github.com/PaddlePaddle/Paddle/pull/62665)、[#61407](https://github.com/PaddlePaddle/Paddle/pull/61407)、[#61502](https://github.com/PaddlePaddle/Paddle/pull/61502)、[#61655](https://github.com/PaddlePaddle/Paddle/pull/61655)、[#64115](https://github.com/PaddlePaddle/Paddle/pull/64115)、[#61791](https://github.com/PaddlePaddle/Paddle/pull/61791)、[#62141](https://github.com/PaddlePaddle/Paddle/pull/62141)、[#63422](https://github.com/PaddlePaddle/Paddle/pull/63422)、[#63577](https://github.com/PaddlePaddle/Paddle/pull/63577)、[#63978](https://github.com/PaddlePaddle/Paddle/pull/63978)、[#63576](https://github.com/PaddlePaddle/Paddle/pull/63576)、[#63947](https://github.com/PaddlePaddle/Paddle/pull/63947)、[#64332](https://github.com/PaddlePaddle/Paddle/pull/64332)、[#63990](https://github.com/PaddlePaddle/Paddle/pull/63990)） +3. Add the Pass Pipline function, including PdToCinn, CinnPreprocess, BuildGroupOp, GroupClusterOp, CinnLowering, Accuracy Check and other Pass strategies, to support the Lowering and execution of subgraphs in dynamic and static shapes, with a clear architecture.（[#61611](https://github.com/PaddlePaddle/Paddle/pull/61611)、[#62612](https://github.com/PaddlePaddle/Paddle/pull/62612)、[#64354](https://github.com/PaddlePaddle/Paddle/pull/64354)、[#61848](https://github.com/PaddlePaddle/Paddle/pull/61848)、[#62316](https://github.com/PaddlePaddle/Paddle/pull/62316)、[#64152](https://github.com/PaddlePaddle/Paddle/pull/64152)、[#61619](https://github.com/PaddlePaddle/Paddle/pull/61619)、[#62318](https://github.com/PaddlePaddle/Paddle/pull/62318)、[#61977](https://github.com/PaddlePaddle/Paddle/pull/61977)、[#62211](https://github.com/PaddlePaddle/Paddle/pull/62211)、[#63972](https://github.com/PaddlePaddle/Paddle/pull/63972)、[#63686](https://github.com/PaddlePaddle/Paddle/pull/63686)、[#64505](https://github.com/PaddlePaddle/Paddle/pull/64505)） +4. Add the support for BuketLower and DyShapeSchdule functions, to realize automatic bucket compilation and optimization according to the range of dynamic shapes; and adapt and upgrade the logic of CodeGen module to support the generation of InferShape function and the distribution of conditional branching function of Host function, so as to support the acceleration of training inference under the dynamic Shape of large models.（[#62730](https://github.com/PaddlePaddle/Paddle/pull/62730)、[#61115](https://github.com/PaddlePaddle/Paddle/pull/61115)、[#59941](https://github.com/PaddlePaddle/Paddle/pull/59941)、[#62207](https://github.com/PaddlePaddle/Paddle/pull/62207)、[#64318](https://github.com/PaddlePaddle/Paddle/pull/64318)、[#64345](https://github.com/PaddlePaddle/Paddle/pull/64345)、[#60519](https://github.com/PaddlePaddle/Paddle/pull/60519)、[#62584](https://github.com/PaddlePaddle/Paddle/pull/62584)、[#60828](https://github.com/PaddlePaddle/Paddle/pull/60828)、[#60533](https://github.com/PaddlePaddle/Paddle/pull/60533)、[#61436](https://github.com/PaddlePaddle/Paddle/pull/61436)、[#62071](https://github.com/PaddlePaddle/Paddle/pull/62071)、[#63971](https://github.com/PaddlePaddle/Paddle/pull/63971)、[#61656](https://github.com/PaddlePaddle/Paddle/pull/61656)、[#63083](https://github.com/PaddlePaddle/Paddle/pull/63083)、[#64405](https://github.com/PaddlePaddle/Paddle/pull/64405)、[#63047](https://github.com/PaddlePaddle/Paddle/pull/63047)、[#64655](https://github.com/PaddlePaddle/Paddle/pull/64655)、[#63095](https://github.com/PaddlePaddle/Paddle/pull/63095)、[#63829](https://github.com/PaddlePaddle/Paddle/pull/63829)、[#63572](https://github.com/PaddlePaddle/Paddle/pull/63572)） +5. Add support for compilation caching strategy, to automatically recognize, merge and reuse compilation results of the same subgraph structure, improve compilation efficiency by using multi-threading, so as to enhance the user experience.（[#62952](https://github.com/PaddlePaddle/Paddle/pull/62952)、[#63269](https://github.com/PaddlePaddle/Paddle/pull/63269)、[#64718](https://github.com/PaddlePaddle/Paddle/pull/64718)、[#61367](https://github.com/PaddlePaddle/Paddle/pull/61367)、[#63305](https://github.com/PaddlePaddle/Paddle/pull/63305)、[#63750](https://github.com/PaddlePaddle/Paddle/pull/63750)、[#63871](https://github.com/PaddlePaddle/Paddle/pull/63871)、[#64893](https://github.com/PaddlePaddle/Paddle/pull/64893)） +6. Add support for GenerateShape mechanism, add corresponding AST Compute operator definitions, support automatic resolution of dynamic symbols, and automatic generation of ShapeOp in the Lowering stage.（[#64167](https://github.com/PaddlePaddle/Paddle/pull/64167)、[#64636](https://github.com/PaddlePaddle/Paddle/pull/64636)、[#61993](https://github.com/PaddlePaddle/Paddle/pull/61993)、[#64843](https://github.com/PaddlePaddle/Paddle/pull/64843)、[#62587](https://github.com/PaddlePaddle/Paddle/pull/62587)） -- Basic data structure supplementation: Added DistTensor, Placements and other distributed specific basic data structures on C++ end, and exposed to Python end. Supports debugging and printing of related attributes and values. [#58930](https://github.com/PaddlePaddle/Paddle/pull/58930),[#59068](https://github.com/PaddlePaddle/Paddle/pull/59068),[#55436](https://github.com/PaddlePaddle/Paddle/pull/55436),[#56449](https://github.com/PaddlePaddle/Paddle/pull/56449),[#59683](https://github.com/PaddlePaddle/Paddle/pull/59683),[#55593](https://github.com/PaddlePaddle/Paddle/pull/55593),[#58032](https://github.com/PaddlePaddle/Paddle/pull/58032),[#56368](https://github.com/PaddlePaddle/Paddle/pull/56368),[#59086](https://github.com/PaddlePaddle/Paddle/pull/59086) -- Added SPMD derivation and Reshard generation logic in execution flow for all operators, and adapted to multiple types of inputs and outputs such as vector and optional, as well as special mechanisms such as cpu fallback and multi-kernel selection. [#56602](https://github.com/PaddlePaddle/Paddle/pull/56602),[#57321](https://github.com/PaddlePaddle/Paddle/pull/57321),[#57092](https://github.com/PaddlePaddle/Paddle/pull/57092),[#56831](https://github.com/PaddlePaddle/Paddle/pull/56831),[#57119](https://github.com/PaddlePaddle/Paddle/pull/57119),[#58819](https://github.com/PaddlePaddle/Paddle/pull/58819),[#58254](https://github.com/PaddlePaddle/Paddle/pull/58254),[#55698](https://github.com/PaddlePaddle/Paddle/pull/55698),[#59241](https://github.com/PaddlePaddle/Paddle/pull/59241),[#59328](https://github.com/PaddlePaddle/Paddle/pull/59328),[#58644](https://github.com/PaddlePaddle/Paddle/pull/58644),[#56202](https://github.com/PaddlePaddle/Paddle/pull/56202),[#59159](https://github.com/PaddlePaddle/Paddle/pull/59159),[#58573](https://github.com/PaddlePaddle/Paddle/pull/58573),[#59246](https://github.com/PaddlePaddle/Paddle/pull/59246),[#59133](https://github.com/PaddlePaddle/Paddle/pull/59133),[#59186](https://github.com/PaddlePaddle/Paddle/pull/59186),[#57505](https://github.com/PaddlePaddle/Paddle/pull/57505),[#57241](https://github.com/PaddlePaddle/Paddle/pull/57241),[#58928](https://github.com/PaddlePaddle/Paddle/pull/58928) +### Function Optimization -- Adapted auto parallel execution logic for special types of operators, such as custom operators. Supports automatic conversion of DistTensor and DenseTensor as mixed inputs. [#57774](https://github.com/PaddlePaddle/Paddle/pull/57774),[#59108](https://github.com/PaddlePaddle/Paddle/pull/59108),[#58436](https://github.com/PaddlePaddle/Paddle/pull/58436),[#59523](https://github.com/PaddlePaddle/Paddle/pull/59523),[#59136](https://github.com/PaddlePaddle/Paddle/pull/59136),[#59352](https://github.com/PaddlePaddle/Paddle/pull/59352),[#59062](https://github.com/PaddlePaddle/Paddle/pull/59062),[#58434](https://github.com/PaddlePaddle/Paddle/pull/58434),[#59148](https://github.com/PaddlePaddle/Paddle/pull/59148),[#58553](https://github.com/PaddlePaddle/Paddle/pull/58553),[#58716](https://github.com/PaddlePaddle/Paddle/pull/58716),[#58369](https://github.com/PaddlePaddle/Paddle/pull/58369),[#59061](https://github.com/PaddlePaddle/Paddle/pull/59061),[#58841](https://github.com/PaddlePaddle/Paddle/pull/58841),[#59139](https://github.com/PaddlePaddle/Paddle/pull/59139),[#59141](https://github.com/PaddlePaddle/Paddle/pull/59141),[#58837](https://github.com/PaddlePaddle/Paddle/pull/58837),[#59137](https://github.com/PaddlePaddle/Paddle/pull/59137),[#59143](https://github.com/PaddlePaddle/Paddle/pull/59143) +1. Optimize BuildCinnPass logic, upgrade the compiler's perception strategy for black and white list operators, and improve the robustness of Pass logic.（[#62372](https://github.com/PaddlePaddle/Paddle/pull/62372)、[#61081](https://github.com/PaddlePaddle/Paddle/pull/61081)、[#61225](https://github.com/PaddlePaddle/Paddle/pull/61225)、[#58863](https://github.com/PaddlePaddle/Paddle/pull/58863)） +2. Optimize the OpLoweringGroup data structure, remove unnecessary interfaces and members, and reduce the coupling between upstream and downstream modules.（[#62339](https://github.com/PaddlePaddle/Paddle/pull/62339)） +3. Optimize the component design of the compiler on the architecture Arch, to abstract the concept of hardware, and reduce the cost of adapting to domestic hardware.（[#63530](https://github.com/PaddlePaddle/Paddle/pull/63530)、[#64347](https://github.com/PaddlePaddle/Paddle/pull/64347)、[#64506](https://github.com/PaddlePaddle/Paddle/pull/64506)、[#64587](https://github.com/PaddlePaddle/Paddle/pull/64587)） +4. Upgrade the AST Compute module of the compiler's back-end operator, to adapt to support the computing logic of dynamic Shape.（[#62488](https://github.com/PaddlePaddle/Paddle/pull/62488)、[#63581](https://github.com/PaddlePaddle/Paddle/pull/63581)、[#63687](https://github.com/PaddlePaddle/Paddle/pull/63687)、[#63654](https://github.com/PaddlePaddle/Paddle/pull/63654)、[#64217](https://github.com/PaddlePaddle/Paddle/pull/64217)） -- Optimized dynamic graph execution system: Adapted Autograd execution process. Supports dynamic graph's inverse gradient aggregation, AMP, Hook, PyLayer, View, custom operators, and other surrounding mechanisms. [#58437](https://github.com/PaddlePaddle/Paddle/pull/58437),[#58769](https://github.com/PaddlePaddle/Paddle/pull/58769),[#58796](https://github.com/PaddlePaddle/Paddle/pull/58796),[#58339](https://github.com/PaddlePaddle/Paddle/pull/58339),[#58409](https://github.com/PaddlePaddle/Paddle/pull/58409),[#58772](https://github.com/PaddlePaddle/Paddle/pull/58772),[#58380](https://github.com/PaddlePaddle/Paddle/pull/58380),[#58447](https://github.com/PaddlePaddle/Paddle/pull/58447),[#58706](https://github.com/PaddlePaddle/Paddle/pull/58706),[#58656](https://github.com/PaddlePaddle/Paddle/pull/58656),[#58172](https://github.com/PaddlePaddle/Paddle/pull/58172),[#59401](https://github.com/PaddlePaddle/Paddle/pull/59401),[#58727](https://github.com/PaddlePaddle/Paddle/pull/58727),[#58238](https://github.com/PaddlePaddle/Paddle/pull/58238),[#59243](https://github.com/PaddlePaddle/Paddle/pull/59243),[#58469](https://github.com/PaddlePaddle/Paddle/pull/58469),[#58442](https://github.com/PaddlePaddle/Paddle/pull/58442),[#58487](https://github.com/PaddlePaddle/Paddle/pull/58487),[#58476](https://github.com/PaddlePaddle/Paddle/pull/58476),[#59706](https://github.com/PaddlePaddle/Paddle/pull/59706) +### Performance Optimization -- Added support for Pipeline Parallelism, Sequence Parallelism and other distributed parallelism. [#58126](https://github.com/PaddlePaddle/Paddle/pull/58126),[#59766](https://github.com/PaddlePaddle/Paddle/pull/59766),[#59060](https://github.com/PaddlePaddle/Paddle/pull/59060),[#59841](https://github.com/PaddlePaddle/Paddle/pull/59841),[#58609](https://github.com/PaddlePaddle/Paddle/pull/58609),[#59688](https://github.com/PaddlePaddle/Paddle/pull/59688),[#58449](https://github.com/PaddlePaddle/Paddle/pull/58449)、[#59598](https://github.com/PaddlePaddle/Paddle/pull/59598) -- Added various Reshard strategies and support tensor conversions between different distributed states. [#58592](https://github.com/PaddlePaddle/Paddle/pull/58592),[#59138](https://github.com/PaddlePaddle/Paddle/pull/59138),[#59367](https://github.com/PaddlePaddle/Paddle/pull/59367),[#59621](https://github.com/PaddlePaddle/Paddle/pull/59621),[#59758](https://github.com/PaddlePaddle/Paddle/pull/59758),[#59777](https://github.com/PaddlePaddle/Paddle/pull/59777),[#56975](https://github.com/PaddlePaddle/Paddle/pull/56975),[#58550](https://github.com/PaddlePaddle/Paddle/pull/58550),[#58703](https://github.com/PaddlePaddle/Paddle/pull/58703),[#57210](https://github.com/PaddlePaddle/Paddle/pull/57210),[#58734](https://github.com/PaddlePaddle/Paddle/pull/58734),[#56833](https://github.com/PaddlePaddle/Paddle/pull/56833),[#59292](https://github.com/PaddlePaddle/Paddle/pull/59292),[#57432](https://github.com/PaddlePaddle/Paddle/pull/57432),[#57568](https://github.com/PaddlePaddle/Paddle/pull/57568),[#56553](https://github.com/PaddlePaddle/Paddle/pull/56553),[#58284](https://github.com/PaddlePaddle/Paddle/pull/58284),[#56039](https://github.com/PaddlePaddle/Paddle/pull/56039),[#55552](https://github.com/PaddlePaddle/Paddle/pull/55552),[#56149](https://github.com/PaddlePaddle/Paddle/pull/56149) +1. Optimize the Schedule logic of AST IR, restructure the core modules such as Vectorize, Unroll, AxisBind, and ComputeAt, and merged the iterative paths of dynamic and static shapes, so as to reduce the development and maintenance costs.（[#60449](https://github.com/PaddlePaddle/Paddle/pull/60449)、[#60155](https://github.com/PaddlePaddle/Paddle/pull/60155)、[#60342](https://github.com/PaddlePaddle/Paddle/pull/60342)、[#60498](https://github.com/PaddlePaddle/Paddle/pull/60498)、[#60538](https://github.com/PaddlePaddle/Paddle/pull/60538)、[#60190](https://github.com/PaddlePaddle/Paddle/pull/60190)、[#61197](https://github.com/PaddlePaddle/Paddle/pull/61197)、[#63140](https://github.com/PaddlePaddle/Paddle/pull/63140)、[#61156](https://github.com/PaddlePaddle/Paddle/pull/61156)） +2. Optimize the Tiling strategy and temp Buffer function, support warp-level memory continuous Read and cache_read cache_write function, and improve the subgraph execution performance.（[#64240](https://github.com/PaddlePaddle/Paddle/pull/64240)、[#60562](https://github.com/PaddlePaddle/Paddle/pull/60562)、[#64711](https://github.com/PaddlePaddle/Paddle/pull/64711)、[#62856](https://github.com/PaddlePaddle/Paddle/pull/62856)、[#61576](https://github.com/PaddlePaddle/Paddle/pull/61576)、[#61901](https://github.com/PaddlePaddle/Paddle/pull/61901)、[#62581](https://github.com/PaddlePaddle/Paddle/pull/62581)、[#61987](https://github.com/PaddlePaddle/Paddle/pull/61987)、[#60190](https://github.com/PaddlePaddle/Paddle/pull/60190)、[#63138](https://github.com/PaddlePaddle/Paddle/pull/63138)、[#62517](https://github.com/PaddlePaddle/Paddle/pull/62517)） +3. Support automatic search function of Schedule configuration and AOT offline saving mechanism to accelerate the performance of subgraph Kernel.（[#64271](https://github.com/PaddlePaddle/Paddle/pull/64271)、[#64588](https://github.com/PaddlePaddle/Paddle/pull/64588)、[#64694](https://github.com/PaddlePaddle/Paddle/pull/64694)、[#64620](https://github.com/PaddlePaddle/Paddle/pull/64620)、[#64702](https://github.com/PaddlePaddle/Paddle/pull/64702)、[#63086](https://github.com/PaddlePaddle/Paddle/pull/63086)） +4. Support OptimizeReductionTactic optimization strategy to improve kernel performance in Reduce scenarios.（[#6066](https://github.com/PaddlePaddle/Paddle/pull/60661)、[#61363](https://github.com/PaddlePaddle/Paddle/pull/61363)、[#60881](https://github.com/PaddlePaddle/Paddle/pull/60881)、[#63859](https://github.com/PaddlePaddle/Paddle/pull/63859)） +5. Enhance DCE Pass function, remove redundant If/For branch codes and improve execution efficiency.（[#61682](https://github.com/PaddlePaddle/Paddle/pull/61682)） +6. Add support for FuseParallelMatmulPass Pass, integrate multiple Matmul operators to achieve acceleration.（[#63623](https://github.com/PaddlePaddle/Paddle/pull/63623)） -#### Enhanced semi-auto parallel for static graphs +### Bug Fixing -- Added Sequence Parallel Parallelism; added FThenB, Interleaved 1F1B, Eager 1F1B, VPP and other scheduling modes for Pipeline Parallel, and supported the hybrid parallel between the above new parallelism and original parallelism. Supported visualization of pipeline scheduling. Upgraded gradient synchronization mechanism which supports gradient synchronization when data is sharded on any broadcast dimension. [#57605](https://github.com/PaddlePaddle/Paddle/pull/57605),[#54727](https://github.com/PaddlePaddle/Paddle/pull/54727),[#54409](https://github.com/PaddlePaddle/Paddle/pull/54409),[#54787](https://github.com/PaddlePaddle/Paddle/pull/54787),[#58313](https://github.com/PaddlePaddle/Paddle/pull/58313),[#59179](https://github.com/PaddlePaddle/Paddle/pull/59179),[#59416](https://github.com/PaddlePaddle/Paddle/pull/59416),[#59719](https://github.com/PaddlePaddle/Paddle/pull/59719),[#59822](https://github.com/PaddlePaddle/Paddle/pull/59822),[#59057](https://github.com/PaddlePaddle/Paddle/pull/59057),[#59522](https://github.com/PaddlePaddle/Paddle/pull/59522),[#57061](https://github.com/PaddlePaddle/Paddle/pull/57061) -- Adapted the executor to PIR, and supported PIR optimization Pass. In distributed scenarios, supports fuse_linear fuse, and etc., to improve performance. [#58459](https://github.com/PaddlePaddle/Paddle/pull/58459),[#58528](https://github.com/PaddlePaddle/Paddle/pull/58528),[#55555](https://github.com/PaddlePaddle/Paddle/pull/55555),[#59757](https://github.com/PaddlePaddle/Paddle/pull/59757),[#59102](https://github.com/PaddlePaddle/Paddle/pull/59102),[#57917](https://github.com/PaddlePaddle/Paddle/pull/57917) -- Upgraded underlying architecture: upgraded the executor to reuse the results of data-flow dependency analysis and static kernel selection; upgraded entire graph based sharding completion mechanism, to switch to new sharding derivation rules and support some long-tailed cases; optimized the support of control flow under distributed static graph to adapt to more scenarios; reduced the graph compilation time and refined error message format to improve user experience. [#55389](https://github.com/PaddlePaddle/Paddle/pull/55389),[#55650](https://github.com/PaddlePaddle/Paddle/pull/55650),[#54938](https://github.com/PaddlePaddle/Paddle/pull/54938),[#57447](https://github.com/PaddlePaddle/Paddle/pull/57447),[#57751](https://github.com/PaddlePaddle/Paddle/pull/57751),[#57742](https://github.com/PaddlePaddle/Paddle/pull/57742),[#59524](https://github.com/PaddlePaddle/Paddle/pull/59524),[#59526](https://github.com/PaddlePaddle/Paddle/pull/59526),[#58669](https://github.com/PaddlePaddle/Paddle/pull/58669),[#57616](https://github.com/PaddlePaddle/Paddle/pull/57616),[#56511](https://github.com/PaddlePaddle/Paddle/pull/56511),[#55727](https://github.com/PaddlePaddle/Paddle/pull/55727),[#58906](https://github.com/PaddlePaddle/Paddle/pull/58906),[#56016](https://github.com/PaddlePaddle/Paddle/pull/56016),[#54897](https://github.com/PaddlePaddle/Paddle/pull/54897) -- Optimized the gpu memory usage in static graph mode, and added refined recomputing strategy; optimized auto mixed precision pass, and allows users to manually specify auto-cast region and fixed some bugs; supports parallel computation of cross-entropy; supports fusion operators such as scaled_dot_product_attention, fuse_rope, etc.; performs scheduling optimization to support better overlap between communication and computation in tensor parallelism and pipeline parallelsim. [#58421](https://github.com/PaddlePaddle/Paddle/pull/58421),[#58533](https://github.com/PaddlePaddle/Paddle/pull/58533),[#59498](https://github.com/PaddlePaddle/Paddle/pull/59498),[#59498](https://github.com/PaddlePaddle/Paddle/pull/59498),[#59187](https://github.com/PaddlePaddle/Paddle/pull/59187),[#59188](https://github.com/PaddlePaddle/Paddle/pull/59188),[#58172](https://github.com/PaddlePaddle/Paddle/pull/58172),[#58628](https://github.com/PaddlePaddle/Paddle/pull/58628),[#56185](https://github.com/PaddlePaddle/Paddle/pull/56185),[#56696](https://github.com/PaddlePaddle/Paddle/pull/56696),[#59497](https://github.com/PaddlePaddle/Paddle/pull/59497),[#58304](https://github.com/PaddlePaddle/Paddle/pull/58304),[#58977](https://github.com/PaddlePaddle/Paddle/pull/58977) +1. Fix the bug when Lowering some special operators to the compiler, to improve the end-to-end user experience.（[#60800](https://github.com/PaddlePaddle/Paddle/pull/60800)、[#64720](https://github.com/PaddlePaddle/Paddle/pull/64720)、[#62593](https://github.com/PaddlePaddle/Paddle/pull/62593)、[#62661](https://github.com/PaddlePaddle/Paddle/pull/62661)、[#64626](https://github.com/PaddlePaddle/Paddle/pull/64626)、[#63320](https://github.com/PaddlePaddle/Paddle/pull/63320)、[#64581](https://github.com/PaddlePaddle/Paddle/pull/64581)、[#61608](https://github.com/PaddlePaddle/Paddle/pull/61608)、[#64135](https://github.com/PaddlePaddle/Paddle/pull/64135)、[#64659](https://github.com/PaddlePaddle/Paddle/pull/64659)、[#62391](https://github.com/PaddlePaddle/Paddle/pull/62391)、[#62490](https://github.com/PaddlePaddle/Paddle/pull/62490)、[#63891](https://github.com/PaddlePaddle/Paddle/pull/63891)、[#64529](https://github.com/PaddlePaddle/Paddle/pull/64529)） +2. Fix a bug in the symbolic derivation logic of some operators.（[#62141](https://github.com/PaddlePaddle/Paddle/pull/62141)、[#62376](https://github.com/PaddlePaddle/Paddle/pull/62376)、[#62941](https://github.com/PaddlePaddle/Paddle/pull/62941)、[#63322](https://github.com/PaddlePaddle/Paddle/pull/63322)、[#64672](https://github.com/PaddlePaddle/Paddle/pull/64672)、[#64407](https://github.com/PaddlePaddle/Paddle/pull/64407)、[#60241](https://github.com/PaddlePaddle/Paddle/pull/60241)、[#60440](https://github.com/PaddlePaddle/Paddle/pull/60440)、[#62503](https://github.com/PaddlePaddle/Paddle/pull/62503)、[#62997](https://github.com/PaddlePaddle/Paddle/pull/62997)、[#63169](https://github.com/PaddlePaddle/Paddle/pull/63169)、[#61098](https://github.com/PaddlePaddle/Paddle/pull/61098)、[#63973](https://github.com/PaddlePaddle/Paddle/pull/63973)、[#62248](https://github.com/PaddlePaddle/Paddle/pull/62248)、[#62321](https://github.com/PaddlePaddle/Paddle/pull/62321)、[#63755](https://github.com/PaddlePaddle/Paddle/pull/63755)、[#63917](https://github.com/PaddlePaddle/Paddle/pull/63917)、[#63903](https://github.com/PaddlePaddle/Paddle/pull/63903)、[#64173](https://github.com/PaddlePaddle/Paddle/pull/64173)、[#64525](https://github.com/PaddlePaddle/Paddle/pull/64525)、[#64615](https://github.com/PaddlePaddle/Paddle/pull/64615)、[#62247](https://github.com/PaddlePaddle/Paddle/pull/62247)、[#62455](https://github.com/PaddlePaddle/Paddle/pull/62455)、[#62898](https://github.com/PaddlePaddle/Paddle/pull/62898)、[#62867](https://github.com/PaddlePaddle/Paddle/pull/62867)、[#63608](https://github.com/PaddlePaddle/Paddle/pull/63608)、[#63789](https://github.com/PaddlePaddle/Paddle/pull/63789)、[#64085](https://github.com/PaddlePaddle/Paddle/pull/64085)、[#64136](https://github.com/PaddlePaddle/Paddle/pull/64136)、[#64181](https://github.com/PaddlePaddle/Paddle/pull/64181)） +3. Fix the problems of compiler execution errors under dynamic and static shapes, to improve the robustness of the framework mechanism.（[#60813](https://github.com/PaddlePaddle/Paddle/pull/60813)、[#61877](https://github.com/PaddlePaddle/Paddle/pull/61877)、[#61909](https://github.com/PaddlePaddle/Paddle/pull/61909)、[#62954](https://github.com/PaddlePaddle/Paddle/pull/62954)、[#63614](https://github.com/PaddlePaddle/Paddle/pull/63614)、[#60339](https://github.com/PaddlePaddle/Paddle/pull/60339)、[#60623](https://github.com/PaddlePaddle/Paddle/pull/60623)、[#60658](https://github.com/PaddlePaddle/Paddle/pull/60658)、[#60669](https://github.com/PaddlePaddle/Paddle/pull/60669)、[#58823](https://github.com/PaddlePaddle/Paddle/pull/58823)、[#62483](https://github.com/PaddlePaddle/Paddle/pull/62483)、[#62742](https://github.com/PaddlePaddle/Paddle/pull/62742)、[#61797](https://github.com/PaddlePaddle/Paddle/pull/61797)、[#63411](https://github.com/PaddlePaddle/Paddle/pull/63411)、[#64077](https://github.com/PaddlePaddle/Paddle/pull/64077)、[#62736](https://github.com/PaddlePaddle/Paddle/pull/62736)、[#62390](https://github.com/PaddlePaddle/Paddle/pull/62390)、[#63689](https://github.com/PaddlePaddle/Paddle/pull/63689)） -#### AutoTuner +### Deprecated Features -This release implements a profiling based automatic search and tuning tool named AutoTuner for parallel strategies, to automatically combine parallel and optimization strategies. Users can select effective combination configurations for experiments, and AutoTuner will search for the optimal configuration for large model training and inference given the model and hardware specification. In addition, AutoTuner implements a variety of pruning methods, including gpu memory modelling based pruning, so the search space and search time can be significantly reduced. [#54460](https://github.com/PaddlePaddle/Paddle/pull/54460),[#54668](https://github.com/PaddlePaddle/Paddle/pull/54668),[#59794](https://github.com/PaddlePaddle/Paddle/pull/59794),[#59727](https://github.com/PaddlePaddle/Paddle/pull/59727),[#59782](https://github.com/PaddlePaddle/Paddle/pull/59782),[#54834](https://github.com/PaddlePaddle/Paddle/pull/54834),[#58127](https://github.com/PaddlePaddle/Paddle/pull/58127),[#56968](https://github.com/PaddlePaddle/Paddle/pull/56968),[#55466](https://github.com/PaddlePaddle/Paddle/pull/55466),[#56939](https://github.com/PaddlePaddle/Paddle/pull/56939),[#58183](https://github.com/PaddlePaddle/Paddle/pull/58183),[#58314](https://github.com/PaddlePaddle/Paddle/pull/58314),[#55499](https://github.com/PaddlePaddle/Paddle/pull/55499),[#59748](https://github.com/PaddlePaddle/Paddle/pull/59748) +1. Remove useless symbol-related components such as adt DimExpr, SymbolicDimExpr and ShapedTypeInterface.（[#60901](https://github.com/PaddlePaddle/Paddle/pull/60901)、[#60933](https://github.com/PaddlePaddle/Paddle/pull/60933)、[#60744](https://github.com/PaddlePaddle/Paddle/pull/60744)、[#64176](https://github.com/PaddlePaddle/Paddle/pull/64176)、[#64140](https://github.com/PaddlePaddle/Paddle/pull/64140)） +2. Remove the old Group Cluster, and the front-end representation under the old IR, to improve the simplicity of the architecture.（[#63683](https://github.com/PaddlePaddle/Paddle/pull/63683)、[#64630](https://github.com/PaddlePaddle/Paddle/pull/64630)、[#61380](https://github.com/PaddlePaddle/Paddle/pull/61380)） -### Operator library +## Auto-Parallel Architecture -#### Incompatible upgrade +In order to further enhance the usability of the Auto Parallel architecture in large model training scenarios, PaddlePaddle has improved the Auto Parallel functionality in dynamic-static graphs, including the newly added parallel strategies such as sharding parallelism and interleaved pipeline parallelism, including support of lazy initialization parameters. Add and enhance the SPMD derivation rules for some of the operators. The auto-parallel architecture has been comprehensively verified in a number of mainstream large language models. Meanwhile, in order to build the new 3.0 architecture of PaddlePaddle, the static graph auto parallel architecture has been comprehensively upgraded based on PIR, the new generation intermediate representation of Paddlepaddle. It introduces DistDialect for distributed related components, and natively support DistAttr and DistTensor in the computation graph representation, and smooth the transfom from static to dynmaic graph, further enhance the unity of auto parallel usage in dynamic and static graph mode. Finally, a number of performance optimization technologies have been added and improved, including zero bubble pipeline scheduling strategy, achieving the same or even better end-to-end training performance compared to the manual parallelism on typical large models such as Llama-2 13B/70B. -In order to improve maintainability of PaddlePaddle framework, some deprecated operators in the framework (e.g. diag_v1, isfinite_v1, pad2d_v1, etc.) have been removed, and models using these operators saved through the PaddlePaddle 1.x training will not be able to infer on new version of PaddlePaddle. [#57895](https://github.com/PaddlePaddle/Paddle/pull/57895),[#57892](https://github.com/PaddlePaddle/Paddle/pull/57892),[#57898](https://github.com/PaddlePaddle/Paddle/pull/57898),[#57730](https://github.com/PaddlePaddle/Paddle/pull/57730),[#57732](https://github.com/PaddlePaddle/Paddle/pull/57732),[#57810](https://github.com/PaddlePaddle/Paddle/pull/57810),[#57884](https://github.com/PaddlePaddle/Paddle/pull/57884),[#57794](https://github.com/PaddlePaddle/Paddle/pull/57794),[#57926](https://github.com/PaddlePaddle/Paddle/pull/57926),[#57925](https://github.com/PaddlePaddle/Paddle/pull/57925),[#57807](https://github.com/PaddlePaddle/Paddle/pull/57807),[#57808](https://github.com/PaddlePaddle/Paddle/pull/57808) +### Function Improvements -#### Operator library enhancements +- Add the dtensor_from_local interface for creating DistTensor from local tensor after sharding (correspondingly, shard_tensor is the created DistTensor from global tensor before sharding). [#60206](https://github.com/PaddlePaddle/Paddle/pull/60206) +- Add the unshard_tensor interface to convert DistTensor to global tensor, which is reciprocal operation to shard_tensor. [#60272](https://github.com/PaddlePaddle/Paddle/pull/60272) +- To reduce the GPU memory usage during training, add Sharding parallelism, and support stage1, stage2 and stage3 modes. [#61926](https://github.com/PaddlePaddle/Paddle/pull/61926), [#62711](https://github.com/PaddlePaddle/Paddle/pull/62711), [#62486](https://github.com/PaddlePaddle/Paddle/pull/62486), [#62230](https://github.com/PaddlePaddle/Paddle/pull/62230) +- To solve the problem of insufficient GPU memory when initializing parameters first and then sharding them, add the LazyInit function, to support slicing parameters first and then initializing them. [#60316](https://github.com/PaddlePaddle/Paddle/pull/60316), [#60441](https://github.com/PaddlePaddle/Paddle/pull/60441), [#60563](https://github.com/PaddlePaddle/Paddle/pull/60563), [#61792](https://github.com/PaddlePaddle/Paddle/pull/61792) +- In order to reduce the bubble of pipeline parallel, add the interleaved pipeline parallel parallelism has been added, and support automatically converting the pipeline parallel of the user's networking to interleaved pipeline parallel through configuration, so that the user doesn't need to perform complicated marking in the networking. [#59751](https://github.com/PaddlePaddle/Paddle/pull/59751), [#60050](https://github.com/PaddlePaddle/Paddle/pull/60050), [#60467](https://github.com/PaddlePaddle/Paddle/pull/60467), [#60868](https://github.com/PaddlePaddle/Paddle/pull/60868), [#60187](https://github.com/PaddlePaddle/Paddle/pull/60187), [#62884](https://github.com/PaddlePaddle/Paddle/pull/62884), [#60560](https://github.com/PaddlePaddle/Paddle/pull/60560), [#61541](https://github.com/PaddlePaddle/Paddle/pull/61541) +- Add the SPMD derivation rules for stack, gather, scatter_grad, cumsum, unbind, swiglu, and fused_linear_param_grad. Improve and optimize the implementation of fused_rope, reshape, flatten, fused_rms_norm, slice, tile, flash_attn, cross_entropy and other operator slice derivation rules, to solve the problem of incompatibility in some of the model networking scenarios. [#62720](https://github.com/PaddlePaddle/Paddle/pull/62720), [#64202](https://github.com/PaddlePaddle/Paddle/pull/64202), [#63361](https://github.com/PaddlePaddle/Paddle/pull/63361), [#63290](https://github.com/PaddlePaddle/Paddle/pull/63290), [#61460](https://github.com/PaddlePaddle/Paddle/pull/61460), [#59986](https://github.com/PaddlePaddle/Paddle/pull/59986), [#61184](https://github.com/PaddlePaddle/Paddle/pull/61184), [#60144](https://github.com/PaddlePaddle/Paddle/pull/60144), [#62525](https://github.com/PaddlePaddle/Paddle/pull/62525), [#62053](https://github.com/PaddlePaddle/Paddle/pull/62053), [#60709](https://github.com/PaddlePaddle/Paddle/pull/60709), [#60111](https://github.com/PaddlePaddle/Paddle/pull/60111), [#63681](https://github.com/PaddlePaddle/Paddle/pull/63681), [#62180](https://github.com/PaddlePaddle/Paddle/pull/62180), [#60794](https://github.com/PaddlePaddle/Paddle/pull/60794), [#60632](https://github.com/PaddlePaddle/Paddle/pull/60632), [#62439](https://github.com/PaddlePaddle/Paddle/pull/62439) +- Improve the distributed checkpoint storage and loading function, support master_weights strategy, and fix the random hanging problem. [#60027](https://github.com/PaddlePaddle/Paddle/pull/60027), [#59872](https://github.com/PaddlePaddle/Paddle/pull/59872) +- In order to support the auto parallel of arbitrary shape tensor, add the non-uniform tensor sharding feature. [#62611](https://github.com/PaddlePaddle/Paddle/pull/62611), [#61432](https://github.com/PaddlePaddle/Paddle/pull/61432) +- In order to support users to use customized operators in the auto parallel networking, support user registration outside the framework to customize the SPMD derivation rules for this class of operators. [#60509](https://github.com/PaddlePaddle/Paddle/pull/60509) +- Improve the slice SPMD rule, and support the transition from any state to replicate and from replicate state to any state. [#60281](https://github.com/PaddlePaddle/Paddle/pull/60281), [#59869](https://github.com/PaddlePaddle/Paddle/pull/59869) +- Add MoE expert parallelism (experimental). Currently, only dynamic graph auto parallel is supported. [#63904](https://github.com/PaddlePaddle/Paddle/pull/63904) +- Fix some process adaptation problems of auto parallel and dynamic diagram execution, and dynamic to static. [#60214](https://github.com/PaddlePaddle/Paddle/pull/60214), [#60546](https://github.com/PaddlePaddle/Paddle/pull/60546), [#62082](https://github.com/PaddlePaddle/Paddle/pull/62082), [#61313](https://github.com/PaddlePaddle/Paddle/pull/61313), [#61840](https://github.com/PaddlePaddle/Paddle/pull/61840), [#60614](https://github.com/PaddlePaddle/Paddle/pull/60614), [#60234](https://github.com/PaddlePaddle/Paddle/pull/60234), [#64813](https://github.com/PaddlePaddle/Paddle/pull/64813), [#61606](https://github.com/PaddlePaddle/Paddle/pull/61606), [#63405](https://github.com/PaddlePaddle/Paddle/pull/63405), [#64334](https://github.com/PaddlePaddle/Paddle/pull/64334), [#60504](https://github.com/PaddlePaddle/Paddle/pull/60504) -- The complex kernels of PaddlePaddle PHI operator library have been further enhanced, and a total of 40+ complex kernels have been added. [#55380](https://github.com/PaddlePaddle/Paddle/pull/55380), [#56349](https://github.com/PaddlePaddle/Paddle/pull/56349), [#56412](https://github.com/PaddlePaddle/Paddle/pull/56412), [#56323](https://github.com/PaddlePaddle/Paddle/pull/56323), [#56723](https://github.com/PaddlePaddle/Paddle/pull/56723), [#56457](https://github.com/PaddlePaddle/Paddle/pull/56457), [#56903](https://github.com/PaddlePaddle/Paddle/pull/56903)[#56914](https://github.com/PaddlePaddle/Paddle/pull/56914), [#57116](https://github.com/PaddlePaddle/Paddle/pull/57116), [#56048](https://github.com/PaddlePaddle/Paddle/pull/56048), [#57244](https://github.com/PaddlePaddle/Paddle/pull/57244), [#57639](https://github.com/PaddlePaddle/Paddle/pull/57639), [#57638](https://github.com/PaddlePaddle/Paddle/pull/57638), [#57540](https://github.com/PaddlePaddle/Paddle/pull/57540), [#58545](https://github.com/PaddlePaddle/Paddle/pull/58545), [#58336](https://github.com/PaddlePaddle/Paddle/pull/58336), [#58532](https://github.com/PaddlePaddle/Paddle/pull/58532), [#58839](https://github.com/PaddlePaddle/Paddle/pull/58839), [#59079](https://github.com/PaddlePaddle/Paddle/pull/59079), [#59277](https://github.com/PaddlePaddle/Paddle/pull/59277), [#59122](https://github.com/PaddlePaddle/Paddle/pull/59122), [#57058](https://github.com/PaddlePaddle/Paddle/pull/57058) +### Performance Optimization -- Optimized and added XPU kernels for some operators, and enhanced the support for data types such as bfloat16 on XPU kernel. [#54478](https://github.com/PaddlePaddle/Paddle/pull/54478), [#57740](https://github.com/PaddlePaddle/Paddle/pull/57740), [#58346](https://github.com/PaddlePaddle/Paddle/pull/58346), [#58456](https://github.com/PaddlePaddle/Paddle/pull/58456), [#58662](https://github.com/PaddlePaddle/Paddle/pull/58662), [#59066](https://github.com/PaddlePaddle/Paddle/pull/59066), [#59263](https://github.com/PaddlePaddle/Paddle/pull/59263)), [#59375](https://github.com/PaddlePaddle/Paddle/pull/59375), [#59505](https://github.com/PaddlePaddle/Paddle/pull/59505), [#59653](https://github.com/PaddlePaddle/Paddle/pull/59653), [#55001](https://github.com/PaddlePaddle/Paddle/pull/55001), [#57272](https://github.com/PaddlePaddle/Paddle/pull/57272), [#56169](https://github.com/PaddlePaddle/Paddle/pull/56169), [#59454](https://github.com/PaddlePaddle/Paddle/pull/59454), [#59480](https://github.com/PaddlePaddle/Paddle/pull/59480), [#55914](https://github.com/PaddlePaddle/Paddle/pull/55914), [#54758](https://github.com/PaddlePaddle/Paddle/pull/54758), [#54827](https://github.com/PaddlePaddle/Paddle/pull/54827), [#58364](https://github.com/PaddlePaddle/Paddle/pull/58364), [#58419](https://github.com/PaddlePaddle/Paddle/pull/58419), [#58982](https://github.com/PaddlePaddle/Paddle/pull/58982), [#57216](https://github.com/PaddlePaddle/Paddle/pull/57216), [#59166](https://github.com/PaddlePaddle/Paddle/pull/59166), [#55033](https://github.com/PaddlePaddle/Paddle/pull/55033), [#55375](https://github.com/PaddlePaddle/Paddle/pull/55375), [#58805](https://github.com/PaddlePaddle/Paddle/pull/58805), [#59389](https://github.com/PaddlePaddle/Paddle/pull/59389), [#57077](https://github.com/PaddlePaddle/Paddle/pull/57077), [#55166](https://github.com/PaddlePaddle/Paddle/pull/55166), [#56773](https://github.com/PaddlePaddle/Paddle/pull/56773) +- In order to reduce the bubble in pipeline parallel, support the reverse computation of parameter and activation splitting in backward, and add zero bubble pipeline scheduling strategy to improve the training performance. [#62865](https://github.com/PaddlePaddle/Paddle/pull/62865), [#62737](https://github.com/PaddlePaddle/Paddle/pull/62737), [#64534](https://github.com/PaddlePaddle/Paddle/pull/64534), +- To improve the performance of sequence parallel, perform fusion on related communication operations and computation operations, and optimize redundant transopse operations. [#64807](https://github.com/PaddlePaddle/Paddle/pull/64807), [#63948](https://github.com/PaddlePaddle/Paddle/pull/63948), [#64316](https://github.com/PaddlePaddle/Paddle/pull/64316), [#64119](https://github.com/PaddlePaddle/Paddle/pull/64119) +- Optimize the time consumption of auto parallel graph optimization for static graphs, to reduce the delay from the start of training to the completion of the first step. [#59912](https://github.com/PaddlePaddle/Paddle/pull/59912), [#61817](https://github.com/PaddlePaddle/Paddle/pull/61817), [#60022](https://github.com/PaddlePaddle/Paddle/pull/60022), [#60125](https://github.com/PaddlePaddle/Paddle/pull/60125) +- Optimize the time consumption of related communication operations in hybrid parallel scenarios. [#62157](https://github.com/PaddlePaddle/Paddle/pull/62157), [#61622](https://github.com/PaddlePaddle/Paddle/pull/61622) +- Optimize the redundant video memory consumption of parameters under the auto parallel dynamic-to-static. [#62746](https://github.com/PaddlePaddle/Paddle/pull/62746) +- Improve the hybrid precision training function of auto parallel, support the setting of local auto_cast and black/white list, support master grad function, and adapt to different parallel strategies. [60158](https://github.com/PaddlePaddle/Paddle/pull/60158), [#59987](https://github.com/PaddlePaddle/Paddle/pull/59987), [#62629](https://github.com/PaddlePaddle/Paddle/pull/62629), [#60385](https://github.com/PaddlePaddle/Paddle/pull/60385), [#62015](https://github.com/PaddlePaddle/Paddle/pull/62015), [#60514](https://github.com/PaddlePaddle/Paddle/pull/60514), [#61221](https://github.com/PaddlePaddle/Paddle/pull/61221), [#60779](https://github.com/PaddlePaddle/Paddle/pull/60779), [#63228](https://github.com/PaddlePaddle/Paddle/pull/63228) +- Optimize non-essential casts caused by type promotion and amp to improve performance. [#63293](https://github.com/PaddlePaddle/Paddle/pull/63293), [#63228](https://github.com/PaddlePaddle/Paddle/pull/63228) -- Added some operators for optimizing large model training and inference performance. [#55758](https://github.com/PaddlePaddle/Paddle/pull/55758), [#54998](https://github.com/PaddlePaddle/Paddle/pull/54998), [#55400](https://github.com/PaddlePaddle/Paddle/pull/55400), [#54630](https://github.com/PaddlePaddle/Paddle/pull/54630), [#55969](https://github.com/PaddlePaddle/Paddle/pull/55969), [#55026](https://github.com/PaddlePaddle/Paddle/pull/55026), [#58986](https://github.com/PaddlePaddle/Paddle/pull/58986) +### Upgrade Static Graph Auto Parallel Architecture -- Improved mechanism of Tensor Strided in the operator library. [#59422](https://github.com/PaddlePaddle/Paddle/pull/59422), [#59325](https://github.com/PaddlePaddle/Paddle/pull/59325), [#56863](https://github.com/PaddlePaddle/Paddle/pull/56863), [#56882](https://github.com/PaddlePaddle/Paddle/pull/56882), [#56947](https://github.com/PaddlePaddle/Paddle/pull/56947) +- Based on the new generation of Intermediate Representation(PIR), add the new DistDialect, natively supporting DistAttr and DistTensor in computation graph representation, and realizing the direct binding of distributed attributes between tensor or operator, which making the auto-parallel architecture more simple and unified. [#63828](https://github.com/PaddlePaddle/Paddle/pull/63828), [#64299](https://github.com/PaddlePaddle/Paddle/pull/64299), [#63870](https://github.com/PaddlePaddle/Paddle/pull/63870), [#64144](https://github.com/PaddlePaddle/Paddle/pull/64144), [#62524](https://github.com/PaddlePaddle/Paddle/pull/62524), [#62630](https://github.com/PaddlePaddle/Paddle/pull/62630), [#62897](https://github.com/PaddlePaddle/Paddle/pull/62897), [#60478](https://github.com/PaddlePaddle/Paddle/pull/60478), [#60574](https://github.com/PaddlePaddle/Paddle/pull/60574), [#63876](https://github.com/PaddlePaddle/Paddle/pull/63876), [#63798](https://github.com/PaddlePaddle/Paddle/pull/63798), [#62560](https://github.com/PaddlePaddle/Paddle/pull/62560), [#63676](https://github.com/PaddlePaddle/Paddle/pull/63676) +- Improve APIs such as shard_tensor, reshard, and to_static, to support users to convert the dynamic graph model networking directly into PIR static computation graph for better performance. [#62945](https://github.com/PaddlePaddle/Paddle/pull/62945), [#62356](https://github.com/PaddlePaddle/Paddle/pull/62356), [#60175](https://github.com/PaddlePaddle/Paddle/pull/60175), [#62654](https://github.com/PaddlePaddle/Paddle/pull/62654), [#63347](https://github.com/PaddlePaddle/Paddle/pull/63347) +- Optimize the auto-parallel graph optimization compilation process, and reduce the compilation and optimization time of static graphs by refactoring and optimizing the procedure of computation graph parallelization and communication resolution. [#64137](https://github.com/PaddlePaddle/Paddle/pull/64137), [#62201](https://github.com/PaddlePaddle/Paddle/pull/62201), [#64143](https://github.com/PaddlePaddle/Paddle/pull/64143), [#62560](https://github.com/PaddlePaddle/Paddle/pull/62560) +- Optimize the procedure of the SPMD derivation in static graphs to achieve the consistency results under dynamic-static graphs, which improves the unity and stability of the architecture. [#62659](https://github.com/PaddlePaddle/Paddle/pull/62659), [#62547](https://github.com/PaddlePaddle/Paddle/pull/62547), [#63117](https://github.com/PaddlePaddle/Paddle/pull/63117), [#63434](https://github.com/PaddlePaddle/Paddle/pull/63434), [#63770](https://github.com/PaddlePaddle/Paddle/pull/63770), [#64361](https://github.com/PaddlePaddle/Paddle/pull/64361), [#63073](https://github.com/PaddlePaddle/Paddle/pull/63073) +- Upgrade the implementation of Reshard conversion in static graphs, and use consistent conversion rules under dynamic-static graphs to ensure the consistency of the execution logic and results of tensor reshard conversion in dynamic-static graphs, so as to improve user experience. [#62718](https://github.com/PaddlePaddle/Paddle/pull/62718), [#62694](https://github.com/PaddlePaddle/Paddle/pull/62694), [#60215](https://github.com/PaddlePaddle/Paddle/pull/60215), [#63362](https://github.com/PaddlePaddle/Paddle/pull/63362), [#63072](https://github.com/PaddlePaddle/Paddle/pull/63072), [#63962](https://github.com/PaddlePaddle/Paddle/pull/63962), [#64223](https://github.com/PaddlePaddle/Paddle/pull/64223), [#61796](https://github.com/PaddlePaddle/Paddle/pull/61796), [#64465](https://github.com/PaddlePaddle/Paddle/pull/64465), [#64623](https://github.com/PaddlePaddle/Paddle/pull/64623), [#64418](https://github.com/PaddlePaddle/Paddle/pull/64418) -- Optimized function implementation and template function in some kernels to reduce size of complied library package. [#57083](https://github.com/PaddlePaddle/Paddle/pull/57083), [#57299](https://github.com/PaddlePaddle/Paddle/pull/57299), [#57261](https://github.com/PaddlePaddle/Paddle/pull/57261), [#57290](https://github.com/PaddlePaddle/Paddle/pull/57290), [#57118](https://github.com/PaddlePaddle/Paddle/pull/57118), [#57551](https://github.com/PaddlePaddle/Paddle/pull/57551), [#57509](https://github.com/PaddlePaddle/Paddle/pull/57509), [#57558](https://github.com/PaddlePaddle/Paddle/pull/57558), [#57064](https://github.com/PaddlePaddle/Paddle/pull/57064), [#57365](https://github.com/PaddlePaddle/Paddle/pull/57365), [#57327](https://github.com/PaddlePaddle/Paddle/pull/57327), [#57603](https://github.com/PaddlePaddle/Paddle/pull/57603), [#57671](https://github.com/PaddlePaddle/Paddle/pull/57671), [#57672](https://github.com/PaddlePaddle/Paddle/pull/57672), [#57631](https://github.com/PaddlePaddle/Paddle/pull/57631), [#57082](https://github.com/PaddlePaddle/Paddle/pull/57082), [#57721](https://github.com/PaddlePaddle/Paddle/pull/57721), [#57823](https://github.com/PaddlePaddle/Paddle/pull/57823), [#57821](https://github.com/PaddlePaddle/Paddle/pull/57821), [#57815](https://github.com/PaddlePaddle/Paddle/pull/57815), [#57822](https://github.com/PaddlePaddle/Paddle/pull/57822), [#57541](https://github.com/PaddlePaddle/Paddle/pull/57541), [#57817](https://github.com/PaddlePaddle/Paddle/pull/57817), [#57838](https://github.com/PaddlePaddle/Paddle/pull/57838) +### Automatic Search and Tuning of Training Strategies -#### Fixed bug +In order to improve the ease of use of the training strategy automatic search and tuning tool (AutoTuner), support user-defined search items, support for setting the priority of search items, and support for user-configured illegal strategy combinations, to comprehensively enhance the error reporting information in the runtime and post-run logs, and support for AutoTuner on NPU devices. [#60101](https://github.com/PaddlePaddle/Paddle/pull/60101), [#60294](https://github.com/PaddlePaddle/Paddle/pull/60294), [#61898](https://github.com/PaddlePaddle/Paddle/pull/61898), [#60248](https://github.com/PaddlePaddle/Paddle/pull/60248), [#60417](https://github.com/PaddlePaddle/Paddle/pull/60417), [#60954](https://github.com/PaddlePaddle/Paddle/pull/60954), [#61499](https://github.com/PaddlePaddle/Paddle/pull/61499), [#62724](https://github.com/PaddlePaddle/Paddle/pull/62724), [#60954](https://github.com/PaddlePaddle/Paddle/pull/60954), [#63693](https://github.com/PaddlePaddle/Paddle/pull/63693), [#62853](https://github.com/PaddlePaddle/Paddle/pull/62853), [#62984](https://github.com/PaddlePaddle/Paddle/pull/62984) -- Fixed some bugs with CUDA 12 adaptation of the PaddlePaddle framework. [#54640](https://github.com/PaddlePaddle/Paddle/pull/54640), [#57820](https://github.com/PaddlePaddle/Paddle/pull/57820), [#58958](https://github.com/PaddlePaddle/Paddle/pull/58958), [#58179](https://github.com/PaddlePaddle/Paddle/pull/58179), [#55594](https://github.com/PaddlePaddle/Paddle/pull/55594) +## Cuda Training Performance Optimization -### CUDA +This upgrade achieves the improvement of large model training efficiency from multiple perspectives, such as operator computation efficiency, distributed communication optimization, and video memory optimization. -#### New features +### Function Improvements -- Added debugging class API paddle.amp.debugging.check_check_numerics. Calculated and returned number of outliers (NaN, Inf) and zero elements in this Tensor value. [#54301](https://github.com/PaddlePaddle/Paddle/pull/54301) -- Added fused_rope fusion operator to accelerate LLaMA class large model training.[#54351](https://github.com/PaddlePaddle/Paddle/pull/54351) -- Updated CUDNN Frontend API version to v0.9.1 and added fused_scale_bias_add_relu fusion operator to accelerate ResNet networks. Note this feature is in experimental period and is disabled by default. [#58367](https://github.com/PaddlePaddle/Paddle/pull/58367), [#54949](https://github.com/PaddlePaddle/Paddle/pull/54949), [#58504](https://github.com/PaddlePaddle/Paddle/pull/58504) -- Based on Flash-Attention v2, added Tensor-like Mask function support. Inverse operator supports deterministic computation for debugging. [#57276](https://github.com/PaddlePaddle/Paddle/pull/57276), [#56363](https://github.com/PaddlePaddle/Paddle/pull/56363) -- Modified sparse conv3d backend implementation to support 2d shapes, avoiding front-end reshape overhead. [#54707](https://github.com/PaddlePaddle/Paddle/pull/54707) -- Added matmul_int8 operator. ([#55228](https://github.com/PaddlePaddle/Paddle/pull/55228)) +- Enhance the FlashAttention operator function, including support for NVIDIA SM90 GPU compilation, support for Group Query Attention, support for cuDNN access, support for QKV-packed form inputs, and so on. [#59820](https://github.com/PaddlePaddle/Paddle/pull/59820)，[#60776](https://github.com/PaddlePaddle/Paddle/pull/60776)，[#58680](https://github.com/PaddlePaddle/Paddle/pull/58680)，[#63289](https://github.com/PaddlePaddle/Paddle/pull/63289) +- In the Repeat_interleave operator, add support for BFloat16 data type. [#61854](https://github.com/PaddlePaddle/Paddle/pull/61854) +- For the issues of many interface parameters of ResNet-like models such as fused_scale_bias_add_relu, fused_scale_bias_relu_conv_bn, and fused_dconv_drelu_dbn, and the ease of use of operators, add the fuse_resunit pass, to support automatic fusion of the abovementioned operators, to achieve generic performance optimization. ([#59771](https://github.com/PaddlePaddle/Paddle/pull/59771)) -#### Function optimization +### Performance Improvement -- Optimized CUDA Graph’s support for random number operators.[#58310](https://github.com/PaddlePaddle/Paddle/pull/58310) -- Enhanced automatic mixed-precision training default functionality, including: - - Optimizing the experience of using automatic mixed precision training interface.[#58152](https://github.com/PaddlePaddle/Paddle/pull/58152),[#55364](https://github.com/PaddlePaddle/Paddle/pull/55364),[#57903](https://github.com/PaddlePaddle/Paddle/pull/57903) - - Added matrix computation class operators such as fused_attention, fused_feedforward, and fused_gemm_epilogue to framework's default whitelist, and unified default black and white list settings for dynamic and static graphs. [#55373](https://github.com/PaddlePaddle/Paddle/pull/55373), [#55713](https://github.com/PaddlePaddle/Paddle/pull/55713) - - The argsort, dist, erfinv, nanmedian, poisson operators and lamb optimizer operators support FP16 and BF16 low precision computing. [#51662](https://github.com/PaddlePaddle/Paddle/pull/51662), [#55105](https://github.com/PaddlePaddle/Paddle/pull/55105), [#55287](https://github.com/PaddlePaddle/Paddle/pull/55287), [#55824](https://github.com/PaddlePaddle/Paddle/pull/55824), [#56056](https://github.com/PaddlePaddle/Paddle/pull/56056), [#56184](https://github.com/PaddlePaddle/Paddle/pull/56184), [#55641](https://github.com/PaddlePaddle/Paddle/pull/55641) - - Fixed elementwise_max operator low-precision implementation. Changed to use FP32 type for numerical computing, and reduce precision loss. [#54799](https://github.com/PaddlePaddle/Paddle/pull/54799) - - Changed temporary result Tensor needed for Reduce class operator to FP32 type, to avoid precision loss caused by converting intermediate result to low precision. [#55709](https://github.com/PaddlePaddle/Paddle/pull/55709)) -- Optimized GPU codes for flip, roll & roll_grad, index_put & index_put_grad, etc. Removed unnecessary C++ templates to optimize compilation time and reduce compiled binary size without performance degradation. [#57309](https://github.com/PaddlePaddle/Paddle/pull/57309), [#57525](https://github.com/PaddlePaddle/Paddle/pull/57525) -- For the bernoulli operator, added a check on legitimacy of input probabilities. [#59174](https://github.com/PaddlePaddle/Paddle/pull/59174) +- To address the problem of large GPU memory consumption during the computation of SwiGLU activation module of the Llama models, add the SwiGLU fusion operator to save the memory consumption of intermediate variables, thus reducing the memory overhead during the training process of the large model, and reducing the recomputation to improve the performance. The performance of the Llama-70B model is improved by 9%. [#61508](https://github.com/PaddlePaddle/Paddle/pull/61508) +- To address the problem of higher percentage of communications in Sequence Parallel, realize the overlap between Sequence Parallel reverse process communication and Matmul computation, saving the end-to-end time consumption and improving the end-to-end performance of large model training scenarios by 1%~2%. [#62284](https://github.com/PaddlePaddle/Paddle/pull/62284)，[#63531](https://github.com/PaddlePaddle/Paddle/pull/63531) +- For the problem of slow training speed due to the need to divide by nranks after sharding reverse communications, support the fusion of reverse communication and division by nranks operation, and support the mode of ReduceScatter Average, to improve the performance of large model training. [#62623](https://github.com/PaddlePaddle/Paddle/pull/62623) +- For the problem of jitter training speed caused by the input data broadcasting process of the tensor model parallel process, fix the unnecessary synchronization between CPU and GPU in the data broadcasting, to ensure the stability of the training speed. [#60816](https://github.com/PaddlePaddle/Paddle/pull/60816) +- For the problem of low training speed due to the long parallel P2P communication time of pipelined models, realize the overlap of P2P communication and forward-backward computation. The end-to-end training performance of large models is improved by 2%~3%. [#61935](https://github.com/PaddlePaddle/Paddle/pull/61935),[#62051](https://github.com/PaddlePaddle/Paddle/pull/62051，[#62051](https://github.com/PaddlePaddle/Paddle/pull/62051)) +- For the problem of low inefficiency of bias gradient computation of fused_linear_param_grad_add operator, optimize the computation efficiency of bias gradient computation, and improve the end-to-end training performance of large model by 0.2%. [#63114](https://github.com/PaddlePaddle/Paddle/pull/63114) +- For the problem of long time-consuming parameter broadcasting process after the end of sharding reverse computation, implement the overlap between parameter broadcasting and next step computation. As a result, the end-to-end training performance of large model is improved by more than 2%. [#63945](https://github.com/PaddlePaddle/Paddle/pull/63945) +- To address the problem that the gradient occupies too much video memory during the pipelined parallel training, as a result of slow training speed due to the introduction of multiple computations, we have implemented the gradient dynamic release technique, to improve the end-to-end training performance of large models by 3.4%. [#59739](https://github.com/PaddlePaddle/Paddle/pull/59739) -#### Performance optimization +### Bug Fixing -- Optimized BroadcastKernel's support for large Tensor. Change to call INT32 version implementation for multiple times for large Tensor Sharding, improving operator performance by 7.27x. [#57313](https://github.com/PaddlePaddle/Paddle/pull/57313), [#57996](https://github.com/PaddlePaddle/Paddle/pull/57996) -- Optimized performance of Tensor save interface by copying the Tensor to CPU and then converting to numpy, to avoid overhead of automatically converting the Tensor to a continuous Tensor when Tensor is not continuous. [#57040](https://github.com/PaddlePaddle/Paddle/pull/57040) +- Fix the problem of StreamSafeCUDAAllocator CUDA Event resource leakage, as a result of slowdown of large model training. [#64621](https://github.com/PaddlePaddle/Paddle/pull/64621) +- Fix the bug of reverse calculation error of fused_rotary_position_embedding operator. [#60217](https://github.com/PaddlePaddle/Paddle/pull/60217) +- Fix the bug that customized operators cannot control the calculation accuracy by black and white lists in AMP scenarios. [#60052](https://github.com/PaddlePaddle/Paddle/pull/60052) +- Fix the bug that operators such as add_, and divide_ natively supporting operations with different data types have unanticipated type boosting when type boosting occurs. [#64302](https://github.com/PaddlePaddle/Paddle/pull/64302) -#### Bug Fix +## Distributed Strategy Enhancements -- Fixed bug of memmory_efficient_attention operator supporting the sm_90. [#58070](https://github.com/PaddlePaddle/Paddle/pull/58070) -- Fixed the NaN problem of softmax operator when axis=-1 and length is greater than 100000. [#57851](https://github.com/PaddlePaddle/Paddle/pull/57851) -- Fixed bug of GPU access error in some cases for set_constant operator. [#59905](https://github.com/PaddlePaddle/Paddle/pull/59905) -- Fixed GPU storage read/write contention issue in fast implementation version of layer_norm operator. [#56435](https://github.com/PaddlePaddle/Paddle/pull/56435) +Focus on strengthening the functional experience of PaddlePaddle dynamic graph distributed computing, and make various functional improvements to parallel strategies such as AutoTuner, pipeline parallel, and sharding, and enhance the flexibility of large model training. Add the features such as Flash Attention Mask, which significantly reduce the video memory usage of large model training, especially long-sequence training, improve training performance, and provide stronger capability support for large model training. In addition, several bugs and potential security risks have been fixed, which has significantly improved the overall stability of the system. -### Expanded Compiler Infrastructure for Neural Networks (CINN) +### Function Optimization -In this update, PaddlePaddle CINN focuses on optimization of architecture and comprehensive expansion of its capabilities. In view of increasing demand for dynamic shapes for large models, effective operation and optimization strategies of compiler under dynamic shapes are initially explored and implemented. -At the architectural level, Python DSL is introduced, significantly improving CINN's development convenience and Debug capability and enabling developers to write and debug codes more efficiently. Meanwhile, logic of Schedule has been refactored to be dominated by GroupSchedule, enabling more general and stable optimization strategies at operator Group level. In order to enhance stability of CINN, a strong constraint component is explored and introduced. This can effectively reduce uncertainties and potential errors in the system. In addition, historical tool classes and software structure of CINN are systematically organized, optimized and improved, to further enhance readability and maintainability of codes. In terms of integration with other PaddlePaddle components, tight integration of CINN with PIR and Paddle has been further strengthened, making compiler more coherent with overall PaddlePaddle framework. This improvement not only enhances performance of the compiler, but also provides developers with a smoother and more unified development experience. +- Optimize the search space of Autotuner, which significantly improves the performance of search. [#62608](https://github.com/PaddlePaddle/Paddle/pull/62608) +- For the problem of pipeline parallel that the training may be wrong due to the checking of sending type in the eval process, add the training configuration, to skip the redundant receiving check of pipelined sending, featuring higher flexibility and better performance. [#63001](https://github.com/PaddlePaddle/Paddle/pull/63001) +- In the dynamic graph pipeline parallel, add the checking of the size and type of the sent and received data, and add the error message, making the robustness and debuggability better. [#59405](https://github.com/PaddlePaddle/Paddle/pull/59405) +- Support the settings of multiple loss functions with returning multiple losses in dynamic graph pipeline, which improves the flexibility of dynamic graph pipeline. [#63167](https://github.com/PaddlePaddle/Paddle/pull/63167) +- In the dynamic graph pipeline, add the pipeline cache clearing configuration option, to clear the cache sent and received in the pipeline in time to better support dynamic batchsize training. [#62277](https://github.com/PaddlePaddle/Paddle/pull/62277) +- For the problem that the sharding stage3 strategy cannot be aligned bit by bit, replace the unordered set with OrderedSet to avoid the error caused by the accumulation order, as a result of alignment bit by bit after fixing. [#60085](https://github.com/PaddlePaddle/Paddle/pull/60085) +- In order to further reduce the video memory usage in sequence parallel, add a new method of recalculating allgather, to reduce the video memory size of the activation of allgather. [#64244](https://github.com/PaddlePaddle/Paddle/pull/64244) -#### Compatibility upgrade +### New Features for Dynamic Graphs -- Updated storage read interface to be compatible with Paddle 2.0. [#55836](https://github.com/PaddlePaddle/Paddle/pull/55836) -- Updated relu6 Op Mapper compatibility. [#55611](https://github.com/PaddlePaddle/Paddle/pull/55611) +- For the search space of autotuner, add a new search dimension of refined recompute, which makes the search result more accurate and the threshold of model tuning lower. [#62430](https://github.com/PaddlePaddle/Paddle/pull/62430) +- For the problem of limiting the training batch size in virtual pipeline parallel, modify the pipeline scheduling method, to flexibly set the batch size, so as to support more flexible batch size. [#61561](https://github.com/PaddlePaddle/Paddle/pull/61561),[#60314](https://github.com/PaddlePaddle/Paddle/pull/60134) +- In order to solve the problem that the video memory occupation of the mask is a quadratic complexity with low performance in sequence length when using flash attention with a mask, the memory complexity of the mask is reduced from the quadrature of the sequence length to the first square by using the sparse mask, to optimize the memory of the mask. This reduces the number of storage accesses. Meanwhile, use share memory to accelerate memory access, greatly improving the performance. [#62029](https://github.com/PaddlePaddle/Paddle/pull/62029) +- Add the dynamic graph sharding parallel strategy, to improve the communications and computation overlap function, to improve the performance of the training process. [#60455](https://github.com/PaddlePaddle/Paddle/pull/60455) -#### Modification deprecation +### Communication Library Function Optimization -- Removed old Schedule form. [#55566](https://github.com/PaddlePaddle/Paddle/pull/55566),[#55391](https://github.com/PaddlePaddle/Paddle/pull/55391) -- Removed some obsolete tests. [#56245](https://github.com/PaddlePaddle/Paddle/pull/56245),[#57987](https://github.com/PaddlePaddle/Paddle/pull/57987) -- Removed the remove_nested_block Visitor tool that no longer works. [#56972](https://github.com/PaddlePaddle/Paddle/pull/56972) -- Removed other useless codes. [#55413](https://github.com/PaddlePaddle/Paddle/pull/55413) +- Enhance the functionality of the NCCL communication library to support the initialization of customized NCCL libraries by passing additional initialization parameters during initialization. [#62193](https://github.com/PaddlePaddle/Paddle/pull/62193) +- Add the NCCL library path search function to support more flexible NCCL library search methods. [#62492](https://github.com/PaddlePaddle/Paddle/pull/62492) -#### New features +### Bug Fixing -- Added CINN paddle.framework.core.is_run_with_cinn() API on the PaddlePaddle side. [#54355](https://github.com/PaddlePaddle/Paddle/pull/54355) -- Added CINN related operator logics, including various combinatorial operator’s disassembly logic. [#56072](https://github.com/PaddlePaddle/Paddle/pull/56072),[#58210](https://github.com/PaddlePaddle/Paddle/pull/58210),[#58502](https://github.com/PaddlePaddle/Paddle/pull/58502), [#58591](https://github.com/PaddlePaddle/Paddle/pull/58591), [#58981](https://github.com/PaddlePaddle/Paddle/pull/58981), [#59135](https://github.com/PaddlePaddle/Paddle/pull/59135), [#59274](https://github.com/PaddlePaddle/Paddle/pull/59274), [#59306](https://github.com/PaddlePaddle/Paddle/pull/59306), [#59202](https://github.com/PaddlePaddle/Paddle/pull/59202), [#59176](https://github.com/PaddlePaddle/Paddle/pull/59176), [#59534](https://github.com/PaddlePaddle/Paddle/pull/59534), [#59713](https://github.com/PaddlePaddle/Paddle/pull/59713), [#59798](https://github.com/PaddlePaddle/Paddle/pull/59798); Supports bf16, amp and other forms [#54399](https://github.com/PaddlePaddle/Paddle/pull/54399), [#54368](https://github.com/PaddlePaddle/Paddle/pull/54368), [#54608](https://github.com/PaddlePaddle/Paddle/pull/54608); Supports operator zero-dimensional capability [#54892](https://github.com/PaddlePaddle/Paddle/pull/54892), [#54919](https://github.com/PaddlePaddle/Paddle/pull/54919), [#54907](https://github.com/PaddlePaddle/Paddle/pull/54907), [#54966](https://github.com/PaddlePaddle/Paddle/pull/54966) -- Supports CINN and PaddlePaddle PIR, and combinator operator junction operation mode, so new PIR and CINN operation is integrated. [#54732](https://github.com/PaddlePaddle/Paddle/pull/54732), [#56074](https://github.com/PaddlePaddle/Paddle/pull/56074), [#58216](https://github.com/PaddlePaddle/Paddle/pull/58216), [#55680](https://github.com/PaddlePaddle/Paddle/pull/55680), [#56302](https://github.com/PaddlePaddle/Paddle/pull/56302), [#59037](https://github.com/PaddlePaddle/Paddle/pull/59037), [#55186](https://github.com/PaddlePaddle/Paddle/pull/55186), [#58641](https://github.com/PaddlePaddle/Paddle/pull/58641) -- There are strongly constrained components to stabilize CINN changes. [#58719](https://github.com/PaddlePaddle/Paddle/pull/58719), [#59309](https://github.com/PaddlePaddle/Paddle/pull/59309), [#58993](https://github.com/PaddlePaddle/Paddle/pull/58993) -- Added Group Schedule related CINN architecture process. [#58399](https://github.com/PaddlePaddle/Paddle/pull/58399), [#56444](https://github.com/PaddlePaddle/Paddle/pull/56444) -- Added CUTLASS, error handling, and NVRTC Cubin Fmad options to CINN architecture functions preliminarily. [#58079](https://github.com/PaddlePaddle/Paddle/pull/58079), [#57198](https://github.com/PaddlePaddle/Paddle/pull/57198), [#58794](https://github.com/PaddlePaddle/Paddle/pull/58794) -- Added Python interface language for CINN. [#57731](https://github.com/PaddlePaddle/Paddle/pull/57731), [#57515](https://github.com/PaddlePaddle/Paddle/pull/57515), [#57644](https://github.com/PaddlePaddle/Paddle/pull/57644), [#57981](https://github.com/PaddlePaddle/Paddle/pull/57981), [#58009](https://github.com/PaddlePaddle/Paddle/pull/58009) -- Added dynamic shape functionality for CINN to cover ASTGen to generate dynamic shape symbols, to replace the ISL to generate dynamic shape signals [#56360](https://github.com/PaddlePaddle/Paddle/pull/56360), [#57207](https://github.com/PaddlePaddle/Paddle/pull/57207), [#57454](https://github.com/PaddlePaddle/Paddle/pull/57454); Added Bucket Conditional Compilation functionality [#59165](https://github.com/PaddlePaddle/Paddle/pull/59165); Added Schedule, Device, and IR level support for dynamic shape [#58988](https://github.com/PaddlePaddle/Paddle/pull/58988), [#59493](https://github.com/PaddlePaddle/Paddle/pull/59493), [#58717](https://github.com/PaddlePaddle/Paddle/pull/58717), [#58602](https://github.com/PaddlePaddle/Paddle/pull/58602), [#59196](https://github.com/PaddlePaddle/Paddle/pull/59196) -- Supports CINN Group Schedule operator – at Group level, perform more general and stable Schedule optimization. [#56122](https://github.com/PaddlePaddle/Paddle/pull/56122), [#57777](https://github.com/PaddlePaddle/Paddle/pull/57777), [#57569](https://github.com/PaddlePaddle/Paddle/pull/57569) +- Fix the problem of dbias_out space application of fused_linear_param_grad_add_kernel operator, and add the gradient address checking logic to make the error message easier to debug. [#363433](https://github.com/PaddlePaddle/Paddle/pull/63433),[#64460](https://github.com/PaddlePaddle/Paddle/pull/64460) +- Fix the problem that the sharding policy does not scale the gradient when comm_overlap is turned off in the support of reduce_avg operation. [#62702](https://github.com/PaddlePaddle/Paddle/pull/62702) +- Fix the bug related to fusion in the calculation order of main grad in Stage2. [#59142](https://github.com/PaddlePaddle/Paddle/pull/59142) +- Fix the bug that the switch attribute cannot be found when reduce_avg communication operation is turned on under the sharding strategy. [#62502](https://github.com/PaddlePaddle/Paddle/pull/62502) +- Fix the problem of setting stop_gradient=True for some parameters when Sharding stage1 training supports non-training parameter training. [#62616](https://github.com/PaddlePaddle/Paddle/pull/62616) +- Fix the bug of message printing when TCP is turned off, to prevent misleading users. [#62631](https://github.com/PaddlePaddle/Paddle/pull/62631) +- Fix the DataParallel training problem and solve multi-card training error when some gradients are not initialized and segmentation fault error occurs in data parallel training. [#62299](https://github.com/PaddlePaddle/Paddle/pull/62299) +- For the scenario of turning on sequence parallel, fix the bug caused by weight freezing in some models. [#63596](https://github.com/PaddlePaddle/Paddle/pull/63596) +- Fix some bugs for autotuner scenarios with single dp. [#60757](https://github.com/PaddlePaddle/Paddle/pull/60757) +- Fix aadiff bug of streaming parallel strategy. ([#64716](https://github.com/PaddlePaddle/Paddle/pull/64716)) +- Remove some distributed unit tests. ([#62762](https://github.com/PaddlePaddle/Paddle/pull/62762)) -#### Function optimization +### Security Risk Fixing -- Enriched or improved operator functionality, including improvements to various operator processes such as Repair Reverse, FP16, Infershape, Operator Single Test, etc. [#56320](https://github.com/PaddlePaddle/Paddle/pull/56320), [#56845](https://github.com/PaddlePaddle/Paddle/pull/56845), [#54939](https://github.com/PaddlePaddle/Paddle/pull/54939),[#54378](https://github.com/PaddlePaddle/Paddle/pull/54378),[#55321](https://github.com/PaddlePaddle/Paddle/pull/55321),[#55336](https://github.com/PaddlePaddle/Paddle/pull/55336),[#55337](https://github.com/PaddlePaddle/Paddle/pull/55337),[#55442](https://github.com/PaddlePaddle/Paddle/pull/55442),[#55470](https://github.com/PaddlePaddle/Paddle/pull/55470),[#55489](https://github.com/PaddlePaddle/Paddle/pull/55489),[#55510](https://github.com/PaddlePaddle/Paddle/pull/55510),[#55547](https://github.com/PaddlePaddle/Paddle/pull/55547),[#55505](https://github.com/PaddlePaddle/Paddle/pull/55505),[#55563](https://github.com/PaddlePaddle/Paddle/pull/55563),[#54280](https://github.com/PaddlePaddle/Paddle/pull/54280),[#59650](https://github.com/PaddlePaddle/Paddle/pull/59650),[#54862](https://github.com/PaddlePaddle/Paddle/pull/54862),[#55135](https://github.com/PaddlePaddle/Paddle/pull/55135),[#55292](https://github.com/PaddlePaddle/Paddle/pull/55292),[#55333](https://github.com/PaddlePaddle/Paddle/pull/55333),[#55316](https://github.com/PaddlePaddle/Paddle/pull/55316),[#55379](https://github.com/PaddlePaddle/Paddle/pull/55379),[#55326](https://github.com/PaddlePaddle/Paddle/pull/55326) -- Improved CINN, PaddlePaddle, PIR, combinator operator junction operation, including various and PIR and its actuator interface and CINN mutual support. [#59170](https://github.com/PaddlePaddle/Paddle/pull/59170),[#58766](https://github.com/PaddlePaddle/Paddle/pull/58766),[#59255](https://github.com/PaddlePaddle/Paddle/pull/59255),[#59203](https://github.com/PaddlePaddle/Paddle/pull/59203),[#59024](https://github.com/PaddlePaddle/Paddle/pull/59024),[#57829](https://github.com/PaddlePaddle/Paddle/pull/57829),[#58135](https://github.com/PaddlePaddle/Paddle/pull/58135),[#58193](https://github.com/PaddlePaddle/Paddle/pull/58193),[#58207](https://github.com/PaddlePaddle/Paddle/pull/58207),[#58606](https://github.com/PaddlePaddle/Paddle/pull/58606),[#59437](https://github.com/PaddlePaddle/Paddle/pull/59437),[#59759](https://github.com/PaddlePaddle/Paddle/pull/59759),[#55075](https://github.com/PaddlePaddle/Paddle/pull/55075),[#56805](https://github.com/PaddlePaddle/Paddle/pull/56805),[#57764](https://github.com/PaddlePaddle/Paddle/pull/57764),[#58620](https://github.com/PaddlePaddle/Paddle/pull/58620),[#59769](https://github.com/PaddlePaddle/Paddle/pull/59769),[#58702](https://github.com/PaddlePaddle/Paddle/pull/58702),[#58749](https://github.com/PaddlePaddle/Paddle/pull/58749),[#59025](https://github.com/PaddlePaddle/Paddle/pull/59025),[#58820](https://github.com/PaddlePaddle/Paddle/pull/58820),[#58908](https://github.com/PaddlePaddle/Paddle/pull/58908),[#58169](https://github.com/PaddlePaddle/Paddle/pull/58169) -- There are strongly constrained components to stabilize CINN changes. [#55090](https://github.com/PaddlePaddle/Paddle/pull/55090),[#55705](https://github.com/PaddlePaddle/Paddle/pull/55705),[#57587](https://github.com/PaddlePaddle/Paddle/pull/57587),[#59501](https://github.com/PaddlePaddle/Paddle/pull/59501) -- Improved CINN IR and related tool codes. [#55145](https://github.com/PaddlePaddle/Paddle/pull/55145),[#55955](https://github.com/PaddlePaddle/Paddle/pull/55955),[#56307](https://github.com/PaddlePaddle/Paddle/pull/56307),[#55519](https://github.com/PaddlePaddle/Paddle/pull/55519),[#56958](https://github.com/PaddlePaddle/Paddle/pull/56958),[#57019](https://github.com/PaddlePaddle/Paddle/pull/57019),[#57230](https://github.com/PaddlePaddle/Paddle/pull/57230),[#57531](https://github.com/PaddlePaddle/Paddle/pull/57531),[#57532](https://github.com/PaddlePaddle/Paddle/pull/57532),[#57524](https://github.com/PaddlePaddle/Paddle/pull/57524),[#58770](https://github.com/PaddlePaddle/Paddle/pull/58770),[#59337](https://github.com/PaddlePaddle/Paddle/pull/59337),[#59096](https://github.com/PaddlePaddle/Paddle/pull/59096),[#56274](https://github.com/PaddlePaddle/Paddle/pull/56274),[#56350](https://github.com/PaddlePaddle/Paddle/pull/56350),[#57312](https://github.com/PaddlePaddle/Paddle/pull/57312),[#55171](https://github.com/PaddlePaddle/Paddle/pull/55171) -- Supports CINN Group Schedule operator – at Group level, perform more general and stable Schedule optimization. [#54982](https://github.com/PaddlePaddle/Paddle/pull/54982),[#57963](https://github.com/PaddlePaddle/Paddle/pull/57963),[#58220](https://github.com/PaddlePaddle/Paddle/pull/58220),[#55484](https://github.com/PaddlePaddle/Paddle/pull/55484),[#55935](https://github.com/PaddlePaddle/Paddle/pull/55935),[#55590](https://github.com/PaddlePaddle/Paddle/pull/55590),[#56530](https://github.com/PaddlePaddle/Paddle/pull/56530),[#58344](https://github.com/PaddlePaddle/Paddle/pull/58344),[#59810](https://github.com/PaddlePaddle/Paddle/pull/59810) -- CINN architectural improvements, including parallel compilation, low-level storage allocation method, print information, Group structure, Pass structure, etc. [#56282](https://github.com/PaddlePaddle/Paddle/pull/56282), [#59014](https://github.com/PaddlePaddle/Paddle/pull/59014),[#59209](https://github.com/PaddlePaddle/Paddle/pull/59209),[#52660](https://github.com/PaddlePaddle/Paddle/pull/52660),[#54749](https://github.com/PaddlePaddle/Paddle/pull/54749),[#58694](https://github.com/PaddlePaddle/Paddle/pull/58694),[#58940](https://github.com/PaddlePaddle/Paddle/pull/58940),[#59504](https://github.com/PaddlePaddle/Paddle/pull/59504),[#56123](https://github.com/PaddlePaddle/Paddle/pull/56123) -- Improved CINN codegen, jit instruction, dim args, and host kernel to support dynamic shape. [#58825](https://github.com/PaddlePaddle/Paddle/pull/58825),[#59395](https://github.com/PaddlePaddle/Paddle/pull/59395),[#59398](https://github.com/PaddlePaddle/Paddle/pull/59398),[#59540](https://github.com/PaddlePaddle/Paddle/pull/59540),[#59470](https://github.com/PaddlePaddle/Paddle/pull/59470),[#59640](https://github.com/PaddlePaddle/Paddle/pull/59640) -- CINN error reporting optimization. [#54983](https://github.com/PaddlePaddle/Paddle/pull/54983),[#55544](https://github.com/PaddlePaddle/Paddle/pull/55544) -- Improved cleanup of CINN codes, including CI, file paths, C++17, Flags, third-party libraries, Docker, etc. [#55018](https://github.com/PaddlePaddle/Paddle/pull/55018),[#55121](https://github.com/PaddlePaddle/Paddle/pull/55121),[#55009](https://github.com/PaddlePaddle/Paddle/pull/55009),[#55888](https://github.com/PaddlePaddle/Paddle/pull/55888),[#56168](https://github.com/PaddlePaddle/Paddle/pull/56168),[#56192](https://github.com/PaddlePaddle/Paddle/pull/56192),[#56896](https://github.com/PaddlePaddle/Paddle/pull/56896),[#53861](https://github.com/PaddlePaddle/Paddle/pull/53861),[#55208](https://github.com/PaddlePaddle/Paddle/pull/55208) +- Fix security vulnerability against security leakage risk in prune_by_memory_estimation operator. [#61320](https://github.com/PaddlePaddle/Paddle/pull/61320) -#### Performance optimization +## Parameter Server -- Fusion of vit attention. [#54139](https://github.com/PaddlePaddle/Paddle/pull/54139) -- Optimized block reduce. [#58196](https://github.com/PaddlePaddle/Paddle/pull/58196) +This update mainly fixes several bugs in the process of using the parameter server as well as compilation and installation issues. -#### Fixed bug +### Bug Fixing -- Fixed operator-related bugs. [#56280](https://github.com/PaddlePaddle/Paddle/pull/56280),[#57767](https://github.com/PaddlePaddle/Paddle/pull/57767),[#58406](https://github.com/PaddlePaddle/Paddle/pull/58406),[#54406](https://github.com/PaddlePaddle/Paddle/pull/54406),[#54494](https://github.com/PaddlePaddle/Paddle/pull/54494),[#54751](https://github.com/PaddlePaddle/Paddle/pull/54751),[#55674](https://github.com/PaddlePaddle/Paddle/pull/55674),[#55684](https://github.com/PaddlePaddle/Paddle/pull/55684),[#55683](https://github.com/PaddlePaddle/Paddle/pull/55683),[#57798](https://github.com/PaddlePaddle/Paddle/pull/57798),[#57816](https://github.com/PaddlePaddle/Paddle/pull/57816),[#57687](https://github.com/PaddlePaddle/Paddle/pull/57687),[#56719](https://github.com/PaddlePaddle/Paddle/pull/56719),[#59756](https://github.com/PaddlePaddle/Paddle/pull/59756),[#59770](https://github.com/PaddlePaddle/Paddle/pull/59770),[#58811](https://github.com/PaddlePaddle/Paddle/pull/58811) -- Fixed process architecture-related bugs. [#54899](https://github.com/PaddlePaddle/Paddle/pull/54899),[#59737](https://github.com/PaddlePaddle/Paddle/pull/59737),[#59356](https://github.com/PaddlePaddle/Paddle/pull/59356),[#56105](https://github.com/PaddlePaddle/Paddle/pull/56105),[#56662](https://github.com/PaddlePaddle/Paddle/pull/56662),[#58146](https://github.com/PaddlePaddle/Paddle/pull/58146),[#58910](https://github.com/PaddlePaddle/Paddle/pull/58910),[#58121](https://github.com/PaddlePaddle/Paddle/pull/58121),[#58943](https://github.com/PaddlePaddle/Paddle/pull/58943),[#58886](https://github.com/PaddlePaddle/Paddle/pull/58886),[#59642](https://github.com/PaddlePaddle/Paddle/pull/59642),[#56164](https://github.com/PaddlePaddle/Paddle/pull/56164),[#56338](https://github.com/PaddlePaddle/Paddle/pull/56338),[#56966](https://github.com/PaddlePaddle/Paddle/pull/56966),[#59112](https://github.com/PaddlePaddle/Paddle/pull/59112),[#55820](https://github.com/PaddlePaddle/Paddle/pull/55820),[#56660](https://github.com/PaddlePaddle/Paddle/pull/56660),[#57307](https://github.com/PaddlePaddle/Paddle/pull/57307),[#57530](https://github.com/PaddlePaddle/Paddle/pull/57530),[#58236](https://github.com/PaddlePaddle/Paddle/pull/58236),[#55190](https://github.com/PaddlePaddle/Paddle/pull/55190),[#55043](https://github.com/PaddlePaddle/Paddle/pull/55043),[#55667](https://github.com/PaddlePaddle/Paddle/pull/55667) -- Other bugs. [#57239](https://github.com/PaddlePaddle/Paddle/pull/57239),[#55530](https://github.com/PaddlePaddle/Paddle/pull/55530),[#56605](https://github.com/PaddlePaddle/Paddle/pull/56605),[#58243](https://github.com/PaddlePaddle/Paddle/pull/58243),[#58197](https://github.com/PaddlePaddle/Paddle/pull/58197),[#58197](https://github.com/PaddlePaddle/Paddle/pull/58197),[#56086](https://github.com/PaddlePaddle/Paddle/pull/56086),[#56065](https://github.com/PaddlePaddle/Paddle/pull/56065),[#58775](https://github.com/PaddlePaddle/Paddle/pull/58775),[#54750](https://github.com/PaddlePaddle/Paddle/pull/54750),[#58595](https://github.com/PaddlePaddle/Paddle/pull/58595),[#58873](https://github.com/PaddlePaddle/Paddle/pull/58873) +- For the problem of reading and writing out of bounds of the unique operator, fix the problem of setting the wrong length in the calculation process of the unique operator to ensure the correctness of the operation of the unique operator. [#60840](https://github.com/PaddlePaddle/Paddle/pull/60840) +- Fixed some bugs in PGLBox save/load and compilation process to ensure the correctness of PGLBox function in response to the lack of save/load function and compilation error in PGLBox training process. [#63905](https://github.com/PaddlePaddle/Paddle/pull/63905) +- Fix the setting value of use_ps_gpu in CPUPS to ensure the correctness of the CPUPS training process, in response to the problem that the CPUPS training process triggers the GPUPS logic and causes the training to crash. [#61406](https://github.com/PaddlePaddle/Paddle/pull/61406) +- For the problem that the cudaErrorInvalidResourceHandle error occurs in GPUPS training in CUDA 12.3, add the device id switching mechanism, to ensure that the corresponding resource operation is carried out on the correct device. [#63391](https://github.com/PaddlePaddle/Paddle/pull/63391) +- For the problem of garbled codes in PGLBox Embedding Dump process, fix the bug of improper use of C++ std::string, to ensure the correctness of Embedding Dump results. [#65179](https://github.com/PaddlePaddle/Paddle/pull/65179) -#### Documentation +### Documentation Improvement -- Added README file. [#58349](https://github.com/PaddlePaddle/Paddle/pull/58349) +- Access security warnings in the RPC interface documentation, to remind users that they need to use this interface under secure network conditions. [#64100](https://github.com/PaddlePaddle/Paddle/pull/64100) -## 4. Deployment Direction (Paddle Inference) +### Security Enhancement -### General inference optimization +- Fix several code security issues to prevent malicious code injection. [#60023](https://github.com/PaddlePaddle/Paddle/pull/60023)，[#60544](https://github.com/PaddlePaddle/Paddle/pull/60544)，[#60615](https://github.com/PaddlePaddle/Paddle/pull/60615) + +## Inference Deployment + +The inference framework is based on PIR upgraded PASS under GPU, XPU, CPU hardware, to significantly reduce the number of lines of codes compared with the previous version, and improve development efficiency. The underlying executor is upgraded to a new version of asynchronous executor, improving inference performance on most models. Complete the adaptive interconnection for inference acceleration based on CINN compiler. Add the switches for these features. Users can turn on the features through settings. In addition, Paddle Inference supports direct loading of optimized serialized models under mixed inference with TensorRT subgraphs natively, to reduce startup time consumption. For Paddle-TensorRT, add the interfaces to flexibly control node computation precision and whether the subgraph enters TensorRT computation. It is convenient for debugging. For performance optimization, GPU, XPU, CPU are added with more Transformer and LLM computing acceleration fusion operator, such as group attention mechanism fusion operator, GQA structure, and WINT4, and support for automatic matching by PASS. -This version of the upgrade improves performance and ease-of-use of the inference engine on GPU and CPU, reducing user cost and application cost of online inference. On GPU: A high-performance multi-threaded asynchronous executor is supported, and inference performance of each model is improved by 5%~10%. The new version of TensorRT and BF16 inference capabilities are also supported, and TensorRT inference performance and ease of use are further improved. On CPU: The latest version of OneDNN high-performance inference is supported. SwinTransformer, FastRCNN and other series of models have greatly improved performance. +### New Features -- matmul supports transpose and broadcast operations. [#56827](https://github.com/PaddlePaddle/Paddle/pull/56827) -- TruncatedNormal and Assign supports FP64 data types. [#57507](https://github.com/PaddlePaddle/Paddle/pull/57507) -- Supports conv2d explicit quantized inference. [#57160](https://github.com/PaddlePaddle/Paddle/pull/57160),[#58015](https://github.com/PaddlePaddle/Paddle/pull/58015) -- Added conv_fuse_pass. Support conv + bn fusion. The conv2d_fusion is renamed fused_conv2d_add_act. [#58724](https://github.com/PaddlePaddle/Paddle/pull/58724),[#55374](https://github.com/PaddlePaddle/Paddle/pull/55374),[#54477](https://github.com/PaddlePaddle/Paddle/pull/54477),[#59431](https://github.com/PaddlePaddle/Paddle/pull/59431) -- Mixed precision inference supports OP whitelisting. [#56535](https://github.com/PaddlePaddle/Paddle/pull/56535) -- OneDNN optimization is enabled by default. Supports SwinTransformer, FastRCNNd and other inference optimizations. [#58560](https://github.com/PaddlePaddle/Paddle/pull/58560),[#59394](https://github.com/PaddlePaddle/Paddle/pull/59394),[#59421](https://github.com/PaddlePaddle/Paddle/pull/59421),[#58435](https://github.com/PaddlePaddle/Paddle/pull/58435),[#58488](https://github.com/PaddlePaddle/Paddle/pull/58488),[#59259](https://github.com/PaddlePaddle/Paddle/pull/59259),[#56303](https://github.com/PaddlePaddle/Paddle/pull/56303),[#56782](https://github.com/PaddlePaddle/Paddle/pull/56782),[#57598](https://github.com/PaddlePaddle/Paddle/pull/57598),[#58361](https://github.com/PaddlePaddle/Paddle/pull/58361),[#59641](https://github.com/PaddlePaddle/Paddle/pull/59641),[#59527](https://github.com/PaddlePaddle/Paddle/pull/59527),[#59663](https://github.com/PaddlePaddle/Paddle/pull/59663),[#59744](https://github.com/PaddlePaddle/Paddle/pull/59744) -- Added share_data and support for pass in specified data. [#57933](https://github.com/PaddlePaddle/Paddle/pull/57933) - -### Large model inference optimized - -The fine-grained fusion inference optimization of generative large models is realized. Optimization solution ensures high-performance inference capability and excellent expandability. Users can flexibly utilize various fine-grained fusion operators and PaddlePaddle native operators to build a network structure of generative large models in free combinations as required, thus achieving efficient and low-cost inference. In addition, our solution also supports mainstream generative large model structure, significantly reducing deployment cost of inference for such models and strongly supports efficient and low-cost implementation of generative large models. +- Paddle-TensorRT + - The API called at the underlying of Paddle-TensorRT is upgraded. When the version of TensorRT is later than 8.5, the EnqueueV2 API called (which will be deprecated in the future) is upgraded to the EnqueueV3 API. [#60807](https://github.com/PaddlePaddle/Paddle/pull/60807) + - Add the config.exp_disable_tensorrt_subgraph() to set some subgraphs not to enter TensorRT. [#61967](https://github.com/PaddlePaddle/Paddle/pull/61967) + - Add the config.exp_disable_tensorrt_dynamic_shape_ops() to set dynamic shape input operators not to enter TensorRT. The default value is False. [#62352](https://github.com/PaddlePaddle/Paddle/pull/62352) + - Add the config.exp_specify_tensorrt_subgraph_precision() to set nodes to run different precision types. [#62402](https://github.com/PaddlePaddle/Paddle/pull/62402) +- In the Inference, add switch to turn on CINN compiler. When configuring inference config, turn on CINN through config.enable_cinn(). [#61949](https://github.com/PaddlePaddle/Paddle/pull/61949) +- PIR use mechanism in the Inference upgrade + - In the config, add enable_new_ir() interface to enable PIR. [#61968](https://github.com/PaddlePaddle/Paddle/pull/61968) + - In the config, add set_optimization_level() interface to set different optimization levels. [#61968](https://github.com/PaddlePaddle/Paddle/pull/61968) + - In the PIR mechanism, the PASS function supports custom C++PASS. [#62468](https://github.com/PaddlePaddle/Paddle/pull/62468) + - The inference library exposes PIR-related implementation header files to the outside world. Support users' secondary development based on PIR, such as custom Pass development. [#61863](https://github.com/PaddlePaddle/Paddle/pull/61863),[#62293](https://github.com/PaddlePaddle/Paddle/pull/62293) + - The PIR mechanism supports input and output of the Hook operator by registering the Predictor. [#63101](https://github.com/PaddlePaddle/Paddle/pull/63101) +- The multi-layer Transformer fusion operator fused_multi_transformer_op supports GQA calculation. [#64125](https://github.com/PaddlePaddle/Paddle/pull/64125) + +### Function Improvements + +- The inference supports loading optimized models directly, making it possible to skip IR optimization altogether. The deployment in this way can minimize framework overhead. [#61598](https://github.com/PaddlePaddle/Paddle/pull/61598) +- Re-specify the shape range information file when loading the saved IR PASS optimized model inference. [#60457](https://github.com/PaddlePaddle/Paddle/pull/60457) +- Collect the Shape information within the subgraph of the control flow operator, supporting the use of Paddle-TensorRT inference acceleration. [#60451](https://github.com/PaddlePaddle/Paddle/pull/60451) ,[#59588](https://github.com/PaddlePaddle/Paddle/pull/59588) +- The mixed-precision PASS (auto_mixed_precision_pass) for GPU-native inference supports the handling of sparse Tensor. [#62656](https://github.com/PaddlePaddle/Paddle/pull/62656) +- XPU hardware related function + - XPU's fused PASS for Conv and FC supports conversion from Float to INT31 type. [#59981](https://github.com/PaddlePaddle/Paddle/pull/59981) + - XPU's strided slice operator supports the setting of strides non-negative. [#62268](https://github.com/PaddlePaddle/Paddle/pull/62268) + - XPU's multi-layer Encoder fusion PASS is adaptive to sequence length and supports variable length. [#63825](https://github.com/PaddlePaddle/Paddle/pull/63825) +- Paddle TensorRT INT8 computation mode supports tile operator into TensorRT computation, to improve INT8 performance of some models. [#60189](https://github.com/PaddlePaddle/Paddle/pull/60189) + +### Model Compression + +Fix bugs and optimize functions mainly for Post Training Quantization (PTQ) and Quantization Aware Training (QAT). + +- Support the simulation quantization grouped by channel. [#61828](https://github.com/PaddlePaddle/Paddle/pull/61828) +- Support automatic saving of quantization scale to model parameter file under dynamic graphs. [#59441](https://github.com/PaddlePaddle/Paddle/pull/59441) +- Remove the restriction that the dataloader must be a DataLoader instance. [#61798](https://github.com/PaddlePaddle/Paddle/pull/61798) + +### Performance Optimization + +- Upgrade the inference executor to reduce the video memory usage at runtime while keeping the performance unchanged. This can be used through config.enable_use_executor(True). [#57920](https://github.com/PaddlePaddle/Paddle/pull/57920),[#58452](https://github.com/PaddlePaddle/Paddle/pull/58452),[#63350](https://github.com/PaddlePaddle/Paddle/pull/63350),[#64466](https://github.com/PaddlePaddle/Paddle/pull/64466) +- Upgrade oneDNN version of paddle inference to v3.4. Its overall performance has been improved compared with v3.3. [#64661](https://github.com/PaddlePaddle/Paddle/pull/64661) +- Upgrade the CUTLASS-based support for matrix multiplication and activation fusion calculation. ([#61925](https://github.com/PaddlePaddle/Paddle/pull/61925)) + +#### Add generic PASS in PIR mechanism + +- Add identity_op_clean_pass and matmul_scale_fuse_pass. [#59840](https://github.com/PaddlePaddle/Paddle/pull/59840) +- Add fused_flash_attn_pass. The pass can call flash_attention to replace the original attentions computation. [#64213](https://github.com/PaddlePaddle/Paddle/pull/64213),[#64707](https://github.com/PaddlePaddle/Paddle/pull/64707),[#63304](https://github.com/PaddlePaddle/Paddle/pull/63304) +- In the inference PIR new architecture, upgrade layout adjustment algorithm, support the NHWC inference of conv class and norm class. The performance tested on SD models is significantly improved. [#63628](https://github.com/PaddlePaddle/Paddle/pull/63628),[#64634](https://github.com/PaddlePaddle/Paddle/pull/64634),[#64658](https://github.com/PaddlePaddle/Paddle/pull/64658),[#64708](https://github.com/PaddlePaddle/Paddle/pull/64708),[#64830](https://github.com/PaddlePaddle/Paddle/pull/64830),[#64896](https://github.com/PaddlePaddle/Paddle/pull/64896) +- Add remove_redundant_transpose PASS. [#63357](https://github.com/PaddlePaddle/Paddle/pull/63357) +- Enable CSE PASS in inference to improve inference performance. [#64523](https://github.com/PaddlePaddle/Paddle/pull/64523) + +#### GPU Performance Optimizations + +Include new fusion operators and new PASS under PIR mechanism. + +- Optimize the performance of sparse convolution operator (sparse conv) to improve the inference performance of BEV and other models. [#63067](https://github.com/PaddlePaddle/Paddle/pull/63067) +- Add the fusion PASS based on flash attention. [#63220](https://github.com/PaddlePaddle/Paddle/pull/63220) +- The inference supports elementwise_add+group_norm+silu activated operator fusion pattern and its corresponding fusion kernel. [#64199](https://github.com/PaddlePaddle/Paddle/pull/64199) +- The Matrix multiplication calculation supports groupwise's Weight only INT4 calculation. [#60422](https://github.com/PaddlePaddle/Paddle/pull/60422) 、[#63212](https://github.com/PaddlePaddle/Paddle/pull/63212) 、[#60204](https://github.com/PaddlePaddle/Paddle/pull/60204)) +- The implementation of the group attention mechanism fusion operator block_multi_head_attention supports KV Cache quantization. [#59951](https://github.com/PaddlePaddle/Paddle/pull/59951)) +- The Inference uses CUTLASS upgraded conv fusion operator to implement and support PASS automatic fusion. Support bias and activation. Compared to the original cuDNN, the new operator has significant performance acceleration. It is used through config.exp_enable_use_cutlass(True). [#64201](https://github.com/PaddlePaddle/Paddle/pull/64201)、[#64641](https://github.com/PaddlePaddle/Paddle/pull/64641) +- Add the blha_get_max_len operator and remove every call to get_max_len in block_multihead_attention. The function application is used for large model dynamic inference acceleration. [#64246](https://github.com/PaddlePaddle/Paddle/pull/64246) +- Data layout optimization: PASS prohibits using NHWC mode calculation in the conv fusion operator FP32 precision type, because cuDNN will cause performance degradation under this condition. [#63400](https://github.com/PaddlePaddle/Paddle/pull/63400) +- GPU peak video memory optimization: upgrade the underlying interface TryShrinkMemory, and upgrade to support GPU place under the support for the release of the idle video memory in the pool. In certain scenarios, peak video memory can be significantly cut. [#61319](https://github.com/PaddlePaddle/Paddle/pull/61319) + +#### CPU performance optimization + +Include new fusion operator. Add PASS under PIR mechanism and optimize part of Kernel. + +- Add scale_matmul_fuse_pass. [#63313](https://github.com/PaddlePaddle/Paddle/pull/63313) +- Add CPU implementation in fused_bias_residual_layernorm and fused_rms_norm to improve inference speed. [#63196](https://github.com/PaddlePaddle/Paddle/pull/63196)、[#63165](https://github.com/PaddlePaddle/Paddle/pull/63165) +- Add the cache optimization for Deconvolution kernel, to greatly improve the execution speed of this operator. [#60922](https://github.com/PaddlePaddle/Paddle/pull/60922) +- In PIR, add depthwise_conv fusion PASS, to convert the depthwise_conv operator to conv2d, thus using the onednn conv2d kernel optimization to improve the inference speed of this operator. [#63051](https://github.com/PaddlePaddle/Paddle/pull/63051) +- In PIR, add Conv and Activation Fusion PASS (conv_activation_mkldnn_fuse_pass), to support the fusion of conv and 13 kinds of activation functions, thus greatly improving the inference speed of conv-related operators. [#63145](https://github.com/PaddlePaddle/Paddle/pull/63145) +- In PIR, add the fusion PASS (operator_unsqueeze_onednn_fuse_pass) between multiple operators and unsqueeze, to improve inference speed. [#63592](https://github.com/PaddlePaddle/Paddle/pull/63592) +- In PIR, add PASS (operator_reshape_onednn_fuse_pass) to fuse reshape into multiple operators. [#63812](https://github.com/PaddlePaddle/Paddle/pull/63812) +- In PIR, add scale fusion PASS (operator_scale_onednn_fuse_pass). [#63811](https://github.com/PaddlePaddle/Paddle/pull/63811) +- In PIR, add PASS (conv2d_transpose_bias operator) that fuses conv and bias. [#62241](https://github.com/PaddlePaddle/Paddle/pull/62241) +- In PIR, add onednn_placement_pass, which supports 151 operators to convert from Phi operators to oneDNN operators, so that the oneDNN high-performance library can be used for optimization, to improve the inference speed. [#63982](https://github.com/PaddlePaddle/Paddle/pull/63982) +- In PIR, add the fusion between Elementwise type operators and 13 activation functions, to greatly improve the inference speed of enabling Onednn on the CPU. [#63516](https://github.com/PaddlePaddle/Paddle/pull/63516) +- In PIR, add the fusion of multiple conv + concat + activation functions and fused_conv + concat + activation functions, to greatly improve the inference speed when there are concat and activation functions in conv. [#62993](https://github.com/PaddlePaddle/Paddle/pull/62993)、 [#62713](https://github.com/PaddlePaddle/Paddle/pull/62713) +- In PIR, add matmul+add operator fusion PASS (matmul_elementwise_add_fuse_pass). [#62715](https://github.com/PaddlePaddle/Paddle/pull/62715) +- In PIR, add the scale parameter to fold PASS (scale_matmul_fuse_pass). [#63313](https://github.com/PaddlePaddle/Paddle/pull/63313) +- In PIR, add the fusion PASS (softplus_activation_fuse_pass) between softplus and 12 activation functions. [#63617](https://github.com/PaddlePaddle/Paddle/pull/63617) +- In PIR, add fc operator conversion PASS (fc_onednn_enable_pass). [#63518](https://github.com/PaddlePaddle/Paddle/pull/63518) +- In PIR, add self-attention operator fusion PASS (self_attention_fuse_pass). [#63726](https://github.com/PaddlePaddle/Paddle/pull/63726) +- In PIR, add fusion PASS (fc_activation_fuse_pass) between fc and 12 activation functions. [#63853](https://github.com/PaddlePaddle/Paddle/pull/63853) +- In PIR, add BatchNorm folded PASS (conv2d_bn_onednn_fuse_pass) to amplify the fusion probability of subsequent PASS. [#64524](https://github.com/PaddlePaddle/Paddle/pull/64524) +- In PIR, add the fusion PASS (matmul_activation_fuse_pass) between matmul and 12 activation functions. [#62901](https://github.com/PaddlePaddle/Paddle/pull/62901) +- In PIR, add reshape + transpose + reshape fusion PASS (shuffle_channel_detect_pass), which is fused into a shuffle_channel operator under specific conditions. [#64053](https://github.com/PaddlePaddle/Paddle/pull/64053) +- In PIR, add reshape + transpose + matmul fusion PASS (reshape_transpose_matmul_fuse_pass). [#62998](https://github.com/PaddlePaddle/Paddle/pull/62998) +- In PIR, add matmul + transpose + reshape fusion PASS (matmul_transpose_reshape_fuse_pass) to PIR to significantly improve performance in some scenarios. [#63151](https://github.com/PaddlePaddle/Paddle/pull/63151)(https://github.com/PaddlePaddle/Paddle/pull/63151) +- XPU hardware new fusion PASS optimization: + - Add qk_qkv_attention_xpu_fuse_pass and qkv_attention_xpu_kernel in XPU hardware. [#60089](https://github.com/PaddlePaddle/Paddle/pull/60089) + - Add rotary position encoded fusion operator, to support elementwise_mul + strided_slice + sin/cos+ stack fusion to 1 operator in XPU hardware. [#60025](https://github.com/PaddlePaddle/Paddle/pull/60025) + - Add group_norm_silu_xpu_fuse_pass. [#62689](https://github.com/PaddlePaddle/Paddle/pull/62689) + - Add weight_only_linear_xpu_pass. [#64185](https://github.com/PaddlePaddle/Paddle/pull/64185) + - Add block_multihead_attention operator and PASS, to support large model inference for LLaMA2 models in XPU devices. [#65036](https://github.com/PaddlePaddle/Paddle/pull/65036) + - Support float16 type for squeeze_excitation_block_xpu_kernel. [#61023](https://github.com/PaddlePaddle/Paddle/pull/61023) + +### Bug Fixing + +- Fix mixed-precision conversions in models such as faster_rcnn_swin_tiny_fpn_1x_coco, and solve the mixed_precision_pass error. [#64673](https://github.com/PaddlePaddle/Paddle/pull/64673) +- Block fused_conv2d_add_act pass from being validated in activation functions that are sigmoid (fused conv2d and sigmoid cause performance degradation between cudnn versions 8.0 and 8.7). [#64717](https://github.com/PaddlePaddle/Paddle/pull/64717) +- Fix compilation issues with self_dp_attention and fused_layer_norm_avx_kernel in Clang12. [#63414](https://github.com/PaddlePaddle/Paddle/pull/63414) +- Fix the issue that scale and zeroPoints in the qdq operator of some models are deleted prematurely in the IR/Pass stage. [#62225](https://github.com/PaddlePaddle/Paddle/pull/62225) +- Fix the issue that causes an error to be reported when both Config.UseOptimizedModel() and config.EnableMemoryOptim() are turned on. [#62501](https://github.com/PaddlePaddle/Paddle/pull/62501) +- Add constraint on matmul_scale_fuse_pass, where input w must be a weight or the pass will not be matched. [#62850](https://github.com/PaddlePaddle/Paddle/pull/62850) +- Keep inference model output key ordering guaranteed to be the same as when dynamic graph models are exported. [#63791](https://github.com/PaddlePaddle/Paddle/pull/63791) +- Fix the error in subgraph when the constant fold PASS is in "the folded op and its input and output are not in the same subgraph." [#62148](https://github.com/PaddlePaddle/Paddle/pull/62148) +- Fix several runtime problems in PaddleTRT mode. Include the failure of quantization calibration table generation caused by yolo_box operator in int8 mode, and the error caused by incorrect handling of dim attribute data type in reduce operator. [#61596](https://github.com/PaddlePaddle/Paddle/pull/61596) +- Fix some runtime error problems in mixed-precision inference mode.Include the errors caused by sharing weights among fused conv2d operators without correctly converting weight layout, fused conv2d operator backend not properly selected as cuDNN, fused conv2d operator incorrectly handling bias dimension under NHWC, incorrectly handling input data type of norm class operator. [#60955](https://github.com/PaddlePaddle/Paddle/pull/60955)、[#60076](https://github.com/PaddlePaddle/Paddle/pull/60076)、[#63007](https://github.com/PaddlePaddle/Paddle/pull/63007)、[#63988](https://github.com/PaddlePaddle/Paddle/pull/63988) +- Fix the problem that config.delete_pass function does not take effect. [#61056](https://github.com/PaddlePaddle/Paddle/pull/61056) +- Fix the GC mechanism of While control flow in PIR to recycle unwanted inputs in advance and reduce the peak memory, for example, 2GB memory reduction in LLaMA 7B model. [#63062](https://github.com/PaddlePaddle/Paddle/pull/63062) +- Fix the OneDNN mean kernel rollback error. [#64676](https://github.com/PaddlePaddle/Paddle/pull/64676) +- Fix the conv_bias_fuse_pass strong constraints newly added, e.g., the shape of the bias cannot be 1, so as to ensure the stability of the pass inference result. [#64412](https://github.com/PaddlePaddle/Paddle/pull/64412) +- Fix the conv_elementwise_add_onednn_fuse_pass strong constraints newly added, e.g., conv2d_out and residual_param must have the same size, so that the pass inference is stable. [#64448](https://github.com/PaddlePaddle/Paddle/pull/64448) +- Fix the problem of repeatedly inserting quantized inverse-quantization operators under certain circumstances [#63082](https://github.com/PaddlePaddle/Paddle/pull/63082) + +## Hardware Adaptation + +### Adaptation Scheme (Custom Device) + +For PaddlePaddle hardware access, add the daily release supports for 4 hardware Kunlun XPU, Ascend NPU, Hygon DCU and Cambricon MLU this time. Meanwhile, the problems in distributed communications have been fixed through large model training and inference deployment, and performance is optimized through functions such as video memory optimization, and overlap of computation and communication. Furthermore, each hardware is also added to support a large number of BFloat16 data type operators this time, as well as many operator fusion Pass and fusion operators on each hardware. Through the hardware and software together, hardware large Transformer operator library is accessed to fully improve the performance of large models. + +#### New Features + +- Add the support for distributed policy sharding stage1 v2. [#61500](https://github.com/PaddlePaddle/Paddle/pull/61500) +- Support the distributed communication module in BF16 data type.Add some operators to support for BF16 data types such as empty, shape, etc. [#60768](https://github.com/PaddlePaddle/Paddle/pull/60768),[#62140](https://github.com/PaddlePaddle/Paddle/pull/62140),[#62604](https://github.com/PaddlePaddle/Paddle/pull/62604) +- Add the support for get_comm_name interface, support for memory stat function, and support for Profiler to record memory time. [#62556](https://github.com/PaddlePaddle/Paddle/pull/62556),[#61030](https://github.com/PaddlePaddle/Paddle/pull/61030),[#62292](https://github.com/PaddlePaddle/Paddle/pull/62292) +- Add support for some fusion strategies and operators, including silu_fuse_pass, conv_elementwise_add_act_fuse_pass, and generator offset. [#60595](https://github.com/PaddlePaddle/Paddle/pull/60595),[#60708](https://github.com/PaddlePaddle/Paddle/pull/60708),[#60616](https://github.com/PaddlePaddle/Paddle/pull/60616) + +#### Performance Optimization + +- The distributed communication strategy Sharing uses asynchronous strategy in Broadcast parameter, to improve the overlap between computation and communication. [#59745](https://github.com/PaddlePaddle/Paddle/pull/59745) +- Add the support for STRIDED Layout operator to improve the performance of the operator. [#62532](https://github.com/PaddlePaddle/Paddle/pull/62532),[#62697](https://github.com/PaddlePaddle/Paddle/pull/62697),[#62649](https://github.com/PaddlePaddle/Paddle/pull/62649) +- Optimize the memory usage of elementwise_mul operator.[#62377](https://github.com/PaddlePaddle/Paddle/pull/62377) + +#### Bug Fixing + +- Fix the bug under the distributed strategy Sharing. [#61942](https://github.com/PaddlePaddle/Paddle/pull/61942),[#62236](https://github.com/PaddlePaddle/Paddle/pull/62236),[#62305](https://github.com/PaddlePaddle/Paddle/pull/62305),[#62535](https://github.com/PaddlePaddle/Paddle/pull/62535),[#62572](https://github.com/PaddlePaddle/Paddle/pull/62572),[#61601](https://github.com/PaddlePaddle/Paddle/pull/61601) +- Fix the problem that the operator cannot be registered due to c_embedding operator is not under PHI namespace. [#60774](https://github.com/PaddlePaddle/Paddle/pull/60774) +- Fix the xccl_comm release issue. [#60465](https://github.com/PaddlePaddle/Paddle/pull/60465) +- Fix data address error caused by index_put operator fallbacking cpu. [#61842](https://github.com/PaddlePaddle/Paddle/pull/61842) +- Fix stream_safe_custom_device_allocator issue. [#63369](https://github.com/PaddlePaddle/Paddle/pull/63369) +- Fix the distributed worker port conflict issue. [#61409](https://github.com/PaddlePaddle/Paddle/pull/61409) +- Fix comm data type to improve device compatibility. [#62306](https://github.com/PaddlePaddle/Paddle/pull/62306) +- Unify the use of comm data type to phi::DataType. [#62464](https://github.com/PaddlePaddle/Paddle/pull/62464),[#62562](https://github.com/PaddlePaddle/Paddle/pull/62562) +- Fix the problem of missing precision parameter in PD_ConfigEnableCustomDevice. [#63702](https://github.com/PaddlePaddle/Paddle/pull/63702) + +### Kunlun XPU + +#### New Features + +- Add the support for BF16 data types for some operators, including compare_kernel and add reduce_all_kernel ([#63602](https://github.com/PaddlePaddle/Paddle/pull/63602)), empty([#60212](https://github.com/PaddlePaddle/Paddle/pull/60212)), hybrid_parallel_optimizer([#60213](https://github.com/PaddlePaddle/Paddle/pull/60213)), reduce_max/reduce_min([#60453](https://github.com/PaddlePaddle/Paddle/pull/60453)), all_reduce/concat/split([#62364](https://github.com/PaddlePaddle/Paddle/pull/62364)), tile/tile_grad([#63075](https://github.com/PaddlePaddle/Paddle/pull/63075)), accuracy([#63863](https://github.com/PaddlePaddle/Paddle/pull/63863)), swiglu/set_value([#64070](https://github.com/PaddlePaddle/Paddle/pull/64070)), amp_master_grad([#63865](https://github.com/PaddlePaddle/Paddle/pull/63865)), c_concat ([#63403](https://github.com/PaddlePaddle/Paddle/pull/63403)), flatten ([#63997](https://github.com/PaddlePaddle/Paddle/pull/63997)), compare_op ([#64473](https://github.com/PaddlePaddle/Paddle/pull/64473)), moment1/moment2 ([#62688](https://github.com/PaddlePaddle/Paddle/pull/62688)), fused_rope ([#60064](https://github.com/PaddlePaddle/Paddle/pull/60064)), c_softmax_with_cross_entropy ([#60472](https://github.com/PaddlePaddle/Paddle/pull/60472)), elementwise_pow/square/sin/cos ([#60402](https://github.com/PaddlePaddle/Paddle/pull/60402)), strided_slice ([#60382](https://github.com/PaddlePaddle/Paddle/pull/60382)), tile/sigmoid_grad ([#60119](https://github.com/PaddlePaddle/Paddle/pull/60119)), elementwise_sub/elementwise_div ([#60386](https://github.com/PaddlePaddle/Paddle/pull/60386)), softmax_with_cross_entropy ([#63759](https://github.com/PaddlePaddle/Paddle/pull/63759)) +- Add the support for INT8 data types for some operators, including multi_encoder_xpu ([#61212](https://github.com/PaddlePaddle/Paddle/pull/61212)), qkv_attention ([#63105](https://github.com/PaddlePaddle/Paddle/pull/63105)) +- Update Kunlun SDK versions including BKCL, XHPC, XCCL, etc. [#59895](https://github.com/PaddlePaddle/Paddle/pull/59895)、[#59888](https://github.com/PaddlePaddle/Paddle/pull/59888)、[#63624](https://github.com/PaddlePaddle/Paddle/pull/63624), [#60305](https://github.com/PaddlePaddle/Paddle/pull/60305), [#62076](https://github.com/PaddlePaddle/Paddle/pull/62076), [#62646](https://github.com/PaddlePaddle/Paddle/pull/62646), [#63520](https://github.com/PaddlePaddle/Paddle/pull/63520), [#64163](https://github.com/PaddlePaddle/Paddle/pull/64163), [#64326](https://github.com/PaddlePaddle/Paddle/pull/64326), [#60617](https://github.com/PaddlePaddle/Paddle/pull/60617), [#60377](https://github.com/PaddlePaddle/Paddle/pull/60377), [#60421](https://github.com/PaddlePaddle/Paddle/pull/60421), [#60598](https://github.com/PaddlePaddle/Paddle/pull/60598), [#61199](https://github.com/PaddlePaddle/Paddle/pull/61199) +- Add the support for memory stat function. [#61116](https://github.com/PaddlePaddle/Paddle/pull/61116) +- Add multi-stream support, to assign default l3/gm buffer size to each stream. [#62729](https://github.com/PaddlePaddle/Paddle/pull/62729) +- Add nonzero operator, to support simulator XPUSIM_SKIP_RUN mode. [#60224](https://github.com/PaddlePaddle/Paddle/pull/60224)。[#60388](https://github.com/PaddlePaddle/Paddle/pull/60388) +- Add stride_slice and stride_slice_grad operators, to support strides < 0. [#62749](https://github.com/PaddlePaddle/Paddle/pull/62749) +- Add rotary_embedding, to support use_neox_rotary_style == True. [#64090](https://github.com/PaddlePaddle/Paddle/pull/64090) +- Add fusion Pass and fusion operators including cross_attention ([#63203](https://github.com/PaddlePaddle/Paddle/pull/63203)), fused_bias_act ([#62232](https://github.com/PaddlePaddle/Paddle/pull/62232)), fused_layernorm ([#62228](https://github.com/PaddlePaddle/Paddle/pull/62228)), group_norm_silu_xpu_fuse_pass ([#63342](https://github.com/PaddlePaddle/Paddle/pull/63342)) +- Add the support for distributed policy sharding stage3. [#57457](https://github.com/PaddlePaddle/Paddle/pull/57457) +- Add the support for tf32 fc quantization mode. [#62273](https://github.com/PaddlePaddle/Paddle/pull/62273) +- Add the flash attention operator. [#60065](https://github.com/PaddlePaddle/Paddle/pull/60065) +- Add the roformer relative embedding pass & kernel and support multi_encoder_xpu. [#62089](https://github.com/PaddlePaddle/Paddle/pull/62089) +- Add the support for pp + sharding strategy. [#63640](https://github.com/PaddlePaddle/Paddle/pull/63640) +- Upgrade the XPU communication library architecture to support dynamic-static unified communication library function. [#63817](https://github.com/PaddlePaddle/Paddle/pull/63817) + +#### Performance Optimization + +- Add XHPC buffer manager to improve the performance of Paddle and XHPC memory collaboration. [#63924](https://github.com/PaddlePaddle/Paddle/pull/63924) +- Enhance TensorSetConstantXPU performance and support BF16 data type. [#63920](https://github.com/PaddlePaddle/Paddle/pull/63920),[#61818](https://github.com/PaddlePaddle/Paddle/pull/61818) +- Fusion multiple group norm + silu + conv modules and compress the video memory. [#62892](https://github.com/PaddlePaddle/Paddle/pull/62892) +- Optimize XPU memory allocation in comm manager. [#64139](https://github.com/PaddlePaddle/Paddle/pull/64139) +- Optimize operator performance, including mean_all_grad ([#61148](https://github.com/PaddlePaddle/Paddle/pull/61148)), dropout_v2 ([#61029](https://github.com/PaddlePaddle/Paddle/pull/61029)), fused_rotary_position_embedding ([#62846](https://github.com/PaddlePaddle/Paddle/pull/62846)), cross_entropy ([#63159](https://github.com/PaddlePaddle/Paddle/pull/63159)), elementwise_add ([#64289](https://github.com/PaddlePaddle/Paddle/pull/64289)), fused_gemm_epilogue ([#61350](https://github.com/PaddlePaddle/Paddle/pull/61350), check_nan_or_inf ([#60853](https://github.com/PaddlePaddle/Paddle/pull/60853)) + +#### Bug Fixing + +- Fix the tile operator support for 0-dimensional Tensor. [#64279](https://github.com/PaddlePaddle/Paddle/pull/64279) +- Fix the group_norm_silu_fuse_pass. [#63449](https://github.com/PaddlePaddle/Paddle/pull/63449) +- Fix the XPU API GM memory issue. [#60260](https://github.com/PaddlePaddle/Paddle/pull/60260),[#60387](https://github.com/PaddlePaddle/Paddle/pull/60387),[#62940](https://github.com/PaddlePaddle/Paddle/pull/62940) +- Fix the distributed strategy Sharing stage1 v2 bug. [#64209](https://github.com/PaddlePaddle/Paddle/pull/64209) +- Fix the XPU constant issue. [#60763](https://github.com/PaddlePaddle/Paddle/pull/60763) +- Fix some operator issues, including AdamW ([#62251](https://github.com/PaddlePaddle/Paddle/pull/62251)), dropout_v3 ([#62726](https://github.com/PaddlePaddle/Paddle/pull/62726)), softmax([#63780](https://github.com/PaddlePaddle/Paddle/pull/63780)) , fused rope embedding ([#62143](https://github.com/PaddlePaddle/Paddle/pull/62143)), elementwise_add ([#60252](https://github.com/PaddlePaddle/Paddle/pull/60252)), resnet_basic_block ([#62914](https://github.com/PaddlePaddle/Paddle/pull/62914)) +- Fix XPU runtime and installation related issues. [#60028](https://github.com/PaddlePaddle/Paddle/pull/60028),[#61970](https://github.com/PaddlePaddle/Paddle/pull/61970) +- Fix XPU compilation bugs. [#63307](https://github.com/PaddlePaddle/Paddle/pull/63307) +- Fix end-side memory related bugs when initializing XPU communication library. [#64396](https://github.com/PaddlePaddle/Paddle/pull/64396) -- Supports the FMHA/MMHA for CacheKV division block scheduling. [#59462](https://github.com/PaddlePaddle/Paddle/pull/59462) -- RoPE encoding fusion operator supports input sin/cos values. [#55415](https://github.com/PaddlePaddle/Paddle/pull/55415) -- Added fine-grained fusion operators. Supports high-performance inference optimization of generative large models. Added operators such as quant_linear, weight_quantize, and linear_compress for support of large model quantitative inference. [#57852](https://github.com/PaddlePaddle/Paddle/pull/57852),[#55128](https://github.com/PaddlePaddle/Paddle/pull/55128),[#59090](https://github.com/PaddlePaddle/Paddle/pull/59090),[#56706](https://github.com/PaddlePaddle/Paddle/pull/56706),[#59951](https://github.com/PaddlePaddle/Paddle/pull/59951),[#55490](https://github.com/PaddlePaddle/Paddle/pull/55490),[#59291](https://github.com/PaddlePaddle/Paddle/pull/59291),[#59441](https://github.com/PaddlePaddle/Paddle/pull/59441),[#59778](https://github.com/PaddlePaddle/Paddle/pull/59778),[#59651](https://github.com/PaddlePaddle/Paddle/pull/59651)[#55301](https://github.com/PaddlePaddle/Paddle/pull/55301),[#58637](https://github.com/PaddlePaddle/Paddle/pull/58637),[#56673](https://github.com/PaddlePaddle/Paddle/pull/56673),[#56401](https://github.com/PaddlePaddle/Paddle/pull/56401) -- Supports variable length inference series API. [#57948](https://github.com/PaddlePaddle/Paddle/pull/57948) -- Supports the GQA inference. [#58472](https://github.com/PaddlePaddle/Paddle/pull/58472),[#58836](https://github.com/PaddlePaddle/Paddle/pull/58836) -- Added masked multihead attention. Supports high performance MMHA inference. [#55344](https://github.com/PaddlePaddle/Paddle/pull/55344),[#56411](https://github.com/PaddlePaddle/Paddle/pull/56411),[#58134](https://github.com/PaddlePaddle/Paddle/pull/58134),[#57936](https://github.com/PaddlePaddle/Paddle/pull/57936) -- weight_quantize/weight_only_linear supports the Volta architecture. [#58082](https://github.com/PaddlePaddle/Paddle/pull/58082) -- Added weight_only_linear_grad for support of large model weight only quantization gradient transfer-back. [#57685](https://github.com/PaddlePaddle/Paddle/pull/57685) -- Fixed large model dynamic to static bug. Optimized communication initialization logic between static graph cards. [#56390](https://github.com/PaddlePaddle/Paddle/pull/56390),[#57169](https://github.com/PaddlePaddle/Paddle/pull/57169),[#56688](https://github.com/PaddlePaddle/Paddle/pull/56688),[#56592](https://github.com/PaddlePaddle/Paddle/pull/56592),[#58868](https://github.com/PaddlePaddle/Paddle/pull/58868) -- Optimized top_p_sampling random number generation logic. [#59494](https://github.com/PaddlePaddle/Paddle/pull/59494) - -### Paddle-TensorRT Inference Optimization - -- elementwise_add fusion supports NHWC format. [#56795](https://github.com/PaddlePaddle/Paddle/pull/56795) -- conv2d supports filter as input. [#55246](https://github.com/PaddlePaddle/Paddle/pull/55246)。 -- Supports BF16 and FP64 inference. [#59765](https://github.com/PaddlePaddle/Paddle/pull/59765),[#55520](https://github.com/PaddlePaddle/Paddle/pull/55520) -- Added MarkTrtEngineOutputs API. Users can specify TensorRT Engine outputs. [#56858](https://github.com/PaddlePaddle/Paddle/pull/56858),[#56188](https://github.com/PaddlePaddle/Paddle/pull/56188),[#57407](https://github.com/PaddlePaddle/Paddle/pull/57407) -- Customized OP can generate TensorRT Plugin automatically. [#58976](https://github.com/PaddlePaddle/Paddle/pull/58976),[#56037](https://github.com/PaddlePaddle/Paddle/pull/56037) -- TensorRT inference allows users to specify input hook to optimize shape collection process. [#59466](https://github.com/PaddlePaddle/Paddle/pull/59466),[#54841](https://github.com/PaddlePaddle/Paddle/pull/54841),[#57498](https://github.com/PaddlePaddle/Paddle/pull/57498),[#54861](https://github.com/PaddlePaddle/Paddle/pull/54861),[#54432](https://github.com/PaddlePaddle/Paddle/pull/54432),[#55503](https://github.com/PaddlePaddle/Paddle/pull/55503) -- TensorRT Inference supports inference model after saving Tuning. [#55893](https://github.com/PaddlePaddle/Paddle/pull/55893),[#56952](https://github.com/PaddlePaddle/Paddle/pull/56952),[#57031](https://github.com/PaddlePaddle/Paddle/pull/57031) -- Supports variable length Transformer model PromptTuning. [#57034](https://github.com/PaddlePaddle/Paddle/pull/57034) -- Added operators such as bitwise_and, bitwise_or, bitwise_not, cumsum, einsum, lookup_table, assign, flip, size, scatter, solve, unbind, reduce, and argsort. Optimized support of existing operators. [#59214](https://github.com/PaddlePaddle/Paddle/pull/59214),[#59293](https://github.com/PaddlePaddle/Paddle/pull/59293),[#54882](https://github.com/PaddlePaddle/Paddle/pull/54882),[#54097](https://github.com/PaddlePaddle/Paddle/pull/54097),[#54860](https://github.com/PaddlePaddle/Paddle/pull/54860),[#55426](https://github.com/PaddlePaddle/Paddle/pull/55426),[#54372](https://github.com/PaddlePaddle/Paddle/pull/54372),[#55688](https://github.com/PaddlePaddle/Paddle/pull/55688),[#56069](https://github.com/PaddlePaddle/Paddle/pull/56069),[#59563](https://github.com/PaddlePaddle/Paddle/pull/59563),[#59317](https://github.com/PaddlePaddle/Paddle/pull/59317),[#59424](https://github.com/PaddlePaddle/Paddle/pull/59424),[#55476](https://github.com/PaddlePaddle/Paddle/pull/55476),[#56043](https://github.com/PaddlePaddle/Paddle/pull/56043),[#58549](https://github.com/PaddlePaddle/Paddle/pull/58549),[#57326](https://github.com/PaddlePaddle/Paddle/pull/57326),[#59409](https://github.com/PaddlePaddle/Paddle/pull/59409)) -- TensorRT enables video memory sharing by default. [#59495](https://github.com/PaddlePaddle/Paddle/pull/59495),[#58251](https://github.com/PaddlePaddle/Paddle/pull/58251) -- PrelnResidualBiasPluginDynamic supports 4D input. [#56304](https://github.com/PaddlePaddle/Paddle/pull/56304) -- Added support for FlashAttention for Paddle-TRT inference for architectures below SM80.[#56492](https://github.com/PaddlePaddle/Paddle/pull/56492) - -### Modification deprecation - -- Removed fc_elementwise_add fusion from OneDNN. [#55504](https://github.com/PaddlePaddle/Paddle/pull/55504) -- Removed redunant op. [#54442](https://github.com/PaddlePaddle/Paddle/pull/54442) - -### Bug Fix - -- Fixed “Inference so” link flags conflict issue. [#59755](https://github.com/PaddlePaddle/Paddle/pull/59755) -- Fixed constant_folding pass execution error. [#55556](https://github.com/PaddlePaddle/Paddle/pull/55556) -- Fixed softmax forward speed bug and reverse accuracy bug. [#56036](https://github.com/PaddlePaddle/Paddle/pull/56036),[#57858](https://github.com/PaddlePaddle/Paddle/pull/57858)[#57538](https://github.com/PaddlePaddle/Paddle/pull/57538) -- Fixed customized OP while error and export bug. [#58898](https://github.com/PaddlePaddle/Paddle/pull/58898),[#59318](https://github.com/PaddlePaddle/Paddle/pull/59318) -- Fixed CUDA 12.0 compilation problem on Windows platform. [#59852](https://github.com/PaddlePaddle/Paddle/pull/59852) -- Fixed bug of inference partial operator error when TensorRT version is later than 8.6. [#54379](https://github.com/PaddlePaddle/Paddle/pull/54379),[#54679](https://github.com/PaddlePaddle/Paddle/pull/54679),[#54251](https://github.com/PaddlePaddle/Paddle/pull/54251) -- Fixed and removed inference fusion Pass. [#54846](https://github.com/PaddlePaddle/Paddle/pull/54846),[#54887](https://github.com/PaddlePaddle/Paddle/pull/54887),[#55573](https://github.com/PaddlePaddle/Paddle/pull/55573),[#56434](https://github.com/PaddlePaddle/Paddle/pull/56434),[#56326](https://github.com/PaddlePaddle/Paddle/pull/56326),[#56753](https://github.com/PaddlePaddle/Paddle/pull/56753),[#57491](https://github.com/PaddlePaddle/Paddle/pull/57491),[#56909](https://github.com/PaddlePaddle/Paddle/pull/56909),[#54536](https://github.com/PaddlePaddle/Paddle/pull/54536),[#55073](https://github.com/PaddlePaddle/Paddle/pull/55073),[#55081](https://github.com/PaddlePaddle/Paddle/pull/55081),[#55240](https://github.com/PaddlePaddle/Paddle/pull/55240),[#56439](https://github.com/PaddlePaddle/Paddle/pull/56439),[#59009](https://github.com/PaddlePaddle/Paddle/pull/59009) -- Fixed error of multi-stream inference context switching. [#57629](https://github.com/PaddlePaddle/Paddle/pull/57629),[#58048](https://github.com/PaddlePaddle/Paddle/pull/58048),[#54994](https://github.com/PaddlePaddle/Paddle/pull/54994) - -## 5. Hardware Support - -### Hardware Integration Solution (Custom Device) - -In this update, added support for distributed advanced strategy, custom operator and custom fusion strategy. By upgrading distributed communication library, supports MP, GroupShared, PP, SP, MOE and other advanced distributed strategies. Meanwhile, enables vendors to flexibly access Transformer operator libraries of different granularities, and modify computation graph through Fusion Pass for performance acceleration. - -#### New features - -- Upgraded CustomDevice to support for Paddle's latest distributed communication library CommContext. Added a variety of advanced distributed strategies such as GroupShared and MOE. [#56301](https://github.com/PaddlePaddle/Paddle/pull/56301),[#54671](https://github.com/PaddlePaddle/Paddle/pull/54671),[#57957](https://github.com/PaddlePaddle/Paddle/pull/57957),[#56669](https://github.com/PaddlePaddle/Paddle/pull/56669),[#54384](https://github.com/PaddlePaddle/Paddle/pull/54384),[#54572](https://github.com/PaddlePaddle/Paddle/pull/54572),[#54573](https://github.com/PaddlePaddle/Paddle/pull/54573),[#54676](https://github.com/PaddlePaddle/Paddle/pull/54676) -- Upgraded CustomDevice to support CustomOP. Users can register undefined operators in Paddle PHI operator library. CustomDevice can support CustomOP via CAPI. [#57038](https://github.com/PaddlePaddle/Paddle/pull/57038),[#55532](https://github.com/PaddlePaddle/Paddle/pull/55532),[#56755](https://github.com/PaddlePaddle/Paddle/pull/56755),[#55532](https://github.com/PaddlePaddle/Paddle/pull/55532),[#55533](https://github.com/PaddlePaddle/Paddle/pull/55533),[#55659](https://github.com/PaddlePaddle/Paddle/pull/55659) -- Added CustomDevice's support for CustomPass function. Modified the computation graph IR through Python API. [#55511](https://github.com/PaddlePaddle/Paddle/pull/55511),[#55728](https://github.com/PaddlePaddle/Paddle/pull/55728) -- Added CustomDevice’s support for Paddle run_check. [#56318](https://github.com/PaddlePaddle/Paddle/pull/56318) -- Added CustomDevice’s support for StreamSafeAllocator. [#55393](https://github.com/PaddlePaddle/Paddle/pull/55393),[#56380](https://github.com/PaddlePaddle/Paddle/pull/56380),[#56536](https://github.com/PaddlePaddle/Paddle/pull/56536),[#58035](https://github.com/PaddlePaddle/Paddle/pull/58035) -- Added CustomDevice’s support for DataTransform. [#56627](https://github.com/PaddlePaddle/Paddle/pull/56627) - -#### Function optimization - -- Added CustomDevice’s support for more PaddlePaddle APIs such as Variable.set_value, adamw, share_external_data, mp_allreduce_sum, tensor.numpy, get_paddle_place, and GeneratorState. [#55272](https://github.com/PaddlePaddle/Paddle/pull/55272), [#56386](https://github.com/PaddlePaddle/Paddle/pull/56386), [#57253](https://github.com/PaddlePaddle/Paddle/pull/57253), [#56927](https://github.com/PaddlePaddle/Paddle/pull/56927),[#56189](https://github.com/PaddlePaddle/Paddle/pull/56189),[#55225](https://github.com/PaddlePaddle/Paddle/pull/55225),[#55247](https://github.com/PaddlePaddle/Paddle/pull/55247) -- Modified CustomDevice dynamic library loading method from RTLD_NOW to RTLD_LAZY, to facilitate subsequent checking of compatibility of CustomDevice related software stack version. [#57544](https://github.com/PaddlePaddle/Paddle/pull/57544) -- Added CustomDevice's detection function for FP16 operator under mixed precision training. [#56053](https://github.com/PaddlePaddle/Paddle/pull/56053),[#56176](https://github.com/PaddlePaddle/Paddle/pull/56176) - -#### Bug Fix - -- Fixed some problems in CustomDevice's support for distributed communication libraries. [#55293](https://github.com/PaddlePaddle/Paddle/pull/55293),[#58038](https://github.com/PaddlePaddle/Paddle/pull/58038),[#59800](https://github.com/PaddlePaddle/Paddle/pull/59800) -- Fixed some problems in CustomDevice on some operators, including c_softmax_with_cross_entropy,data loader,SplitDenseTensor,grad accumulation,atan2 grad.[#56486](https://github.com/PaddlePaddle/Paddle/pull/56486),[#55541](https://github.com/PaddlePaddle/Paddle/pull/55541),[#55615](https://github.com/PaddlePaddle/Paddle/pull/55615),[#56052](https://github.com/PaddlePaddle/Paddle/pull/56052),[#56067](https://github.com/PaddlePaddle/Paddle/pull/56067) -- Fixed some problems of device management in CustomDevice, including device exceptions ([#56556](https://github.com/PaddlePaddle/Paddle/pull/56556),[#58639](https://github.com/PaddlePaddle/Paddle/pull/58639),[#55173](https://github.com/PaddlePaddle/Paddle/pull/55173)), exception events ([#56745](https://github.com/PaddlePaddle/Paddle/pull/56745),[#58059](https://github.com/PaddlePaddle/Paddle/pull/58059)), video memory exception ([#56977](https://github.com/PaddlePaddle/Paddle/pull/56977),[#59247](https://github.com/PaddlePaddle/Paddle/pull/59247),[#54606](https://github.com/PaddlePaddle/Paddle/pull/54606)), device initialization ([#57099](https://github.com/PaddlePaddle/Paddle/pull/57099),[#57994](https://github.com/PaddlePaddle/Paddle/pull/57994)), device release ([#54932](https://github.com/PaddlePaddle/Paddle/pull/54932),[#55351](https://github.com/PaddlePaddle/Paddle/pull/55351),[#55783](https://github.com/PaddlePaddle/Paddle/pull/55783)), and device resource pooling, etc.([#55229](https://github.com/PaddlePaddle/Paddle/pull/55229),[#56580](https://github.com/PaddlePaddle/Paddle/pull/56580)) -- Fixed CustomDevice compilation-related issues. [#56760](https://github.com/PaddlePaddle/Paddle/pull/56760),[#56766](https://github.com/PaddlePaddle/Paddle/pull/56766) - -### Kunlunxin XPU - -#### New features - -- Added XPTI (XPU Profiling Tool Interface) to support collection and analysis function of runtime performance data. [#54685](https://github.com/PaddlePaddle/Paddle/pull/54685),[#54690](https://github.com/PaddlePaddle/Paddle/pull/54690),[#54800](https://github.com/PaddlePaddle/Paddle/pull/54800) -- Supports Paddle's latest distributed communication library CommContext. [#59418](https://github.com/PaddlePaddle/Paddle/pull/59418) -- Added XPU fusion operators, for example, fast_where. [#55628](https://github.com/PaddlePaddle/Paddle/pull/55628) -- Added support for XPU Pluign function, facilitating users to develop XPU customized operators through XTDK programming. [#55101](https://github.com/PaddlePaddle/Paddle/pull/55101),[#59326](https://github.com/PaddlePaddle/Paddle/pull/59326) -- Added XPU’s support for AutoGrowthAllocator. [#54121](https://github.com/PaddlePaddle/Paddle/pull/54121) -- Added operator support list of Kunlun3. [#57683](https://github.com/PaddlePaddle/Paddle/pull/57683) - -#### Function optimization - -- Upgraded XPU Inference API. [#54342](https://github.com/PaddlePaddle/Paddle/pull/54342) -- Optimized performance of some XPU operators. Added support for bf16 in some XPU operators, including unique/index_put,squeeze/unsqueeze kernels,swish/swish_grad,scatter_nd_add_grad/slice,rsqrt/bitwise_or/arange_tensor,where,collective. [#56582](https://github.com/PaddlePaddle/Paddle/pull/56582),[#58161](https://github.com/PaddlePaddle/Paddle/pull/58161),[#58440](https://github.com/PaddlePaddle/Paddle/pull/58440),[#58580](https://github.com/PaddlePaddle/Paddle/pull/58580),[#58950](https://github.com/PaddlePaddle/Paddle/pull/58950),[#58616](https://github.com/PaddlePaddle/Paddle/pull/58616),[#59273](https://github.com/PaddlePaddle/Paddle/pull/59273) -- Optimized XPU memory management to avoid memory leakage. [#59334](https://github.com/PaddlePaddle/Paddle/pull/59334),[#54847](https://github.com/PaddlePaddle/Paddle/pull/54847) -- Supports INT8 inference. [#57258](https://github.com/PaddlePaddle/Paddle/pull/57258) -- Added support for FP16 series inference operators. [#55642](https://github.com/PaddlePaddle/Paddle/pull/55642),[#54410](https://github.com/PaddlePaddle/Paddle/pull/54410) -- Supports share_external_memory interface to pass input and output. [#55170](https://github.com/PaddlePaddle/Paddle/pull/55170) -- Supports open source quantization model XPU inference. [#58568](https://github.com/PaddlePaddle/Paddle/pull/58568) -- Added context_gm_size configuration, instead of allocating global memory in Pass. [#54674](https://github.com/PaddlePaddle/Paddle/pull/54674) -- Added embedding and fast_gather_nd plugin. [#56488](https://github.com/PaddlePaddle/Paddle/pull/56488),[#56103](https://github.com/PaddlePaddle/Paddle/pull/56103) -- Supports fusion of fast_layternorm + leaky_relu. [#57113](https://github.com/PaddlePaddle/Paddle/pull/57113) -- Supports elementwise_min/max/floordiv/where inference in KL1 and KL2 precision. [#58422](https://github.com/PaddlePaddle/Paddle/pull/58422) -- Supports autotune configuration of fc and conv2d operator. [#58801](https://github.com/PaddlePaddle/Paddle/pull/58801) -- Supports conv and fc dynamic quantization. [#59307](https://github.com/PaddlePaddle/Paddle/pull/59307) -- fc + act fusion support for sigmoid, swish and relu6. [#54486](https://github.com/PaddlePaddle/Paddle/pull/54486) -- elementwise_sub/elementwise_div supports int data type. [#55920](https://github.com/PaddlePaddle/Paddle/pull/55920) - -#### Bug Fix - -- Fixed XPU communication library issues and some operator issues including rnn, layer_norm_grad, yolo_box. ([#55475](https://github.com/PaddlePaddle/Paddle/pull/55475),[#55515](https://github.com/PaddlePaddle/Paddle/pull/55515)) ([#55656](https://github.com/PaddlePaddle/Paddle/pull/55656),[#54669](https://github.com/PaddlePaddle/Paddle/pull/54669),[#55310](https://github.com/PaddlePaddle/Paddle/pull/55310) - -### Hygon DCU - -#### Bug Fix - -- Fixed some operator bugs of Hygon DCU, including rnn, concat/split, fft, and so on.[#59402](https://github.com/PaddlePaddle/Paddle/pull/59402),[#55821](https://github.com/PaddlePaddle/Paddle/pull/55821),[#56340](https://github.com/PaddlePaddle/Paddle/pull/56340)) -- Fixed issues related to communication library of Hygon DCU. [#57110](https://github.com/PaddlePaddle/Paddle/pull/57110) -- Fixed compilation-related problems of Hygon DCU. [#59775](https://github.com/PaddlePaddle/Paddle/pull/59775),[#55507](https://github.com/PaddlePaddle/Paddle/pull/55507),[#55612](https://github.com/PaddlePaddle/Paddle/pull/55612),[#54952](https://github.com/PaddlePaddle/Paddle/pull/54952),[#55076](https://github.com/PaddlePaddle/Paddle/pull/55076),[#56079](https://github.com/PaddlePaddle/Paddle/pull/56079),[#54874](https://github.com/PaddlePaddle/Paddle/pull/54874)) -- Fixed support issue of Hygon DCU for BF16 data type. [#56517](https://github.com/PaddlePaddle/Paddle/pull/56517) - -## 6. Environment Adaptation - -Adopted modular compilation to optimize CMake codes, improving efficiency of compilation of PaddlePaddle. This can increase efficiency of RD local development. Meanwhile, supports compilation in Python3.12, CUDA12, and Hopper architecture, and using Clang tool to comprehensively optimize code formats. In addition, C++ unitest is changed from linking static libraries to linking dynamic libraries to reduce compilation size. These improvements provide users with a smoother and more efficient installation and development experience. - -- CMake code optimization: stratify directories into independent static libraries, to improve incremental compilation efficiency. [#59095](https://github.com/PaddlePaddle/Paddle/pull/59095), [#58960](https://github.com/PaddlePaddle/Paddle/pull/58960),[#56591](https://github.com/PaddlePaddle/Paddle/pull/56591),[#58484](https://github.com/PaddlePaddle/Paddle/pull/58484) -- CMake compilation stratification: to realize compilation layering of PaddlePaddle architecture from bottom-up and improve compilation efficiency. [#56442](https://github.com/PaddlePaddle/Paddle/pull/56442),[#54729](https://github.com/PaddlePaddle/Paddle/pull/54729),[#55733](https://github.com/PaddlePaddle/Paddle/pull/55733),[#56352](https://github.com/PaddlePaddle/Paddle/pull/56352),[#55109](https://github.com/PaddlePaddle/Paddle/pull/55109),[#54992](https://github.com/PaddlePaddle/Paddle/pull/54992),[#57698](https://github.com/PaddlePaddle/Paddle/pull/57698),[#55147](https://github.com/PaddlePaddle/Paddle/pull/55147),[#55113](https://github.com/PaddlePaddle/Paddle/pull/55113),[#56691](https://github.com/PaddlePaddle/Paddle/pull/56691),[#58618](https://github.com/PaddlePaddle/Paddle/pull/58618),[#58899](https://github.com/PaddlePaddle/Paddle/pull/58899),[#59140](https://github.com/PaddlePaddle/Paddle/pull/59140),[#59129](https://github.com/PaddlePaddle/Paddle/pull/59129),[#59222](https://github.com/PaddlePaddle/Paddle/pull/59222),[#59105](https://github.com/PaddlePaddle/Paddle/pull/59105),[#59711](https://github.com/PaddlePaddle/Paddle/pull/59711) -- Offline compilation of third-party libraries: Third-party dependent libraries are compiled offline, so CI/CE system does not need to download third-party libraries repeatedly in every compilation, improving operation efficiency of the CI/CE system. [#54344](https://github.com/PaddlePaddle/Paddle/pull/54344),[#54370](https://github.com/PaddlePaddle/Paddle/pull/54370),[#54466](https://github.com/PaddlePaddle/Paddle/pull/54466),[#54438](https://github.com/PaddlePaddle/Paddle/pull/54438),[#54388](https://github.com/PaddlePaddle/Paddle/pull/54388),[#54436](https://github.com/PaddlePaddle/Paddle/pull/54436),[#54392](https://github.com/PaddlePaddle/Paddle/pull/54392),[#54646](https://github.com/PaddlePaddle/Paddle/pull/54646),[#54380](https://github.com/PaddlePaddle/Paddle/pull/54380),[#55501](https://github.com/PaddlePaddle/Paddle/pull/55501),[#55136](https://github.com/PaddlePaddle/Paddle/pull/55136),[#54451](https://github.com/PaddlePaddle/Paddle/pull/54451),[#55631](https://github.com/PaddlePaddle/Paddle/pull/55631),[#55549](https://github.com/PaddlePaddle/Paddle/pull/55549),[#56165](https://github.com/PaddlePaddle/Paddle/pull/56165),[#54391](https://github.com/PaddlePaddle/Paddle/pull/54391),[#54614](https://github.com/PaddlePaddle/Paddle/pull/54614),[#54522](https://github.com/PaddlePaddle/Paddle/pull/54522),[#54764](https://github.com/PaddlePaddle/Paddle/pull/54764),[#54400](https://github.com/PaddlePaddle/Paddle/pull/54400),[#54322](https://github.com/PaddlePaddle/Paddle/pull/54322) -- PaddlePaddle supports Python 3.12. [#59396](https://github.com/PaddlePaddle/Paddle/pull/59396),[#58069](https://github.com/PaddlePaddle/Paddle/pull/58069) -- Using Clang tool to optimize source codes and improve code quality. [#59626](https://github.com/PaddlePaddle/Paddle/pull/59626),[#55895](https://github.com/PaddlePaddle/Paddle/pull/55895),[#56632](https://github.com/PaddlePaddle/Paddle/pull/56632),[#54449](https://github.com/PaddlePaddle/Paddle/pull/54449),[#54523](https://github.com/PaddlePaddle/Paddle/pull/54523),[#54796](https://github.com/PaddlePaddle/Paddle/pull/54796),[#55847](https://github.com/PaddlePaddle/Paddle/pull/55847),[#55807](https://github.com/PaddlePaddle/Paddle/pull/55807),[#56261](https://github.com/PaddlePaddle/Paddle/pull/56261),[#57522](https://github.com/PaddlePaddle/Paddle/pull/57522),[#57868](https://github.com/PaddlePaddle/Paddle/pull/57868),[#57809](https://github.com/PaddlePaddle/Paddle/pull/57809),[#55658](https://github.com/PaddlePaddle/Paddle/pull/55658),[#58285](https://github.com/PaddlePaddle/Paddle/pull/58285),[#55491](https://github.com/PaddlePaddle/Paddle/pull/55491),[#55506](https://github.com/PaddlePaddle/Paddle/pull/55506),[#55279](https://github.com/PaddlePaddle/Paddle/pull/55279),[#55741](https://github.com/PaddlePaddle/Paddle/pull/55741),[#55894](https://github.com/PaddlePaddle/Paddle/pull/55894),[#55704](https://github.com/PaddlePaddle/Paddle/pull/55704),[#55800](https://github.com/PaddlePaddle/Paddle/pull/55800),[#55799](https://github.com/PaddlePaddle/Paddle/pull/55799),[#55983](https://github.com/PaddlePaddle/Paddle/pull/55983),[#55954](https://github.com/PaddlePaddle/Paddle/pull/55954),[#55764](https://github.com/PaddlePaddle/Paddle/pull/55764),[#56246](https://github.com/PaddlePaddle/Paddle/pull/56246),[#56219](https://github.com/PaddlePaddle/Paddle/pull/56219),[#56217](https://github.com/PaddlePaddle/Paddle/pull/56217),[#56216](https://github.com/PaddlePaddle/Paddle/pull/56216),[#56208](https://github.com/PaddlePaddle/Paddle/pull/56208),[#56134](https://github.com/PaddlePaddle/Paddle/pull/56134),[#56253](https://github.com/PaddlePaddle/Paddle/pull/56253),[#56255](https://github.com/PaddlePaddle/Paddle/pull/56255),[#56693](https://github.com/PaddlePaddle/Paddle/pull/56693),[#56692](https://github.com/PaddlePaddle/Paddle/pull/56692),[#56637](https://github.com/PaddlePaddle/Paddle/pull/56637),[#56636](https://github.com/PaddlePaddle/Paddle/pull/56636),[#56647](https://github.com/PaddlePaddle/Paddle/pull/56647),[#56218](https://github.com/PaddlePaddle/Paddle/pull/56218),[#56640](https://github.com/PaddlePaddle/Paddle/pull/56640),[#56635](https://github.com/PaddlePaddle/Paddle/pull/56635),[#55675](https://github.com/PaddlePaddle/Paddle/pull/55675),[#56601](https://github.com/PaddlePaddle/Paddle/pull/56601),[#56485](https://github.com/PaddlePaddle/Paddle/pull/56485),[#56648](https://github.com/PaddlePaddle/Paddle/pull/56648),[#56747](https://github.com/PaddlePaddle/Paddle/pull/56747),[#56676](https://github.com/PaddlePaddle/Paddle/pull/56676),[#56649](https://github.com/PaddlePaddle/Paddle/pull/56649),[#56895](https://github.com/PaddlePaddle/Paddle/pull/56895),[#56994](https://github.com/PaddlePaddle/Paddle/pull/56994),[#56904](https://github.com/PaddlePaddle/Paddle/pull/56904),[#56744](https://github.com/PaddlePaddle/Paddle/pull/56744),[#56954](https://github.com/PaddlePaddle/Paddle/pull/56954),[#57114](https://github.com/PaddlePaddle/Paddle/pull/57114),[#57343](https://github.com/PaddlePaddle/Paddle/pull/57343),[#57483](https://github.com/PaddlePaddle/Paddle/pull/57483),[#57871](https://github.com/PaddlePaddle/Paddle/pull/57871),[#57861](https://github.com/PaddlePaddle/Paddle/pull/57861),[#58028](https://github.com/PaddlePaddle/Paddle/pull/58028),[#57627](https://github.com/PaddlePaddle/Paddle/pull/57627),[#59072](https://github.com/PaddlePaddle/Paddle/pull/59072) -- C++ unitest has changed from linking static libraries to linking dynamic libraries, reducing compilation size and improving compilation efficiency. [#59477](https://github.com/PaddlePaddle/Paddle/pull/59477),[#56630](https://github.com/PaddlePaddle/Paddle/pull/56630),[#57789](https://github.com/PaddlePaddle/Paddle/pull/57789),[#54257](https://github.com/PaddlePaddle/Paddle/pull/54257),[#59620](https://github.com/PaddlePaddle/Paddle/pull/59620),[#59384](https://github.com/PaddlePaddle/Paddle/pull/59384),[#59619](https://github.com/PaddlePaddle/Paddle/pull/59619),[#58583](https://github.com/PaddlePaddle/Paddle/pull/58583),[#58821](https://github.com/PaddlePaddle/Paddle/pull/58821),[#58710](https://github.com/PaddlePaddle/Paddle/pull/58710)，[#58619](https://github.com/PaddlePaddle/Paddle/pull/58619) -- Fixed bug related to source code compilation, improving compilation efficiency. [#56617](https://github.com/PaddlePaddle/Paddle/pull/56617),[#58195](https://github.com/PaddlePaddle/Paddle/pull/58195),[#56136](https://github.com/PaddlePaddle/Paddle/pull/56136),[#54540](https://github.com/PaddlePaddle/Paddle/pull/54540),[#57172](https://github.com/PaddlePaddle/Paddle/pull/57172),[#54429](https://github.com/PaddlePaddle/Paddle/pull/54429),[#55603](https://github.com/PaddlePaddle/Paddle/pull/55603),[#54807](https://github.com/PaddlePaddle/Paddle/pull/54807),[#56102](https://github.com/PaddlePaddle/Paddle/pull/56102),[#56829](https://github.com/PaddlePaddle/Paddle/pull/56829),[#56951](https://github.com/PaddlePaddle/Paddle/pull/56951),[#56555](https://github.com/PaddlePaddle/Paddle/pull/56555),[#57781](https://github.com/PaddlePaddle/Paddle/pull/57781),[#57836](https://github.com/PaddlePaddle/Paddle/pull/57836),[#58807](https://github.com/PaddlePaddle/Paddle/pull/58807),[#54535](https://github.com/PaddlePaddle/Paddle/pull/54535),[#54946](https://github.com/PaddlePaddle/Paddle/pull/54946),[#54437](https://github.com/PaddlePaddle/Paddle/pull/54437),[#54411](https://github.com/PaddlePaddle/Paddle/pull/54411),[#54411](https://github.com/PaddlePaddle/Paddle/pull/54411),[#54391](https://github.com/PaddlePaddle/Paddle/pull/54391),[#54466](https://github.com/PaddlePaddle/Paddle/pull/54466),[#54480](https://github.com/PaddlePaddle/Paddle/pull/54480),[#54480](https://github.com/PaddlePaddle/Paddle/pull/54480),[#54724](https://github.com/PaddlePaddle/Paddle/pull/54724),[#59193](https://github.com/PaddlePaddle/Paddle/pull/59193),[#54735](https://github.com/PaddlePaddle/Paddle/pull/54735),[#54812](https://github.com/PaddlePaddle/Paddle/pull/54812),[#56430](https://github.com/PaddlePaddle/Paddle/pull/56430),[#56655](https://github.com/PaddlePaddle/Paddle/pull/56655),[#56684](https://github.com/PaddlePaddle/Paddle/pull/56684),[#56774](https://github.com/PaddlePaddle/Paddle/pull/56774),[#56936](https://github.com/PaddlePaddle/Paddle/pull/56936),[#56949](https://github.com/PaddlePaddle/Paddle/pull/56949),[#56974](https://github.com/PaddlePaddle/Paddle/pull/56974),[#57171](https://github.com/PaddlePaddle/Paddle/pull/57171),[#57712](https://github.com/PaddlePaddle/Paddle/pull/57712),[#56617](https://github.com/PaddlePaddle/Paddle/pull/56617),[#58181](https://github.com/PaddlePaddle/Paddle/pull/58181),[#58253](https://github.com/PaddlePaddle/Paddle/pull/58253),[#58268](https://github.com/PaddlePaddle/Paddle/pull/58268),[#59051](https://github.com/PaddlePaddle/Paddle/pull/59051),[#59048](https://github.com/PaddlePaddle/Paddle/pull/59048),[#59081](https://github.com/PaddlePaddle/Paddle/pull/59081),[#59076](https://github.com/PaddlePaddle/Paddle/pull/59076),[#59155](https://github.com/PaddlePaddle/Paddle/pull/59155),[#59253](https://github.com/PaddlePaddle/Paddle/pull/59253),[#59347](https://github.com/PaddlePaddle/Paddle/pull/59347),[#58957](https://github.com/PaddlePaddle/Paddle/pull/58957),[#59443](https://github.com/PaddlePaddle/Paddle/pull/59443),[#58998](https://github.com/PaddlePaddle/Paddle/pull/58998),[#57574](https://github.com/PaddlePaddle/Paddle/pull/57574),[#55889](https://github.com/PaddlePaddle/Paddle/pull/55889),[#59078](https://github.com/PaddlePaddle/Paddle/pull/59078),[#55762](https://github.com/PaddlePaddle/Paddle/pull/55762),[#56252](https://github.com/PaddlePaddle/Paddle/pull/56252),[#56715](https://github.com/PaddlePaddle/Paddle/pull/56715),[#54905](https://github.com/PaddlePaddle/Paddle/pull/54905),[#56978](https://github.com/PaddlePaddle/Paddle/pull/56978),[#57032](https://github.com/PaddlePaddle/Paddle/pull/57032),[#57179](https://github.com/PaddlePaddle/Paddle/pull/57179),[#57179](https://github.com/PaddlePaddle/Paddle/pull/57179),[#58996](https://github.com/PaddlePaddle/Paddle/pull/58996),[#59915](https://github.com/PaddlePaddle/Paddle/pull/59915),[#54883](https://github.com/PaddlePaddle/Paddle/pull/54883),[#56746](https://github.com/PaddlePaddle/Paddle/pull/56746),[#57674](https://github.com/PaddlePaddle/Paddle/pull/57674),[#60117](https://github.com/PaddlePaddle/Paddle/pull/60117),[#55627](https://github.com/PaddlePaddle/Paddle/pull/55627),[#54568](https://github.com/PaddlePaddle/Paddle/pull/54568),[#54450](https://github.com/PaddlePaddle/Paddle/pull/54450),[#54513](https://github.com/PaddlePaddle/Paddle/pull/54513),[#54615](https://github.com/PaddlePaddle/Paddle/pull/54615),[#54913](https://github.com/PaddlePaddle/Paddle/pull/54913),[#54916](https://github.com/PaddlePaddle/Paddle/pull/54916),[#55148](https://github.com/PaddlePaddle/Paddle/pull/55148),[#55125](https://github.com/PaddlePaddle/Paddle/pull/55125),[#55479](https://github.com/PaddlePaddle/Paddle/pull/55479),[#55723](https://github.com/PaddlePaddle/Paddle/pull/55723),[#55831](https://github.com/PaddlePaddle/Paddle/pull/55831),[#55904](https://github.com/PaddlePaddle/Paddle/pull/55904),[#56085](https://github.com/PaddlePaddle/Paddle/pull/56085),[#56259](https://github.com/PaddlePaddle/Paddle/pull/56259),[#56366](https://github.com/PaddlePaddle/Paddle/pull/56366),[#56366](https://github.com/PaddlePaddle/Paddle/pull/56366),[#56546](https://github.com/PaddlePaddle/Paddle/pull/56546),[#56679](https://github.com/PaddlePaddle/Paddle/pull/56679),[#57222](https://github.com/PaddlePaddle/Paddle/pull/57222),[#57387](https://github.com/PaddlePaddle/Paddle/pull/57387),[#57993](https://github.com/PaddlePaddle/Paddle/pull/57993),[#59556](https://github.com/PaddlePaddle/Paddle/pull/59556),[#57931](https://github.com/PaddlePaddle/Paddle/pull/57931),[#58112](https://github.com/PaddlePaddle/Paddle/pull/58112),[#54228](https://github.com/PaddlePaddle/Paddle/pull/54228),[#56913](https://github.com/PaddlePaddle/Paddle/pull/56913),[#56993](https://github.com/PaddlePaddle/Paddle/pull/56993),[#55042](https://github.com/PaddlePaddle/Paddle/pull/55042),[#55305](https://github.com/PaddlePaddle/Paddle/pull/55305),[#55286](https://github.com/PaddlePaddle/Paddle/pull/55286),[#56634](https://github.com/PaddlePaddle/Paddle/pull/56634),[#57778](https://github.com/PaddlePaddle/Paddle/pull/57778),[#58374](https://github.com/PaddlePaddle/Paddle/pull/58374),[#58640](https://github.com/PaddlePaddle/Paddle/pull/58640),[#58822](https://github.com/PaddlePaddle/Paddle/pull/58822),[#59055](https://github.com/PaddlePaddle/Paddle/pull/59055),[#59303](https://github.com/PaddlePaddle/Paddle/pull/59303),[#59487](https://github.com/PaddlePaddle/Paddle/pull/59487),[#58400](https://github.com/PaddlePaddle/Paddle/pull/58400),[#59283](https://github.com/PaddlePaddle/Paddle/pull/59283),[#54791](https://github.com/PaddlePaddle/Paddle/pull/54791),[#59134](https://github.com/PaddlePaddle/Paddle/pull/59134),[#56206](https://github.com/PaddlePaddle/Paddle/pull/56206),[#56199](https://github.com/PaddlePaddle/Paddle/pull/56199),[#56670](https://github.com/PaddlePaddle/Paddle/pull/56670),[#58923](https://github.com/PaddlePaddle/Paddle/pull/58923) -- Fixed bug related to Paddle ARM compilation. [#55416](https://github.com/PaddlePaddle/Paddle/pull/55416),[#55548](https://github.com/PaddlePaddle/Paddle/pull/55548) - -## Thanks to Our Contributors - -Azure-Tang, zhaoyinglia, From00, JZ-LIANG, xysheng-baidu, SylarTiaNII, kuizhiqing, zhiqiu, FeixLiu, liuzhenhai93, GhostScreaming, pangengzheng, xiaoyewww, wanghuancoder, ForFishes, hitywt, danleifeng, tianshuo78520a, ykkk2333, houj04, lj970926, XiaociZhang, HarperCy, cqulilujia, runzhech, RuohengMa, Caozhou1995, kangguangli, heavyrain-lzy, zyfncg, SigureMo, YuanRisheng, lchdl, LiYuRio, AndSonder, Wennie396, zhangbo9674, liudongxue01, risemeup1, phlrain, winter-wang, yuanlehome, NALLEIN, Liujie0926, yuguo-Jack, gitliuyf, zh794390558, Aurelius84, 6clc, GGBond8488, xiaoguoguo626807, Wong4j, iosmers, xiaoxiaohehe001, LielinJiang, carryyu, Difers, yangxiaoyu14, xuxinyi389, cxxly, gongshaotian, jjyaoao, lijialin03, lxd-cumt, cyber-pioneer, HydrogenSulfate, MayYouBeProsperous, Charles-hit, Patrick-Star125, ScottWong98, huangjiyi, DrRyanHuang, jinyouzhi, BeingGod, Wanglongzhi2001, yangguohao, zyt1024, longranger2, 2742195759, megemini, thisjiang, kevincheng2, zhoutianzi666, Wangzheee, ming1753, tianhaodongbd, freeliuzc, zhenyun-li, MARD1NO, RichardWooSJTU, eee4017, leo0519, csy0225, wwbitejotunn, bukejiyu, jiweibo, iamsonderr, ckl117, ronny1996, zhanglirong1999, LLee233, ZHUI, wangxn12138, zhwesky2010, Courtesy-Xs, zoooo0820, llyyxx0413, Asthestarsfalll, zxcd, pkuzyc, idontkonwher, sneaxiy, hong19860320, ZibinGuo, leolishaohao, MuShangCC, zhupengyang, shentanyue, Travis-Lee, wz1qqx, frank-oops, newway, QingshuChen, zhangyk0314, HandSomeLEEw, Shixiaowei02, zhangyuqin1998, Xing-lil, zhhsplendid, jiahy0825, xinyu-intel, MarioLulab, 0x45f, Tom-Zheng, xingmingyyj, zhangbopd, gouzil, zeroRains, BiynXu, WintersMontagne10335, wuhuachaocoding, GreatV, chenwhql, deepllz, parap1uie-s, ozogxyz, FisherWY, changeyoung98, zhiboniu, YangQun1 dynamicheart, Xreki, liugddx, Lylinnnnn, YSF-A, zzjjay, YanhuiDua, lishicheng1996, USTCKAY, abenmao, cocoshe, HermitSun, ccsuzzh, sanbuphy, enkilee, RedContritio, Liyulingyue, zrr1999, chen2016013, Galaxy1458, chalsliu, mrcangye, XieYunshen, zhiheng-liu, haohongxiang, ZzSean, JamesLim-sy, yuehuayingxueluo, niuliling123, umiswing, sijunhe, littsk, SecretXV, zhurou603, zhangjun, caizejun, yangjianfengo1, vivienfanghuagood, Xinyu302, lizexu123, yghstill, Li-fAngyU, VigiZhang, co63oc, dhanush-2501, ooooo-create, PommesPeter, zeus2x7, akshatvishu, jzhang533, Sekiro-x, gumblex, BernieHuang2008, YibinLiu666, qiuwenbogdut, XavierZXY, MqLeet, zhangting2020, mingxu1067, Ainavo, SSKlearns, yuchen202, silverling, zade23, wenxiaohahaha, NKNaN, Tsaiyue, fsczz, Tomoko-hjf, rhmaaa, zbt78, Hhankyangg, wangzhen38, zhengqiwen1997, engineer1109, onepick, qili93, Rane2021, nemonameless, DesmonDay, RachelXu7, ceci3, lyuwenyu, liuruyan, LokeZhou, shiyutang, lanxianghit, feifei-111, Sahala08, sunzhongkai588, Kaedeharai, Candy2Tang, liyongchao911, whisky-12, InsaneOnion, yoyoIcy, KongAKun, linzeyang, MuhammadNizamani, eltociear, Ligoml, LUZY0726, Windfarer, FlyingQianMM, jeng1220, junelotus, zlsh80826, Vvsmile, Frida-a, TonibMw, guoshengCS, zhink, ZhangYulongg, AlbertVan, fengxin-hello, mjp9527, entired, DanGuge. - -# 2.5.0 Release Note - -## 1. Highlights -- **New dynamic-static unification architecture**: Implement a new dynamic-to-static plus compiler execution model in combination with the basic operator, and complete the whole dynamic-to-static, combinator and neural network compiler optimization and acceleration process on the ResNet50&Bert model. For the dynamic-to-static, complete the whole graph fallback core function development, and support the fallback to dynamic graph training execution in case of dynamic-to-static failure. For the combinator, design a set of basic operator systems containing more than 150 basic operators, to achieve the python layer forward operator splitting mechanism and the reverse operator splitting mechanism of static graphs, to realize splitting of more than 70 commonly used forward and reverse operators. For the CINN compiler, fix the correctness bug, develop the key Pass, add manual schedule rules, achieve automatic generation of kernel codes, and improve performance of ResNet50 model by 12% and Bert model by 10%. -- **Operator architecture unification of PHI operator library**: Unify all remaining 350+ operator kernels under the original operator system into PHI operator Library. Unify the way of defining operator in the original operator system into the operator definition form of PHI operator library (configuration of operator definition based on YAML), enhancing unity of the architecture, and reducing comprehension cost of framework development. Decouple all the Fluid header files that the PHI operator library depends on and compile them independently as dynamic link libraries to provide a lighter reuse of the operator library for secondary development of the framework. Continue to standardize and adjust unspecified operators, as well as operator kernels in the PaddlePaddle framework. It is easy for developers to understand and reduce the cost of accessing the hardware. -- **Full go-live of new actuator for static graph**: The new actuator for static graph implements a number of functions and performance optimization, and completes unification and replacement of the original multiple sets of old actuators. The new actuator becomes the back-end default execution engine for the static graph single card and distributed training python side entrance, as well as dynamic-to-static, control flow, CINN, etc. This significantly improves scheduling performance of the framework, and the functional architecture is clearer. Secondary development capability is significantly enhanced. -- **Python API supporting 0-dimensional tensor**: clear semantics are defined between tensor of shape [1,] and tensor of shape [], and fixed many API behaviors to support tensor of shape [], such as `paddle.sum` etc. -- **New environment adaptation**: Adapt to CUDA 12. Compilation with gcc12 is supported. - -## **2. Incompatibility Upgrade** -- PaddlePaddle API supports 0-dimensional tensor.PaddlePaddle previously used a 1-dimensional tensor with a shape of [1] instead of a 0-dimensional tensor, which is different from current mainstream habits. It increases development and debugging cost of the model, and sometimes leads to unintended errors. This release fixes 376 APIs that need to support 0-dimensional tensor, and implements tools widely used by the community such as EinOps. For example, in previous cases, output loss in model training was a 1-dimensional tensor. To take out or print the loss, it was often necessary to use codes like `loss.numpy()[0]`.After this modification, output loss in model training is a 0-dimensional tensor. When using `loss.numpy()`, users can take out or print the loss. The codes are short, easy to understand, and in line with the industry's habit. -- `paddle.fluid ` API is fully decommissioned. According to the plan that has been previewed in the last version, 1116 `paddle.fluid ` APIs and related internal interfaces have been decommissioned, and the remaining few related internal interfaces will be cleaned up in the next version.fluid API belongs to the historical APIs that PaddlePaddle 2.0 had planned to remove, but delayed the cleanup in consideration of compatibility and other factors. This decommissioning cleanup will not affect programs developed based on PaddlePaddle 2.0, and the PaddlePaddle API system will be more concise and easier to understand. -- Complete code cleanup at the old version of the dynamic graph Python side.So far, the Python side only uses the new version of dynamic graph to call the C++ core logic. -- In order to unify the training method of data parallel for static graph model, original single-process multi-card training method is abandoned, including `paddle.static.ParallelExecutor ` and `paddle.static. CompiledProgram(). with_data_parallel( )` APIs, because this set of APIs only supports single-computer multi-card, does not support multi-computer multi-card, and the underlying execution performance is poor.It is recommended to use the multi-process multi-card training method uniformly, i.e., `paddle.distributed.launch ` API for distributed training with data parallel. This upgrade affects only static graphs, and does not affect dynamic graphs and dynamic-to-static training. If you use the decommissioned API, please refer to the documentation on [data parallel](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/06_distributed_training/cluster_quick_start_collective_cn.html) to modify model code. [#50351](https://github.com/PaddlePaddle/Paddle/pull/50351)，[#50501](https://github.com/PaddlePaddle/Paddle/pull/50501)，[#51240](https://github.com/PaddlePaddle/Paddle/pull/51240)，[#51701](https://github.com/PaddlePaddle/Paddle/pull/51701)，[#51616](https://github.com/PaddlePaddle/Paddle/pull/51616)，[#51369](https://github.com/PaddlePaddle/Paddle/pull/51369)，[#52671](https://github.com/PaddlePaddle/Paddle/pull/52671) -- Remove the original adaptation code of Ascend NPU and Cambricon MLU in the framework, upgrade all to CustomDevice plug-in adaptation, and migrate the adaptation code of Ascend NPU and Cambricon MLU to PaddleCustomDevice warehouse. - -## 3. Training Framework (Including Distributed) -### Python API -#### API supporting 0-dimensional tensor -- API input supports 0-dimensional tensor, involving `paddle.reshape `, `paddle.trace `, `paddle.linalg.norm ` and other 286 APIs. [#53208](https://github.com/PaddlePaddle/Paddle/pull/53208), [#53592](https://github.com/PaddlePaddle/Paddle/pull/53592), [#47074](https://github.com/PaddlePaddle/Paddle/pull/47074), [#53186](https://github.com/PaddlePaddle/Paddle/pull/53186), [#47677](https://github.com/PaddlePaddle/Paddle/pull/47677), [#49357](https://github.com/PaddlePaddle/Paddle/pull/49357), [#50237](https://github.com/PaddlePaddle/Paddle/pull/50237), [#46555](https://github.com/PaddlePaddle/Paddle/pull/46555), [#47219](https://github.com/PaddlePaddle/Paddle/pull/47219), [#47501](https://github.com/PaddlePaddle/Paddle/pull/47501), [#47858](https://github.com/PaddlePaddle/Paddle/pull/47858), [#47961](https://github.com/PaddlePaddle/Paddle/pull/47961), [#48058](https://github.com/PaddlePaddle/Paddle/pull/48058), [#48007](https://github.com/PaddlePaddle/Paddle/pull/48007), [#49755](https://github.com/PaddlePaddle/Paddle/pull/49755), [#51024](https://github.com/PaddlePaddle/Paddle/pull/51024), [#51566](https://github.com/PaddlePaddle/Paddle/pull/51566), [#51899](https://github.com/PaddlePaddle/Paddle/pull/51899), [#49813](https://github.com/PaddlePaddle/Paddle/pull/49813), [#47812](https://github.com/PaddlePaddle/Paddle/pull/47812), [#47849](https://github.com/PaddlePaddle/Paddle/pull/47849), [#47251](https://github.com/PaddlePaddle/Paddle/pull/47251), [#53125](https://github.com/PaddlePaddle/Paddle/pull/53125), [#53828](https://github.com/PaddlePaddle/Paddle/pull/53828), [#51265](https://github.com/PaddlePaddle/Paddle/pull/51265), [#47689](https://github.com/PaddlePaddle/Paddle/pull/47689), [#48452](https://github.com/PaddlePaddle/Paddle/pull/48452), [#49072](https://github.com/PaddlePaddle/Paddle/pull/49072), [#48638](https://github.com/PaddlePaddle/Paddle/pull/48638), [#49175](https://github.com/PaddlePaddle/Paddle/pull/49175), [#49279](https://github.com/PaddlePaddle/Paddle/pull/49279), [#50857](https://github.com/PaddlePaddle/Paddle/pull/50857), [#49805](https://github.com/PaddlePaddle/Paddle/pull/49805), [#47734](https://github.com/PaddlePaddle/Paddle/pull/47734), [#45992](https://github.com/PaddlePaddle/Paddle/pull/45992), [#49616](https://github.com/PaddlePaddle/Paddle/pull/49616), [#49959](https://github.com/PaddlePaddle/Paddle/pull/49959), [#50536](https://github.com/PaddlePaddle/Paddle/pull/50536), [#49544](https://github.com/PaddlePaddle/Paddle/pull/49544), [#49842](https://github.com/PaddlePaddle/Paddle/pull/49842), [#46909](https://github.com/PaddlePaddle/Paddle/pull/46909), [#49361](https://github.com/PaddlePaddle/Paddle/pull/49361), [#50169](https://github.com/PaddlePaddle/Paddle/pull/50169), [#48314](https://github.com/PaddlePaddle/Paddle/pull/48314), [#48735](https://github.com/PaddlePaddle/Paddle/pull/48735), [#49122](https://github.com/PaddlePaddle/Paddle/pull/49122), [#49122](https://github.com/PaddlePaddle/Paddle/pull/49122), [#49177](https://github.com/PaddlePaddle/Paddle/pull/49177), [#49501](https://github.com/PaddlePaddle/Paddle/pull/49501), [#49562](https://github.com/PaddlePaddle/Paddle/pull/49562), [#49340](https://github.com/PaddlePaddle/Paddle/pull/49340), [#49550](https://github.com/PaddlePaddle/Paddle/pull/49550), [#49596](https://github.com/PaddlePaddle/Paddle/pull/49596), [#49730](https://github.com/PaddlePaddle/Paddle/pull/49730), [#49667](https://github.com/PaddlePaddle/Paddle/pull/49667), [#49692](https://github.com/PaddlePaddle/Paddle/pull/49692), [#49854](https://github.com/PaddlePaddle/Paddle/pull/49854), [#49845](https://github.com/PaddlePaddle/Paddle/pull/49845), [#49803](https://github.com/PaddlePaddle/Paddle/pull/49803), [#49889](https://github.com/PaddlePaddle/Paddle/pull/49889), [#49904](https://github.com/PaddlePaddle/Paddle/pull/49904), [#49518](https://github.com/PaddlePaddle/Paddle/pull/49518), [#49884](https://github.com/PaddlePaddle/Paddle/pull/49884), [#49880](https://github.com/PaddlePaddle/Paddle/pull/49880), [#49862](https://github.com/PaddlePaddle/Paddle/pull/49862), [#49921](https://github.com/PaddlePaddle/Paddle/pull/49921), [#49260](https://github.com/PaddlePaddle/Paddle/pull/49260), [#49929](https://github.com/PaddlePaddle/Paddle/pull/49929), [#49570](https://github.com/PaddlePaddle/Paddle/pull/49570), [#49882](https://github.com/PaddlePaddle/Paddle/pull/49882), [#50213](https://github.com/PaddlePaddle/Paddle/pull/50213), [#49780](https://github.com/PaddlePaddle/Paddle/pull/49780), [#50271](https://github.com/PaddlePaddle/Paddle/pull/50271), [#50289](https://github.com/PaddlePaddle/Paddle/pull/50289), [#50293](https://github.com/PaddlePaddle/Paddle/pull/50293), [#49735](https://github.com/PaddlePaddle/Paddle/pull/49735), [#50433](https://github.com/PaddlePaddle/Paddle/pull/50433), [#49847](https://github.com/PaddlePaddle/Paddle/pull/49847), [#50635](https://github.com/PaddlePaddle/Paddle/pull/50635), [#50950](https://github.com/PaddlePaddle/Paddle/pull/50950), [#50947](https://github.com/PaddlePaddle/Paddle/pull/50947), [#49460](https://github.com/PaddlePaddle/Paddle/pull/49460), [#53087](https://github.com/PaddlePaddle/Paddle/pull/53087), [#51687](https://github.com/PaddlePaddle/Paddle/pull/51687), [#52185](https://github.com/PaddlePaddle/Paddle/pull/52185), [#54649](https://github.com/PaddlePaddle/Paddle/pull/54649) -- API output supports 0-dimensional tensor, involving `paddle.sum `, `paddle.min/max `, `paddle.any/all ` and other 90 APIs. [#52891](https://github.com/PaddlePaddle/Paddle/pull/52891), [#52861](https://github.com/PaddlePaddle/Paddle/pull/52861), [#52775](https://github.com/PaddlePaddle/Paddle/pull/52775), [#52850](https://github.com/PaddlePaddle/Paddle/pull/52850), [#52843](https://github.com/PaddlePaddle/Paddle/pull/52843), [#52857](https://github.com/PaddlePaddle/Paddle/pull/52857), [#51721](https://github.com/PaddlePaddle/Paddle/pull/51721), [#53051](https://github.com/PaddlePaddle/Paddle/pull/53051), [#53192](https://github.com/PaddlePaddle/Paddle/pull/53192), [#52739](https://github.com/PaddlePaddle/Paddle/pull/52739), [#52741](https://github.com/PaddlePaddle/Paddle/pull/52741), [#53175](https://github.com/PaddlePaddle/Paddle/pull/53175), [#51889](https://github.com/PaddlePaddle/Paddle/pull/51889), [#53199](https://github.com/PaddlePaddle/Paddle/pull/53199), [#53242](https://github.com/PaddlePaddle/Paddle/pull/53242), [#53421](https://github.com/PaddlePaddle/Paddle/pull/53421) -- In addition to the support of 0-dimensional tensor, fix the original non-standard codes, and provide hints and compatibility for non-standard usage in the model codes. [#51562](https://github.com/PaddlePaddle/Paddle/pull/51562), [#51586](https://github.com/PaddlePaddle/Paddle/pull/51586), [#51757](https://github.com/PaddlePaddle/Paddle/pull/51757), [#52197](https://github.com/PaddlePaddle/Paddle/pull/52197), [#54117](https://github.com/PaddlePaddle/Paddle/pull/54117)。 - -#### new API -- Add `paddle.autograd.jacobian` and `paddle.autograd.hessian` APIs for scientific computing. [#53331](https://github.com/PaddlePaddle/Paddle/pull/53331) -- Add sparse computing API. For example, `paddle.sparse.reshape `, `paddle.sparse.sum ` and `paddle.sparse.slice `. [#46694](https://github.com/PaddlePaddle/Paddle/pull/46694), [#51513](https://github.com/PaddlePaddle/Paddle/pull/51513), [#53794](https://github.com/PaddlePaddle/Paddle/pull/53794), [#51406](https://github.com/PaddlePaddle/Paddle/pull/51406) -- Add APIsFor example, `paddle.optimizer.LBFGS `, `paddle.index_put ` and `paddle.logaddexp `. [#53314](https://github.com/PaddlePaddle/Paddle/pull/53314), [#51912](https://github.com/PaddlePaddle/Paddle/pull/51912), [#52886](https://github.com/PaddlePaddle/Paddle/pull/52886), [#50843](https://github.com/PaddlePaddle/Paddle/pull/50843), [#47282](https://github.com/PaddlePaddle/Paddle/pull/47282), [#52284](https://github.com/PaddlePaddle/Paddle/pull/52284) - -### Dynamic graphs -#### New features -- Add paddle.nn.utils.clip_grad_norm_ for gradient clipping support and paddle.Tensor.data_ptr for getting the address of the Tensor data's memory/GPU memory. [PR49935](https://github.com/PaddlePaddle/Paddle/pull/49935)[, PR48235](https://github.com/PaddlePaddle/Paddle/pull/48235), [PR49173](https://github.com/PaddlePaddle/Paddle/pull/49173) -- Add the saved_tensors_hooks mechanism, for temporary storage and retrieval of forward Tensor used in backward computation. [PR45763](https://github.com/PaddlePaddle/Paddle/pull/45763), [PR46215](https://github.com/PaddlePaddle/Paddle/pull/46215), [PR48124](https://github.com/PaddlePaddle/Paddle/pull/48124) -- Tensor supports pickler, for serialization of Tensor. [PR47025](https://github.com/PaddlePaddle/Paddle/pull/47025), [PR48179](https://github.com/PaddlePaddle/Paddle/pull/48179) -- Add debug logs, to print forward Python stacks when nan/inf appears in reverse. [PR53217](https://github.com/PaddlePaddle/Paddle/pull/53217) [PR52639](https://github.com/PaddlePaddle/Paddle/pull/52639) [PR52729](https://github.com/PaddlePaddle/Paddle/pull/52729) -- Add the support for expand_v2, tile, concat, assign, slice higher-order differentiation. [PR45941](https://github.com/PaddlePaddle/Paddle/pull/45941), [PR45942](https://github.com/PaddlePaddle/Paddle/pull/45942), [PR45940](https://github.com/PaddlePaddle/Paddle/pull/45940), [PR45879](https://github.com/PaddlePaddle/Paddle/pull/45879), [PR45960](https://github.com/PaddlePaddle/Paddle/pull/45960) - -#### Improvements -- Optimize log printing for dynamic graphs, including log content, VLog level, and error reporting content. [PR45783](https://github.com/PaddlePaddle/Paddle/pull/45783), [PR46349](https://github.com/PaddlePaddle/Paddle/pull/46349), [PR46934](https://github.com/PaddlePaddle/Paddle/pull/46934), [PR47724](https://github.com/PaddlePaddle/Paddle/pull/47724) -- Add FLAGS_auto_growth_chunk_size_in_mb for minimum chunk size settings of auto_growth_allocator. [PR52204](https://github.com/PaddlePaddle/Paddle/pull/52204) - -#### bug fix -- Fix bugs in some operators, including batch_norm, slice, set_value, scale, multinomial, adam, conv, transpose2_grad, conv2d_transpose_double_grad. [PR47802](https://github.com/PaddlePaddle/Paddle/pull/47802), [PR47634](https://github.com/PaddlePaddle/Paddle/pull/47634), [PR47349](https://github.com/PaddlePaddle/Paddle/pull/47349), [PR46124](https://github.com/PaddlePaddle/Paddle/pull/46124), [PR46147](https://github.com/PaddlePaddle/Paddle/pull/46147), [PR50388](https://github.com/PaddlePaddle/Paddle/pull/50388), [PR48626](https://github.com/PaddlePaddle/Paddle/pull/48626), [PR48519](https://github.com/PaddlePaddle/Paddle/pull/48519), [PR50386](https://github.com/PaddlePaddle/Paddle/pull/50386), [PR48432](https://github.com/PaddlePaddle/Paddle/pull/48432), [PR51851](https://github.com/PaddlePaddle/Paddle/pull/51851) -- Fix some PyLayer bugs. [PR51740](https://github.com/PaddlePaddle/Paddle/pull/51740), [PR47154](https://github.com/PaddlePaddle/Paddle/pull/47154), [PR47323](https://github.com/PaddlePaddle/Paddle/pull/47323), [PR54041](https://github.com/PaddlePaddle/Paddle/pull/54041), [PR48533](https://github.com/PaddlePaddle/Paddle/pull/48533) -- Makes sure sync_batch_norm is sequential in reverse to avoid hang or precision errors due to misordering. [PR52268](https://github.com/PaddlePaddle/Paddle/pull/52268), [PR52860](https://github.com/PaddlePaddle/Paddle/pull/52860), [PR52779](https://github.com/PaddlePaddle/Paddle/pull/52779) -- Fix a bug of linspace under AMP. [PR46088](https://github.com/PaddlePaddle/Paddle/pull/46088) -- Fix Python C API’s incorrect call that causes Windows to crash. [PR46833](https://github.com/PaddlePaddle/Paddle/pull/46833) -- Fix the bug that DataLoader may miss deleting/dev/shm. [PR48511](https://github.com/PaddlePaddle/Paddle/pull/48511) -- Fix some bugs of paddle.grad. [PR47151](https://github.com/PaddlePaddle/Paddle/pull/47151) -- Add error message for operators that do not support higher order differentiation. [PR47231](https://github.com/PaddlePaddle/Paddle/pull/47231) -- Add numpyarray support for python operators. [PR48229](https://github.com/PaddlePaddle/Paddle/pull/48229) -- Delete either of element_size APIs. [PR49631](https://github.com/PaddlePaddle/Paddle/pull/49631) -- Fix the bug of crash when opening old dynamic graph VLOG. [PR47115](https://github.com/PaddlePaddle/Paddle/pull/47115) -- For XPU, change to d2h+h2d in case of d2d, to solve the multi-threading problem. [PR48373](https://github.com/PaddlePaddle/Paddle/pull/48373) - -#### Performance optimization -- Python operators sink to C++ implementation, to improve API performance. There is a 3x to 6x performance improvement in this class of APIs after sinking. [PR45811](https://github.com/PaddlePaddle/Paddle/pull/45811), [PR46326](https://github.com/PaddlePaddle/Paddle/pull/46326), [PR46329](https://github.com/PaddlePaddle/Paddle/pull/46329), [PR46520](https://github.com/PaddlePaddle/Paddle/pull/46520), [PR46542](https://github.com/PaddlePaddle/Paddle/pull/46542), [PR46565](https://github.com/PaddlePaddle/Paddle/pull/46565), [PR47060](https://github.com/PaddlePaddle/Paddle/pull/47060), [PR47077](https://github.com/PaddlePaddle/Paddle/pull/47077), [PR47174](https://github.com/PaddlePaddle/Paddle/pull/47174), [PR47315](https://github.com/PaddlePaddle/Paddle/pull/47315) -- Optimize the Optimizer CPU scheduling performance to reduce GPU Gap caused by Optimizer phase. [PR49787](https://github.com/PaddlePaddle/Paddle/pull/49787), [PR50188](https://github.com/PaddlePaddle/Paddle/pull/50188)[, PR51340](https://github.com/PaddlePaddle/Paddle/pull/51340), [PR49864](https://github.com/PaddlePaddle/Paddle/pull/49864), [PR50158](https://github.com/PaddlePaddle/Paddle/pull/50158), [PR50335](https://github.com/PaddlePaddle/Paddle/pull/50335) -- According to the logic that API can be sunk to C++, API is sunk to C++ to improve API performance. [PR46412](https://github.com/PaddlePaddle/Paddle/pull/46412), [PR46190](https://github.com/PaddlePaddle/Paddle/pull/46190) -- Optimize unnecessary call logic on Python side under dynamic graph, to improve API performance. [PR46221](https://github.com/PaddlePaddle/Paddle/pull/46221), [PR49473](https://github.com/PaddlePaddle/Paddle/pull/49473), [PR49574](https://github.com/PaddlePaddle/Paddle/pull/49574), [PR49589](https://github.com/PaddlePaddle/Paddle/pull/49589), [PR49612](https://github.com/PaddlePaddle/Paddle/pull/49612), [PR49717](https://github.com/PaddlePaddle/Paddle/pull/49717)[, PR49733](https://github.com/PaddlePaddle/Paddle/pull/49733), [PR49823](https://github.com/PaddlePaddle/Paddle/pull/49823)[, PR49508](https://github.com/PaddlePaddle/Paddle/pull/49508), [PR46840](https://github.com/PaddlePaddle/Paddle/pull/46840) -- Optimize use of Allocator to improve dynamic graph API scheduling performance. [PR47125](https://github.com/PaddlePaddle/Paddle/pull/47125), [PR48548](https://github.com/PaddlePaddle/Paddle/pull/48548), [PR50995](https://github.com/PaddlePaddle/Paddle/pull/50995), [PR47731](https://github.com/PaddlePaddle/Paddle/pull/47731) -- Optimize fused_attention operator performance. [PR48902](https://github.com/PaddlePaddle/Paddle/pull/48902) -- For optimizer's _add_accumulator, if device is CPU and under dynamic graphs, use full to initialize var directly. [PR48189](https://github.com/PaddlePaddle/Paddle/pull/48189) -- Prune unnecessarily executed subgraphs for inverse graphs to improve performance. [PR47827](https://github.com/PaddlePaddle/Paddle/pull/47827) -- Optimize performance of initalizers. [PR46033](https://github.com/PaddlePaddle/Paddle/pull/46033) -- Add fused dropout add operator to improve computation performance when dropout and add are used together. [#52903](https://github.com/PaddlePaddle/Paddle/pull/52903) - -### Static graphs -#### The new static graph executor is now fully go-live. -The new actuator for static graph implements a number of functions and performance optimizations, and completes unification and replacement of the original multiple sets of old actuators. The new actuator becomes the back-end default execution engine for the static graph single card and distributed training python side entrance, as well as dynamic-to-static, control flow, CINN, etc. This significantly improves scheduling performance of the framework, and the functional architecture is clearer. Secondary development capability is significantly enhanced. [#45913](https://github.com/PaddlePaddle/Paddle/pull/45913)，[#46025](https://github.com/PaddlePaddle/Paddle/pull/46025)，[#48911](https://github.com/PaddlePaddle/Paddle/pull/48911)，[#50239](https://github.com/PaddlePaddle/Paddle/pull/50239)，[#45696](https://github.com/PaddlePaddle/Paddle/pull/45696)，[#46092](https://github.com/PaddlePaddle/Paddle/pull/46092)，[#48158](https://github.com/PaddlePaddle/Paddle/pull/48158),[#51389](https://github.com/PaddlePaddle/Paddle/pull/51389)，[#49708](https://github.com/PaddlePaddle/Paddle/pull/49708)，[#49275](https://github.com/PaddlePaddle/Paddle/pull/49275),[#48789](https://github.com/PaddlePaddle/Paddle/pull/48789)，[#49939](https://github.com/PaddlePaddle/Paddle/pull/49939)，[#51149](https://github.com/PaddlePaddle/Paddle/pull/51149)，[#52652](https://github.com/PaddlePaddle/Paddle/pull/52652) - -### Operator library -#### Enhance functions of customized operators -New function support for custom extension mechanism to achieve the C++ extension of the arithmetic function binding to the Python side, to further enhance the framework's secondary development capabilities. The extension supports custom hardware to use a custom operator mechanism to meet the needs of hardware manufacturers to implement non-Paddle existing operations. The extension supports custom operators in the implementation of the `inplace `, `vector < Tensor> ` output, `optional < Tnesor> ` input and other high-level mechanisms in custom operators. Optimized scheduling performance of custom operators in dynamic graph mode, with a 25.4% performance improvement for operators with multiple input parameters. Add new commonly used operators and APIs for custom operator Tensor extensions. Support chaining calls and simplify code writing. Optimize the operator kernel selection mechanism. Improve the logic of some operator kernels, enhance supported data types and optimize performance. Add and improve XPU kernels 100+. Fix 170+ bugs. -[#49222](https://github.com/PaddlePaddle/Paddle/pull/49222), [#51773](https://github.com/PaddlePaddle/Paddle/pull/51773), [#51923](https://github.com/PaddlePaddle/Paddle/pull/51923), [#53080](https://github.com/PaddlePaddle/Paddle/pull/53080), [#50731](https://github.com/PaddlePaddle/Paddle/pull/50731), [#50563](https://github.com/PaddlePaddle/Paddle/pull/50563), [#50840](https://github.com/PaddlePaddle/Paddle/pull/50840), [#50983](https://github.com/PaddlePaddle/Paddle/pull/50983), [#51713](https://github.com/PaddlePaddle/Paddle/pull/51713), [#48733](https://github.com/PaddlePaddle/Paddle/pull/48733), [#50558](https://github.com/PaddlePaddle/Paddle/pull/50558), [#50764](https://github.com/PaddlePaddle/Paddle/pull/50764), [#51973](https://github.com/PaddlePaddle/Paddle/pull/51973), [#52216](https://github.com/PaddlePaddle/Paddle/pull/52216), [#51027](https://github.com/PaddlePaddle/Paddle/pull/51027), [#50745](https://github.com/PaddlePaddle/Paddle/pull/50745), [#50756](https://github.com/PaddlePaddle/Paddle/pull/50756), [#50886](https://github.com/PaddlePaddle/Paddle/pull/50886), [#50813](https://github.com/PaddlePaddle/Paddle/pull/50813), [#50869](https://github.com/PaddlePaddle/Paddle/pull/50869), [#51085](https://github.com/PaddlePaddle/Paddle/pull/51085), [#51646](https://github.com/PaddlePaddle/Paddle/pull/51646), [#51620](https://github.com/PaddlePaddle/Paddle/pull/51620), [#51844](https://github.com/PaddlePaddle/Paddle/pull/51844), [#52421](https://github.com/PaddlePaddle/Paddle/pull/52421), [#52872](https://github.com/PaddlePaddle/Paddle/pull/52872), [#52597](https://github.com/PaddlePaddle/Paddle/pull/52597), [#50582](https://github.com/PaddlePaddle/Paddle/pull/50582), [#52114](https://github.com/PaddlePaddle/Paddle/pull/52114), [#52915](https://github.com/PaddlePaddle/Paddle/pull/52915), [#50928](https://github.com/PaddlePaddle/Paddle/pull/50928), [#48272](https://github.com/PaddlePaddle/Paddle/pull/48272), [#48702](https://github.com/PaddlePaddle/Paddle/pull/48702), [#52191](https://github.com/PaddlePaddle/Paddle/pull/52191), [#52191](https://github.com/PaddlePaddle/Paddle/pull/52191), [#47374](https://github.com/PaddlePaddle/Paddle/pull/47374), [#47375](https://github.com/PaddlePaddle/Paddle/pull/47375), [#47378](https://github.com/PaddlePaddle/Paddle/pull/47378), [#54126](https://github.com/PaddlePaddle/Paddle/pull/54126), [#47638](https://github.com/PaddlePaddle/Paddle/pull/47638), [#47661](https://github.com/PaddlePaddle/Paddle/pull/47661), [#50606](https://github.com/PaddlePaddle/Paddle/pull/50606), [#53528](https://github.com/PaddlePaddle/Paddle/pull/53528), [#50599](https://github.com/PaddlePaddle/Paddle/pull/50599), [#51727](https://github.com/PaddlePaddle/Paddle/pull/51727), [#50825](https://github.com/PaddlePaddle/Paddle/pull/50825), [#50773](https://github.com/PaddlePaddle/Paddle/pull/50773), [#50979](https://github.com/PaddlePaddle/Paddle/pull/50979), [#53336](https://github.com/PaddlePaddle/Paddle/pull/53336), [#53555](https://github.com/PaddlePaddle/Paddle/pull/53555), [#53716](https://github.com/PaddlePaddle/Paddle/pull/53716), [#53753](https://github.com/PaddlePaddle/Paddle/pull/53753), [#53981](https://github.com/PaddlePaddle/Paddle/pull/53981), [#53977](https://github.com/PaddlePaddle/Paddle/pull/53977), [#53980](https://github.com/PaddlePaddle/Paddle/pull/53980), [#54043](https://github.com/PaddlePaddle/Paddle/pull/54043), [#54066](https://github.com/PaddlePaddle/Paddle/pull/54066), [#52866](https://github.com/PaddlePaddle/Paddle/pull/52866), [#53043](https://github.com/PaddlePaddle/Paddle/pull/53043), [#53325](https://github.com/PaddlePaddle/Paddle/pull/53325), [#54323](https://github.com/PaddlePaddle/Paddle/pull/54323), [#54367](https://github.com/PaddlePaddle/Paddle/pull/54367), [#51353](https://github.com/PaddlePaddle/Paddle/pull/51353), [#53749](https://github.com/PaddlePaddle/Paddle/pull/53749), [#50013](https://github.com/PaddlePaddle/Paddle/pull/50013), [#47570](https://github.com/PaddlePaddle/Paddle/pull/47570), [#50997](https://github.com/PaddlePaddle/Paddle/pull/50997), [#51241](https://github.com/PaddlePaddle/Paddle/pull/51241), [#49537](https://github.com/PaddlePaddle/Paddle/pull/49537) - -#### Unification of operator architecture -Unify all remaining 350+ operator kernels under the original operator system into PHI operator library. Unify the way of defining operator in the original operator system into the operator definition form of PHI operator library (configuration of operator definition based on YAML), enhancing unity of the architecture, and reducing comprehension cost of framework development. Decouple all Fluid header files the PHI operator library depends on and compile them independently as dynamic link libraries to provide a lighter reuse of the operator library for secondary development of the framework. Continue to standardize and adjust unspecified operators, as well as operator kernels in the PaddlePaddle framework. It is easy for developers to understand and reduce cost of accessing hardware. -[#47856](https://github.com/PaddlePaddle/Paddle/pull/47856), [#49328](https://github.com/PaddlePaddle/Paddle/pull/49328), [#49138](https://github.com/PaddlePaddle/Paddle/pull/49138), [#52014](https://github.com/PaddlePaddle/Paddle/pull/52014), [#52044](https://github.com/PaddlePaddle/Paddle/pull/52044), [#52116](https://github.com/PaddlePaddle/Paddle/pull/52116), [#52486](https://github.com/PaddlePaddle/Paddle/pull/52486), [#52101](https://github.com/PaddlePaddle/Paddle/pull/52101), [#52882](https://github.com/PaddlePaddle/Paddle/pull/52882), [#53003](https://github.com/PaddlePaddle/Paddle/pull/53003), [#53034](https://github.com/PaddlePaddle/Paddle/pull/53034), [#51914](https://github.com/PaddlePaddle/Paddle/pull/51914), [#49116](https://github.com/PaddlePaddle/Paddle/pull/49116), [#52626](https://github.com/PaddlePaddle/Paddle/pull/52626), [#52878](https://github.com/PaddlePaddle/Paddle/pull/52878), [#52879](https://github.com/PaddlePaddle/Paddle/pull/52879), [#52880](https://github.com/PaddlePaddle/Paddle/pull/52880), [#52875](https://github.com/PaddlePaddle/Paddle/pull/52875), [#51600](https://github.com/PaddlePaddle/Paddle/pull/51600), [#51601](https://github.com/PaddlePaddle/Paddle/pull/51601), [#51590](https://github.com/PaddlePaddle/Paddle/pull/51590), [#51887](https://github.com/PaddlePaddle/Paddle/pull/51887), [#51891](https://github.com/PaddlePaddle/Paddle/pull/51891), [#52036](https://github.com/PaddlePaddle/Paddle/pull/52036), [#52130](https://github.com/PaddlePaddle/Paddle/pull/52130), [#52134](https://github.com/PaddlePaddle/Paddle/pull/52134), [#51951](https://github.com/PaddlePaddle/Paddle/pull/51951), [#51886](https://github.com/PaddlePaddle/Paddle/pull/51886), [#52274](https://github.com/PaddlePaddle/Paddle/pull/52274), [#52263](https://github.com/PaddlePaddle/Paddle/pull/52263), [#51913](https://github.com/PaddlePaddle/Paddle/pull/51913), [#52145](https://github.com/PaddlePaddle/Paddle/pull/52145), [#52347](https://github.com/PaddlePaddle/Paddle/pull/52347), [#52370](https://github.com/PaddlePaddle/Paddle/pull/52370), [#52437](https://github.com/PaddlePaddle/Paddle/pull/52437), [#52424](https://github.com/PaddlePaddle/Paddle/pull/52424), [#52231](https://github.com/PaddlePaddle/Paddle/pull/52231), [#52522](https://github.com/PaddlePaddle/Paddle/pull/52522), [#52529](https://github.com/PaddlePaddle/Paddle/pull/52529), [#52802](https://github.com/PaddlePaddle/Paddle/pull/52802), [#52799](https://github.com/PaddlePaddle/Paddle/pull/52799), [#52855](https://github.com/PaddlePaddle/Paddle/pull/52855), [#52711](https://github.com/PaddlePaddle/Paddle/pull/52711), [#52940](https://github.com/PaddlePaddle/Paddle/pull/52940), [#53309](https://github.com/PaddlePaddle/Paddle/pull/53309), [#47817](https://github.com/PaddlePaddle/Paddle/pull/47817), [#48001](https://github.com/PaddlePaddle/Paddle/pull/48001), [#48063](https://github.com/PaddlePaddle/Paddle/pull/48063), [#48049](https://github.com/PaddlePaddle/Paddle/pull/48049), [#48168](https://github.com/PaddlePaddle/Paddle/pull/48168), [#48415](https://github.com/PaddlePaddle/Paddle/pull/48415), [#48696](https://github.com/PaddlePaddle/Paddle/pull/48696), [#48970](https://github.com/PaddlePaddle/Paddle/pull/48970), [#50183](https://github.com/PaddlePaddle/Paddle/pull/50183), [#50407](https://github.com/PaddlePaddle/Paddle/pull/50407), [#50498](https://github.com/PaddlePaddle/Paddle/pull/50498), [#50419](https://github.com/PaddlePaddle/Paddle/pull/50419), [#50282](https://github.com/PaddlePaddle/Paddle/pull/50282), [#50870](https://github.com/PaddlePaddle/Paddle/pull/50870), [#50911](https://github.com/PaddlePaddle/Paddle/pull/50911), [#50865](https://github.com/PaddlePaddle/Paddle/pull/50865), [#51288](https://github.com/PaddlePaddle/Paddle/pull/51288), [#53735](https://github.com/PaddlePaddle/Paddle/pull/53735), [#47248](https://github.com/PaddlePaddle/Paddle/pull/47248), [#47787](https://github.com/PaddlePaddle/Paddle/pull/47787), [#52202](https://github.com/PaddlePaddle/Paddle/pull/52202), -[#47579](https://github.com/PaddlePaddle/Paddle/pull/47579), [#49444](https://github.com/PaddlePaddle/Paddle/pull/49444), [#45772](https://github.com/PaddlePaddle/Paddle/pull/45772), [#51264](https://github.com/PaddlePaddle/Paddle/pull/51264), [#51634](https://github.com/PaddlePaddle/Paddle/pull/51634), [#51631](https://github.com/PaddlePaddle/Paddle/pull/51631), [#47385](https://github.com/PaddlePaddle/Paddle/pull/47385), [#46342](https://github.com/PaddlePaddle/Paddle/pull/46342), [#47510](https://github.com/PaddlePaddle/Paddle/pull/47510), [#47532](https://github.com/PaddlePaddle/Paddle/pull/47532), [#47702](https://github.com/PaddlePaddle/Paddle/pull/47702), [#47860](https://github.com/PaddlePaddle/Paddle/pull/47860), [#49470](https://github.com/PaddlePaddle/Paddle/pull/49470), [#50358](https://github.com/PaddlePaddle/Paddle/pull/50358), [#49121](https://github.com/PaddlePaddle/Paddle/pull/49121), [#50190](https://github.com/PaddlePaddle/Paddle/pull/50190), [#52374](https://github.com/PaddlePaddle/Paddle/pull/52374), [#52372](https://github.com/PaddlePaddle/Paddle/pull/52372), [#52375](https://github.com/PaddlePaddle/Paddle/pull/52375), [#52371](https://github.com/PaddlePaddle/Paddle/pull/52371) - -### Dynamic-to-static plus combinator -#### New features -- Add the combination rules for combinators such as dropout, silu, stack, relu, expand, unsqueeze, pow, squeeze, meshgrid, batch_norm, layer_norm, group_norm, instance_norm, full_like, split, split_with_num, gelu, mean, flatten, rsqrt, hadswish [#50497](https://github.com/PaddlePaddle/Paddle/pull/50497), [#50838](https://github.com/PaddlePaddle/Paddle/pull/50838), [#50861](https://github.com/PaddlePaddle/Paddle/pull/50861), [#50819](https://github.com/PaddlePaddle/Paddle/pull/50819), [#50810](https://github.com/PaddlePaddle/Paddle/pull/50810), [#51527](https://github.com/PaddlePaddle/Paddle/pull/51527), [#51070](https://github.com/PaddlePaddle/Paddle/pull/51070), [#51539](https://github.com/PaddlePaddle/Paddle/pull/51539), [#51061](https://github.com/PaddlePaddle/Paddle/pull/51061), [#49894](https://github.com/PaddlePaddle/Paddle/pull/49894), [#50422](https://github.com/PaddlePaddle/Paddle/pull/50422), [#51874](https://github.com/PaddlePaddle/Paddle/pull/51874), [#51341](https://github.com/PaddlePaddle/Paddle/pull/51341), [#50295](https://github.com/PaddlePaddle/Paddle/pull/50295), [#50298](https://github.com/PaddlePaddle/Paddle/pull/50298), [#50672](https://github.com/PaddlePaddle/Paddle/pull/50672), [#51432](https://github.com/PaddlePaddle/Paddle/pull/51432), [#51003](https://github.com/PaddlePaddle/Paddle/pull/51003) -- Add the vjp rule for combinators such as gather_nd, reduce_max, group_norm, relu, reduce_max, gather, topk, sqrt, elementwise_pow, softmax, batch_norm, prod, multiply, expand, div, relu, slice, cumsum, sigmoid, layer_norm, sin, cos, roll, instance_norm, abs, assign, tile, scatter_nd_add, erf, floor, log, silu, leaky_relu, pad [#50966](https://github.com/PaddlePaddle/Paddle/pull/50966), [#51653](https://github.com/PaddlePaddle/Paddle/pull/51653), [#52663](https://github.com/PaddlePaddle/Paddle/pull/52663), [#51742](https://github.com/PaddlePaddle/Paddle/pull/51742), [#52203](https://github.com/PaddlePaddle/Paddle/pull/52203), [#50794](https://github.com/PaddlePaddle/Paddle/pull/50794), [#50305](https://github.com/PaddlePaddle/Paddle/pull/50305), [#50786](https://github.com/PaddlePaddle/Paddle/pull/50786), [#50679](https://github.com/PaddlePaddle/Paddle/pull/50679), [#51045](https://github.com/PaddlePaddle/Paddle/pull/51045), [#51230](https://github.com/PaddlePaddle/Paddle/pull/51230), [#51474](https://github.com/PaddlePaddle/Paddle/pull/51474), [#51283](https://github.com/PaddlePaddle/Paddle/pull/51283), [#51238](https://github.com/PaddlePaddle/Paddle/pull/51238), [#49831](https://github.com/PaddlePaddle/Paddle/pull/49831), [#51838](https://github.com/PaddlePaddle/Paddle/pull/51838), [#50771](https://github.com/PaddlePaddle/Paddle/pull/50771), [#50565](https://github.com/PaddlePaddle/Paddle/pull/50565), [#51768](https://github.com/PaddlePaddle/Paddle/pull/51768), [#51750](https://github.com/PaddlePaddle/Paddle/pull/51750), [#51748](https://github.com/PaddlePaddle/Paddle/pull/51748), [#52532](https://github.com/PaddlePaddle/Paddle/pull/52532), [#52935](https://github.com/PaddlePaddle/Paddle/pull/52935), [#50963](https://github.com/PaddlePaddle/Paddle/pull/50963), [#51430](https://github.com/PaddlePaddle/Paddle/pull/51430), [#53141](https://github.com/PaddlePaddle/Paddle/pull/53141), [#52469](https://github.com/PaddlePaddle/Paddle/pull/52469), [#50436](https://github.com/PaddlePaddle/Paddle/pull/50436), [#51059](https://github.com/PaddlePaddle/Paddle/pull/51059), [#51296](https://github.com/PaddlePaddle/Paddle/pull/51296), [#52533](https://github.com/PaddlePaddle/Paddle/pull/52533), [#53374](https://github.com/PaddlePaddle/Paddle/pull/53374) -- Add the second-order differentiation rule for combinators such as matmul, tanh, and elementwise [#50452](https://github.com/PaddlePaddle/Paddle/pull/50452), [#52192](https://github.com/PaddlePaddle/Paddle/pull/52192), [#53014](https://github.com/PaddlePaddle/Paddle/pull/53014) -- Add the bf16 datatype support for combinators such as exp, reduce_mean, softmax, divide, cast, layer_norm, prod, meshgrid, expand_as, dropout, concat, gather_nd, elementwise_max, elementwise_pow, reduce_max [#54263](https://github.com/PaddlePaddle/Paddle/pull/54263)， [#54236](https://github.com/PaddlePaddle/Paddle/pull/54236), [#53865](https://github.com/PaddlePaddle/Paddle/pull/53865), [#54175](https://github.com/PaddlePaddle/Paddle/pull/54175), [#54399](https://github.com/PaddlePaddle/Paddle/pull/54399) -- Add support for assigning semantics to containers in control flow in dynamic-to-static. [#51248](https://github.com/PaddlePaddle/Paddle/pull/51248) -- For to_static, add full graph fallback function. When dynamic-to-static conversion fails, the whole graph can fall back to the dynamic graph mode of execution. For the fallback mechanism, add the set_eval_frame API. [#50111](https://github.com/PaddlePaddle/Paddle/pull/50111), [#52006](https://github.com/PaddlePaddle/Paddle/pull/52006) -- For to_static, support the combinator mechanism. Support the scenario of using register_hook under to_static decoration; [#49836](https://github.com/PaddlePaddle/Paddle/pull/49836), [#52948](https://github.com/PaddlePaddle/Paddle/pull/52948), [#53572](https://github.com/PaddlePaddle/Paddle/pull/53572) -- Add a backend parameter to the to_static API. It can be specified as `CINN` or None. When the parameter is specified as CINN, the CINN compiler will be used to accelerate training and inference. [#52596](https://github.com/PaddlePaddle/Paddle/pull/52596) -- Add the code automatic generation function for the primitive API. Based on operator definitions in ops.yaml and legacy_ops.yaml, automatically generate code for the primitive API. Automatically generate the Tensor computation API. [#50315](https://github.com/PaddlePaddle/Paddle/pull/50315), [#49654](https://github.com/PaddlePaddle/Paddle/pull/49654), [#50642](https://github.com/PaddlePaddle/Paddle/pull/50642) -- Add the function of forward combination of operators. By registering the combination rules of forward operators, it can split forward operators into base operators. [#49605](https://github.com/PaddlePaddle/Paddle/pull/49605) -- Add the combinator switch. You can set environmental variables in shell to split operators in different ways. [#50309](https://github.com/PaddlePaddle/Paddle/pull/50309) -- Add `OpTest ` combination test function to guarantee accuracy of operators. Add elementwise class base operator unit test. Add batch_norm CINN unit test. [#50509](https://github.com/PaddlePaddle/Paddle/pull/50509), [#50807](https://github.com/PaddlePaddle/Paddle/pull/50807), [#52815](https://github.com/PaddlePaddle/Paddle/pull/52815) - -#### Improvements -- Add combinator to support FP16 operation and AMP O1 operation. Add AMP logic for softmax and layer_norm operators. [#52397](https://github.com/PaddlePaddle/Paddle/pull/52397), [#52598](https://github.com/PaddlePaddle/Paddle/pull/52598), [#51473](https://github.com/PaddlePaddle/Paddle/pull/51473) -- Simplify combination rules and vjp rules of the combinator batch_norm. [#54012](https://github.com/PaddlePaddle/Paddle/pull/54012), [#51827](https://github.com/PaddlePaddle/Paddle/pull/51827), [#51933](https://github.com/PaddlePaddle/Paddle/pull/51933), -- Optimize combination rules for combinators, and improve performance of combination rules with containing scalar. Optimize log printing for combinators. [#51960](https://github.com/PaddlePaddle/Paddle/pull/51960), [#50160](https://github.com/PaddlePaddle/Paddle/pull/50160) -- Combinator supports the jit.save API. Add custom VJP rule API. [#52344](https://github.com/PaddlePaddle/Paddle/pull/52344), [#50885](https://github.com/PaddlePaddle/Paddle/pull/50885) -- Remove the overwrite parameter from combinator gather_grad. [#52707](https://github.com/PaddlePaddle/Paddle/pull/52707) -- Clean up dynamic-to-static code style, optimize error message, and standardize logs. [#48637](https://github.com/PaddlePaddle/Paddle/pull/48637), [#46128](https://github.com/PaddlePaddle/Paddle/pull/46128), [#52527](https://github.com/PaddlePaddle/Paddle/pull/52527), [#46800](https://github.com/PaddlePaddle/Paddle/pull/46800),[#46415](https://github.com/PaddlePaddle/Paddle/pull/46415) -- For dynamic-to-static, call the append backward to get `grad var name ` to fix the error in the high order gradient computation. [#53250](https://github.com/PaddlePaddle/Paddle/pull/53250) -- Upgrade the dynamic-to-static function, and clean up the temporary directory of to_static to speed up code conversion. Enhance to_static to automatically skip internal API. Support use of to_static decorator in the program. [#47102](https://github.com/PaddlePaddle/Paddle/pull/47102), [#50596](https://github.com/PaddlePaddle/Paddle/pull/50596), [#45768](https://github.com/PaddlePaddle/Paddle/pull/45768) -- For dynamic-to-static, optimize `print ` function conversion to support printing Tensor parameters at the networking stage. Upgrade the parameter collection mechanism. [#48672](https://github.com/PaddlePaddle/Paddle/pull/48672), [#50336](https://github.com/PaddlePaddle/Paddle/pull/50336) - -#### bug fix -- For the combinator, fix cmake compilation errors. Fix cuda 12 test errors. Fix bugs of operators such as meshgird, expand_as, concat, conv, and arrange. [#49643](https://github.com/PaddlePaddle/Paddle/pull/49643), [#54622](https://github.com/PaddlePaddle/Paddle/pull/54622), [#53951](https://github.com/PaddlePaddle/Paddle/pull/53951), [#53951](https://github.com/PaddlePaddle/Paddle/pull/53951), [#53350](https://github.com/PaddlePaddle/Paddle/pull/53350), [#51486](https://github.com/PaddlePaddle/Paddle/pull/51486), [#52764](https://github.com/PaddlePaddle/Paddle/pull/52764) -- For the combinator, fix the bug in a number of scenarios such as rank=1, shape=-1, amp, and multi-process. [#51413](https://github.com/PaddlePaddle/Paddle/pull/51413), [#51435](https://github.com/PaddlePaddle/Paddle/pull/51435), [#50518](https://github.com/PaddlePaddle/Paddle/pull/50518), [#47301](https://github.com/PaddlePaddle/Paddle/pull/47301), -- For the combinator, fix bugs in automatic code generation of composite grad maker and static prim api. Fix bugs that op creation attributes are missing, and some combination rules do not take effect. [#50854](https://github.com/PaddlePaddle/Paddle/pull/50854), [#51445](https://github.com/PaddlePaddle/Paddle/pull/51445), [#50780](https://github.com/PaddlePaddle/Paddle/pull/50780), [#52120](https://github.com/PaddlePaddle/Paddle/pull/52120) -- Fix some other bugs for combinators [#50086](https://github.com/PaddlePaddle/Paddle/pull/50086), [#51208](https://github.com/PaddlePaddle/Paddle/pull/51208), [#51577](https://github.com/PaddlePaddle/Paddle/pull/51577), [#53598](https://github.com/PaddlePaddle/Paddle/pull/53598), [#47500](https://github.com/PaddlePaddle/Paddle/pull/47500), [#52119](https://github.com/PaddlePaddle/Paddle/pull/52119), [#50397](https://github.com/PaddlePaddle/Paddle/pull/50397), [#50527](https://github.com/PaddlePaddle/Paddle/pull/50527), [#50788](https://github.com/PaddlePaddle/Paddle/pull/50788), [#51014](https://github.com/PaddlePaddle/Paddle/pull/51014), [#52154](https://github.com/PaddlePaddle/Paddle/pull/52154), [#52752](https://github.com/PaddlePaddle/Paddle/pull/52752) -- For dynamic-to-static, fix the bugs of dataloader, cond input dict, transformer import, T5 model memory leak, and grad var name parsing error. [#49821](https://github.com/PaddlePaddle/Paddle/pull/49821)， [#47299](https://github.com/PaddlePaddle/Paddle/pull/47299), [#50776](https://github.com/PaddlePaddle/Paddle/pull/50776), [#50883](https://github.com/PaddlePaddle/Paddle/pull/50883), [#51100](https://github.com/PaddlePaddle/Paddle/pull/51100), [#51464](https://github.com/PaddlePaddle/Paddle/pull/51464), [#51966](https://github.com/PaddlePaddle/Paddle/pull/51966), [#52110](https://github.com/PaddlePaddle/Paddle/pull/52110), [#52821](https://github.com/PaddlePaddle/Paddle/pull/52821) -- For dynamic-to-static, fix the bugs of Lazy initialization, Windows training, is_paddle_func failure, and recurrent op failure to delete pass. [#50785](https://github.com/PaddlePaddle/Paddle/pull/50785), [#52580](https://github.com/PaddlePaddle/Paddle/pull/52580), [#51585](https://github.com/PaddlePaddle/Paddle/pull/51585), [#51763](https://github.com/PaddlePaddle/Paddle/pull/51763), [#51763](https://github.com/PaddlePaddle/Paddle/pull/51763) - -#### Performance optimization -- Add scope caching and reuse mechanism during execution of run_program_op in dynamic-to-static, to avoid passing new scope for each step. [#45813](https://github.com/PaddlePaddle/Paddle/pull/45813) - -### Distributed training -#### Dynamic graph distributed training -- Remove the distributed sharding API in the old dynamic graphs. [#49334](https://github.com/PaddlePaddle/Paddle/pull/49334) -- Upgrade fleet to distributed directory. [#50834](https://github.com/PaddlePaddle/Paddle/pull/50834) -- Optimize log printing for distributed strategies. [#47761](https://github.com/PaddlePaddle/Paddle/pull/47761) -- For re-computation, support hook mode, inplace function, and stop_gradient mode. Support more flexible use. [#48471](https://github.com/PaddlePaddle/Paddle/pull/48471), [#47985](https://github.com/PaddlePaddle/Paddle/pull/47985) -- Data parallel - - For data parallel, support no_sync API for blocking parameter gradient communications. Support the parameter synchronization function. Add scale API to scale parameters. [#47536](https://github.com/PaddlePaddle/Paddle/pull/47536),[#51895](https://github.com/PaddlePaddle/Paddle/pull/51895),[#47519](https://github.com/PaddlePaddle/Paddle/pull/47519) - - Fix the problem of video memory leakage under data parallel. [#47369](https://github.com/PaddlePaddle/Paddle/pull/47369),[#47444](https://github.com/PaddlePaddle/Paddle/pull/47444),[#48668](https://github.com/PaddlePaddle/Paddle/pull/48668) - - Support sparse parameter gradient synchronization. [#52785](https://github.com/PaddlePaddle/Paddle/pull/52785) -- Pipeline parallel - - Optimize pipeline performance, and remove communication wait. Optimize scheduling and communication overlap. [#46209](https://github.com/PaddlePaddle/Paddle/pull/46209),[#54003](https://github.com/PaddlePaddle/Paddle/pull/54003),[#54312](https://github.com/PaddlePaddle/Paddle/pull/54312),[#53384](https://github.com/PaddlePaddle/Paddle/pull/53384),[#54310](https://github.com/PaddlePaddle/Paddle/pull/54310),[#46399](https://github.com/PaddlePaddle/Paddle/pull/46399),[#46483](https://github.com/PaddlePaddle/Paddle/pull/46483),[#46780](https://github.com/PaddlePaddle/Paddle/pull/46780),[#46116](https://github.com/PaddlePaddle/Paddle/pull/46116) - - Support custom sharding, log printing, random seed setting, and timer elapsed time printing. [#53344](https://github.com/PaddlePaddle/Paddle/pull/53344), [#47670](https://github.com/PaddlePaddle/Paddle/pull/47670),[#47336](https://github.com/PaddlePaddle/Paddle/pull/47336),[#52656](https://github.com/PaddlePaddle/Paddle/pull/52656),[#53831](https://github.com/PaddlePaddle/Paddle/pull/53831) - - Optimize video memory release logic in pipeline scheduling, and release intermediate variables and data in advance. [#54557](https://github.com/PaddlePaddle/Paddle/pull/54557), [#47199](https://github.com/PaddlePaddle/Paddle/pull/47199),[#47497](https://github.com/PaddlePaddle/Paddle/pull/47497),[#48045](https://github.com/PaddlePaddle/Paddle/pull/48045),[#54672](https://github.com/PaddlePaddle/Paddle/pull/54672) - - Support VPP mode and model saving for pipeline parallel. [#54196](https://github.com/PaddlePaddle/Paddle/pull/54196), [#52927](https://github.com/PaddlePaddle/Paddle/pull/52927),[#47801](https://github.com/PaddlePaddle/Paddle/pull/47801),[#45922](https://github.com/PaddlePaddle/Paddle/pull/45922),[#47242](https://github.com/PaddlePaddle/Paddle/pull/47242) -- Grouping sharding parallel - - sharding stage2 parallel supports the quantization function, hybrid parallel training, gradient accumulation, XPU hardware, BF16 low precision computation, optimizer learning rate setting, offload function, and data parallel. [#47169](https://github.com/PaddlePaddle/Paddle/pull/47169),[#47535](https://github.com/PaddlePaddle/Paddle/pull/47535), [#46795](https://github.com/PaddlePaddle/Paddle/pull/46795),[#47711](https://github.com/PaddlePaddle/Paddle/pull/47711),[#48310](https://github.com/PaddlePaddle/Paddle/pull/48310),[#46846](https://github.com/PaddlePaddle/Paddle/pull/46846),[#48857](https://github.com/PaddlePaddle/Paddle/pull/48857),[#49196](https://github.com/PaddlePaddle/Paddle/pull/49196),[#49931](https://github.com/PaddlePaddle/Paddle/pull/49931),[#47114](https://github.com/PaddlePaddle/Paddle/pull/47114),[#49767](https://github.com/PaddlePaddle/Paddle/pull/49767) - - Optimize sharing stage2 performance. Support the communication computation overlap. [#46495](https://github.com/PaddlePaddle/Paddle/pull/46495),[#46894](https://github.com/PaddlePaddle/Paddle/pull/46894) - - sharding stage3 support shared parameters, and untrainable parameters. [#48695](https://github.com/PaddlePaddle/Paddle/pull/48695),[#48577](https://github.com/PaddlePaddle/Paddle/pull/48577) -- Tensor model parallel - - Optimize tensor model parallel performance to reduce performance impact of stream sharding. [#47715](https://github.com/PaddlePaddle/Paddle/pull/47715),[#51617](https://github.com/PaddlePaddle/Paddle/pull/51617) - - Support parameter, optimizer shapes, gradient synchronization. [#51428](https://github.com/PaddlePaddle/Paddle/pull/51428),[#53254](https://github.com/PaddlePaddle/Paddle/pull/53254), [#53335](https://github.com/PaddlePaddle/Paddle/pull/53335),[#45803](https://github.com/PaddlePaddle/Paddle/pull/45803),[#46303](https://github.com/PaddlePaddle/Paddle/pull/46303),[#52293](https://github.com/PaddlePaddle/Paddle/pull/52293) - - Optimize tensor model parallel operators such as c_embedding, softmax_with_corss_entropy. [#53197](https://github.com/PaddlePaddle/Paddle/pull/53197),[#53547](https://github.com/PaddlePaddle/Paddle/pull/53547),[#53541](https://github.com/PaddlePaddle/Paddle/pull/53541),[#52789](https://github.com/PaddlePaddle/Paddle/pull/52789),[#46491](https://github.com/PaddlePaddle/Paddle/pull/46491),[#52742](https://github.com/PaddlePaddle/Paddle/pull/52742),[#53419](https://github.com/PaddlePaddle/Paddle/pull/53419) -- Launch - - Support distributed Launch function, with keeping independent logs. [#53207](https://github.com/PaddlePaddle/Paddle/pull/53207),[#50405](https://github.com/PaddlePaddle/Paddle/pull/50405) - - Add framework print environment variable function, log overwrite function, log return, and environment check. It is easy to change the debug environment variable. [#53243](https://github.com/PaddlePaddle/Paddle/pull/53243),[#53243](https://github.com/PaddlePaddle/Paddle/pull/53243), [#51803](https://github.com/PaddlePaddle/Paddle/pull/51803), [#53990](https://github.com/PaddlePaddle/Paddle/pull/53990) -- Communication library - - Add custom mixed parallel communication groups, topology information printing, and custom communication topology order. [#47021](https://github.com/PaddlePaddle/Paddle/pull/47021),[#54000](https://github.com/PaddlePaddle/Paddle/pull/54000),[#51781](https://github.com/PaddlePaddle/Paddle/pull/51781) - - Remove communication library dependency on Place information [#47857](https://github.com/PaddlePaddle/Paddle/pull/47857) - - Add communications library to support GLOO operator. Support send/recv/gather. [#52221](https://github.com/PaddlePaddle/Paddle/pull/52221), [#52334](https://github.com/PaddlePaddle/Paddle/pull/52334),[#49084](https://github.com/PaddlePaddle/Paddle/pull/49084) - - Disable reverse computation of communication operator. [#47636](https://github.com/PaddlePaddle/Paddle/pull/47636) - - Add communication library static shape check, to help determine whether communication volume is matched. [#48256](https://github.com/PaddlePaddle/Paddle/pull/48256),[#48915](https://github.com/PaddlePaddle/Paddle/pull/48915),[#48646](https://github.com/PaddlePaddle/Paddle/pull/48646) - - Support communication python object type, BF16 type, alltoall, reduce, allgather, group call, global gather, broadcast, and scatter communication methods. Support XPU device communications. [#51765](https://github.com/PaddlePaddle/Paddle/pull/51765),[#45844](https://github.com/PaddlePaddle/Paddle/pull/45844),[#48059](https://github.com/PaddlePaddle/Paddle/pull/48059),[#48115](https://github.com/PaddlePaddle/Paddle/pull/48115), [#48339](https://github.com/PaddlePaddle/Paddle/pull/48339),[#49252](https://github.com/PaddlePaddle/Paddle/pull/49252),[#49451](https://github.com/PaddlePaddle/Paddle/pull/49451),[#50085](https://github.com/PaddlePaddle/Paddle/pull/50085),[#50701](https://github.com/PaddlePaddle/Paddle/pull/50701),[#48208](https://github.com/PaddlePaddle/Paddle/pull/48208),[#48736](https://github.com/PaddlePaddle/Paddle/pull/48736),[#51762](https://github.com/PaddlePaddle/Paddle/pull/51762),[#52495](https://github.com/PaddlePaddle/Paddle/pull/52495),[#53514](https://github.com/PaddlePaddle/Paddle/pull/53514),[#48232](https://github.com/PaddlePaddle/Paddle/pull/48232),[#49896](https://github.com/PaddlePaddle/Paddle/pull/49896),[#49941](https://github.com/PaddlePaddle/Paddle/pull/49941),[#45584](https://github.com/PaddlePaddle/Paddle/pull/45584) - - Add support for communications between computational streams. [#46182](https://github.com/PaddlePaddle/Paddle/pull/46182),[#46023](https://github.com/PaddlePaddle/Paddle/pull/46023),[#46295](https://github.com/PaddlePaddle/Paddle/pull/46295),[#46761](https://github.com/PaddlePaddle/Paddle/pull/46761),[#47481](https://github.com/PaddlePaddle/Paddle/pull/47481),[#47740](https://github.com/PaddlePaddle/Paddle/pull/47740),[#47976](https://github.com/PaddlePaddle/Paddle/pull/47976),[#48163](https://github.com/PaddlePaddle/Paddle/pull/48163),[#48396](https://github.com/PaddlePaddle/Paddle/pull/48396),[#48308](https://github.com/PaddlePaddle/Paddle/pull/48308),[#47110](https://github.com/PaddlePaddle/Paddle/pull/47110),[#53089](https://github.com/PaddlePaddle/Paddle/pull/53089) - - Optimize communication library TCP linking time. [#49810](https://github.com/PaddlePaddle/Paddle/pull/49810),[#47184](https://github.com/PaddlePaddle/Paddle/pull/47184) - -#### Automatic parallel -- Improve semi-automatic parallel for static graphs: - - Add FLOPs computation function for multiple operators, and add computation Cost modelling based on FLOPs. [#48083](https://github.com/PaddlePaddle/Paddle/pull/48083),[#47978](https://github.com/PaddlePaddle/Paddle/pull/47978),[#47595](https://github.com/PaddlePaddle/Paddle/pull/47595),[#48083](https://github.com/PaddlePaddle/Paddle/pull/48083),[#48084](https://github.com/PaddlePaddle/Paddle/pull/48084),[#47816](https://github.com/PaddlePaddle/Paddle/pull/47816) - - Improve API ease-of-use. Perfect the DistAttr, Process Mesh, Engine API, information printing, input and output modules. Implement the Engine new cost API. It can be used to theoretically analyze model running time and video memory overhead. [#47503](https://github.com/PaddlePaddle/Paddle/pull/47503),[#46416](https://github.com/PaddlePaddle/Paddle/pull/46416),[#46554](https://github.com/PaddlePaddle/Paddle/pull/46554), [#46633](https://github.com/PaddlePaddle/Paddle/pull/46633),[#49214](https://github.com/PaddlePaddle/Paddle/pull/49214),[#53848](https://github.com/PaddlePaddle/Paddle/pull/53848),[#46552](https://github.com/PaddlePaddle/Paddle/pull/46552), [#47043](https://github.com/PaddlePaddle/Paddle/pull/47043), [#49665](https://github.com/PaddlePaddle/Paddle/pull/49665), [#52912](https://github.com/PaddlePaddle/Paddle/pull/52912), [#45776](https://github.com/PaddlePaddle/Paddle/pull/45776), [#47263](https://github.com/PaddlePaddle/Paddle/pull/47263) - - Optimize the generality and ease of use of Pass. Support more scenarios, and reduce time spent on Pass pre-analysis. [#46519](https://github.com/PaddlePaddle/Paddle/pull/46519),[#47358](https://github.com/PaddlePaddle/Paddle/pull/47358),[#46391](https://github.com/PaddlePaddle/Paddle/pull/46391), [#51035](https://github.com/PaddlePaddle/Paddle/pull/51035) - - Enhance debugging capabilities with distributed randomness control mechanisms and hybrid parallel precision alignment tools. [#52903](https://github.com/PaddlePaddle/Paddle/pull/52903),[#49865](https://github.com/PaddlePaddle/Paddle/pull/49865) - - Support automatic sharding of inference generation task networking. Adapt special usage of control flow and conditional block in the generation model. [#46771](https://github.com/PaddlePaddle/Paddle/pull/46771), [#54067](https://github.com/PaddlePaddle/Paddle/pull/54067) - - Improve grad_clip to support load balancing in data parallel scenarios. [#49510](https://github.com/PaddlePaddle/Paddle/pull/49510), [#49249](https://github.com/PaddlePaddle/Paddle/pull/49249) -- Semi-automatic parallel performance improvement for static graphs: - - Add the Sharding Pass automated communication Fuse and multi-streams communication functions, with throughput performance improved by 26% on two machines for GPT 6.7B model. [#48604](https://github.com/PaddlePaddle/Paddle/pull/48604), [#47180](https://github.com/PaddlePaddle/Paddle/pull/47180),[#46180](https://github.com/PaddlePaddle/Paddle/pull/46180) - - Add Recompute optimization strategy tuning function. Select optimal recompute checkpoint settings based on video memory and model size. [#48608](https://github.com/PaddlePaddle/Paddle/pull/48608),[#47846](https://github.com/PaddlePaddle/Paddle/pull/47846),[#49010](https://github.com/PaddlePaddle/Paddle/pull/49010) - - For the pipeline parallel, add 1F1B scheduling optimization Pass [#54260](https://github.com/PaddlePaddle/Paddle/pull/54260), [#45915](https://github.com/PaddlePaddle/Paddle/pull/45915) - - Optimize data parallel. Support optimizations such as converged communication and communication computation Overlap, with performance improved by 5% in GPT 1.3B model. [#48092](https://github.com/PaddlePaddle/Paddle/pull/48092),[#45643](https://github.com/PaddlePaddle/Paddle/pull/45643),[#49744](https://github.com/PaddlePaddle/Paddle/pull/49744), [#47578](https://github.com/PaddlePaddle/Paddle/pull/47578) - - Optimize Reshard module concate performance. Reduce number of concates in some scenarios. [#47809](https://github.com/PaddlePaddle/Paddle/pull/47809) - - Optimize mixing accuracy, upgrade Pass performance, support BF16 low accuracy, and adapt the auto mixing parallel of the while loop control flow. [#51285](https://github.com/PaddlePaddle/Paddle/pull/51285),[#51147](https://github.com/PaddlePaddle/Paddle/pull/51147), [#49219](https://github.com/PaddlePaddle/Paddle/pull/49219), [#49079](https://github.com/PaddlePaddle/Paddle/pull/49079) -- Improve function of fully automatic parallel for static graphs: - - Add new rule-based fully automated search strategy. [#51859](https://github.com/PaddlePaddle/Paddle/pull/51859),[#51908](https://github.com/PaddlePaddle/Paddle/pull/51908),[#52053](https://github.com/PaddlePaddle/Paddle/pull/52053),[#48316](https://github.com/PaddlePaddle/Paddle/pull/48316),[#48464](https://github.com/PaddlePaddle/Paddle/pull/48464), [#52041](https://github.com/PaddlePaddle/Paddle/pull/52041) - - Improve automatic parallel modelling capability, enriching single-node topology modelling and communication volume modelling. [#52723](https://github.com/PaddlePaddle/Paddle/pull/52723),[#46387](https://github.com/PaddlePaddle/Paddle/pull/46387),[#47043](https://github.com/PaddlePaddle/Paddle/pull/47043) - -#### Parameter server -- Clean up the all list in ps directory, in which API is not exposed [#51289](https://github.com/PaddlePaddle/Paddle/pull/51289) -- Clean up cvm operator [#48989](https://github.com/PaddlePaddle/Paddle/pull/48989) -- For GPUPS, add support for AFS. [#46611](https://github.com/PaddlePaddle/Paddle/pull/46611) -- Degrade PGLBOX2.0 log, fix stuck issue of dense parameter, fix the bug that barrier does not take effect, and add get_epoch_finish python side interface [#49946](https://github.com/PaddlePaddle/Paddle/pull/49946),[#50166](https://github.com/PaddlePaddle/Paddle/pull/50166),[#50349](https://github.com/PaddlePaddle/Paddle/pull/50349) -- GPUPs run to switch to specified mode. [#51115](https://github.com/PaddlePaddle/Paddle/pull/51115) -- GPUPS is added to benchmark. [#49587](https://github.com/PaddlePaddle/Paddle/pull/49587),[#49649](https://github.com/PaddlePaddle/Paddle/pull/49649) -- Fix the GPUPS optimizer selection bug, fix reader reading problem, and fix RPC compilation problem. [#47026](https://github.com/PaddlePaddle/Paddle/pull/47026),[#47192](https://github.com/PaddlePaddle/Paddle/pull/47192),[#49878](https://github.com/PaddlePaddle/Paddle/pull/49878), [#46356](https://github.com/PaddlePaddle/Paddle/pull/46356),[#46575](https://github.com/PaddlePaddle/Paddle/pull/46575),[#49389](https://github.com/PaddlePaddle/Paddle/pull/49389),[#46258](https://github.com/PaddlePaddle/Paddle/pull/46258),[#50136](https://github.com/PaddlePaddle/Paddle/pull/50136) -- Add rocksdb compilation method. [#46074](https://github.com/PaddlePaddle/Paddle/pull/46074) - -### CUDA -#### New features -- Add compilation support for CUDA 12.0. Fix related unit test. ([#49539](https://github.com/PaddlePaddle/Paddle/pull/49539), [#54542](https://github.com/PaddlePaddle/Paddle/pull/54542)) -- Add CUDNN Frontend API compilation support and related unit test. You can use `WITH_CUDNN_FRONTEND=ON ` compilation option for start. ([#47524](https://github.com/PaddlePaddle/Paddle/pull/47524), [#47612](https://github.com/PaddlePaddle/Paddle/pull/47612)) - -#### Improvements -- Add mixed precision strategy and optimize precision: - - Add and optimize FP16 and BF16 data type support for more than 200 operators in the framework, including logsumexp, reduce_max, cumprod, sync_batch_norm, compare class OP, etc. Carry out precision optimization and unit test for all FP16 and BF16 operators. Improve the unit test framework function for low-precision operators, to ensure there is no loss of accuracy in the process of large-model training. ([#51193](https://github.com/PaddlePaddle/Paddle/pull/51193), [#51114](https://github.com/PaddlePaddle/Paddle/pull/51114), [#45817](https://github.com/PaddlePaddle/Paddle/pull/45817), [#52862](https://github.com/PaddlePaddle/Paddle/pull/52862), [#52919](https://github.com/PaddlePaddle/Paddle/pull/52919), [#52921](https://github.com/PaddlePaddle/Paddle/pull/52921), [#46413](https://github.com/PaddlePaddle/Paddle/pull/46413), [#48205](https://github.com/PaddlePaddle/Paddle/pull/48205), [#54193](https://github.com/PaddlePaddle/Paddle/pull/54193), [#48041](https://github.com/PaddlePaddle/Paddle/pull/48041), [#48121](https://github.com/PaddlePaddle/Paddle/pull/48121), [#46364](https://github.com/PaddlePaddle/Paddle/pull/46364), [#51153](https://github.com/PaddlePaddle/Paddle/pull/51153), [#53023](https://github.com/PaddlePaddle/Paddle/pull/53023), [#53079](https://github.com/PaddlePaddle/Paddle/pull/53079), [#53137](https://github.com/PaddlePaddle/Paddle/pull/53137), [#46212](https://github.com/PaddlePaddle/Paddle/pull/46212), [#50908](https://github.com/PaddlePaddle/Paddle/pull/50908), [#52555](https://github.com/PaddlePaddle/Paddle/pull/52555), [#51582](https://github.com/PaddlePaddle/Paddle/pull/51582), [#47897](https://github.com/PaddlePaddle/Paddle/pull/47897), [#45601](https://github.com/PaddlePaddle/Paddle/pull/45601), [#53522](https://github.com/PaddlePaddle/Paddle/pull/53522), [#52666](https://github.com/PaddlePaddle/Paddle/pull/52666), [#50101](https://github.com/PaddlePaddle/Paddle/pull/50101), [#48315](https://github.com/PaddlePaddle/Paddle/pull/48315), [#50847](https://github.com/PaddlePaddle/Paddle/pull/50847), [#50905](https://github.com/PaddlePaddle/Paddle/pull/50905), [#50906](https://github.com/PaddlePaddle/Paddle/pull/50906), [#50909](https://github.com/PaddlePaddle/Paddle/pull/50909), [#50916](https://github.com/PaddlePaddle/Paddle/pull/50916), [#50917](https://github.com/PaddlePaddle/Paddle/pull/50917), [#50920](https://github.com/PaddlePaddle/Paddle/pull/50920), [#50919](https://github.com/PaddlePaddle/Paddle/pull/50919), [#50904](https://github.com/PaddlePaddle/Paddle/pull/50904), [#50918](https://github.com/PaddlePaddle/Paddle/pull/50918), [#50938](https://github.com/PaddlePaddle/Paddle/pull/50938), [#50858](https://github.com/PaddlePaddle/Paddle/pull/50858), [#50933](https://github.com/PaddlePaddle/Paddle/pull/50933), [#50945](https://github.com/PaddlePaddle/Paddle/pull/50945), [#50936](https://github.com/PaddlePaddle/Paddle/pull/50936), [#51168](https://github.com/PaddlePaddle/Paddle/pull/51168), [#51493](https://github.com/PaddlePaddle/Paddle/pull/51493), [#50924](https://github.com/PaddlePaddle/Paddle/pull/50924), [#50923](https://github.com/PaddlePaddle/Paddle/pull/50923), [#50926](https://github.com/PaddlePaddle/Paddle/pull/50926), [#50925](https://github.com/PaddlePaddle/Paddle/pull/50925), [#50930](https://github.com/PaddlePaddle/Paddle/pull/50930), [#53284](https://github.com/PaddlePaddle/Paddle/pull/53284), [#53286](https://github.com/PaddlePaddle/Paddle/pull/53286), [#53285](https://github.com/PaddlePaddle/Paddle/pull/53285), [#50976](https://github.com/PaddlePaddle/Paddle/pull/50976), [#50915](https://github.com/PaddlePaddle/Paddle/pull/50915), [#50915](https://github.com/PaddlePaddle/Paddle/pull/50915), [#48192](https://github.com/PaddlePaddle/Paddle/pull/48192), [#50993](https://github.com/PaddlePaddle/Paddle/pull/50993)， [#50998](https://github.com/PaddlePaddle/Paddle/pull/50998), [#51380](https://github.com/PaddlePaddle/Paddle/pull/51380), [#51137](https://github.com/PaddlePaddle/Paddle/pull/51137), [#51106](https://github.com/PaddlePaddle/Paddle/pull/51106), [#51197](https://github.com/PaddlePaddle/Paddle/pull/51197), [#51159](https://github.com/PaddlePaddle/Paddle/pull/51159), [#51552](https://github.com/PaddlePaddle/Paddle/pull/51552), [#51151](https://github.com/PaddlePaddle/Paddle/pull/51151), [#51005](https://github.com/PaddlePaddle/Paddle/pull/51005), [#51565](https://github.com/PaddlePaddle/Paddle/pull/51565), [#51036](https://github.com/PaddlePaddle/Paddle/pull/51036), [#51185](https://github.com/PaddlePaddle/Paddle/pull/51185), [#51791](https://github.com/PaddlePaddle/Paddle/pull/51791), [#51083](https://github.com/PaddlePaddle/Paddle/pull/51083), [#51694](https://github.com/PaddlePaddle/Paddle/pull/51694), [#51689](https://github.com/PaddlePaddle/Paddle/pull/51689), [#51009](https://github.com/PaddlePaddle/Paddle/pull/51009), [#51051](https://github.com/PaddlePaddle/Paddle/pull/51051), [#51532](https://github.com/PaddlePaddle/Paddle/pull/51532), [#51978](https://github.com/PaddlePaddle/Paddle/pull/51978), [#51903](https://github.com/PaddlePaddle/Paddle/pull/51903), [#51888](https://github.com/PaddlePaddle/Paddle/pull/51888), [#52016](https://github.com/PaddlePaddle/Paddle/pull/52016), [#52035](https://github.com/PaddlePaddle/Paddle/pull/52035), [#52184](https://github.com/PaddlePaddle/Paddle/pull/52184), [#52018](https://github.com/PaddlePaddle/Paddle/pull/52018), [#51787](https://github.com/PaddlePaddle/Paddle/pull/51787), [#51640](https://github.com/PaddlePaddle/Paddle/pull/51640), [#52172](https://github.com/PaddlePaddle/Paddle/pull/52172), [#52193](https://github.com/PaddlePaddle/Paddle/pull/52193), [#51160](https://github.com/PaddlePaddle/Paddle/pull/51160), [#51809](https://github.com/PaddlePaddle/Paddle/pull/51809), [#51678](https://github.com/PaddlePaddle/Paddle/pull/51678), [#52158](https://github.com/PaddlePaddle/Paddle/pull/52158), [#51015](https://github.com/PaddlePaddle/Paddle/pull/51015), [#52240](https://github.com/PaddlePaddle/Paddle/pull/52240), [#52276](https://github.com/PaddlePaddle/Paddle/pull/52276), [#52233](https://github.com/PaddlePaddle/Paddle/pull/52233), [#52220](https://github.com/PaddlePaddle/Paddle/pull/52220), [#52107](https://github.com/PaddlePaddle/Paddle/pull/52107), [#52282](https://github.com/PaddlePaddle/Paddle/pull/52282), [#52311](https://github.com/PaddlePaddle/Paddle/pull/52311), [#52315](https://github.com/PaddlePaddle/Paddle/pull/52315), [#52357](https://github.com/PaddlePaddle/Paddle/pull/52357), [#52256](https://github.com/PaddlePaddle/Paddle/pull/52256), [#51649](https://github.com/PaddlePaddle/Paddle/pull/51649), [#52413](https://github.com/PaddlePaddle/Paddle/pull/52413), [#52369](https://github.com/PaddlePaddle/Paddle/pull/52369), [#51837](https://github.com/PaddlePaddle/Paddle/pull/51837), [#52112](https://github.com/PaddlePaddle/Paddle/pull/52112), [#51819](https://github.com/PaddlePaddle/Paddle/pull/51819), [#52388](https://github.com/PaddlePaddle/Paddle/pull/52388), [#52411](https://github.com/PaddlePaddle/Paddle/pull/52411), [#52521](https://github.com/PaddlePaddle/Paddle/pull/52521), [#51300](https://github.com/PaddlePaddle/Paddle/pull/51300), [#51117](https://github.com/PaddlePaddle/Paddle/pull/51117), [#52380](https://github.com/PaddlePaddle/Paddle/pull/52380), [#52317](https://github.com/PaddlePaddle/Paddle/pull/52317), [#51263](https://github.com/PaddlePaddle/Paddle/pull/51263), [#52668](https://github.com/PaddlePaddle/Paddle/pull/52668), [#52259](https://github.com/PaddlePaddle/Paddle/pull/52259), [#50999](https://github.com/PaddlePaddle/Paddle/pull/50999), [#52407](https://github.com/PaddlePaddle/Paddle/pull/52407), [#52288](https://github.com/PaddlePaddle/Paddle/pull/52288), [#52845](https://github.com/PaddlePaddle/Paddle/pull/52845), [#50953](https://github.com/PaddlePaddle/Paddle/pull/50953), [#52667](https://github.com/PaddlePaddle/Paddle/pull/52667), [#52582](https://github.com/PaddlePaddle/Paddle/pull/52582), [#52426](https://github.com/PaddlePaddle/Paddle/pull/52426), [#51884](https://github.com/PaddlePaddle/Paddle/pull/51884), [#52630](https://github.com/PaddlePaddle/Paddle/pull/52630), [#52136](https://github.com/PaddlePaddle/Paddle/pull/52136), [#52604](https://github.com/PaddlePaddle/Paddle/pull/52604), [#51615](https://github.com/PaddlePaddle/Paddle/pull/51615), [#51275](https://github.com/PaddlePaddle/Paddle/pull/51275), [#52898](https://github.com/PaddlePaddle/Paddle/pull/52898), [#52918](https://github.com/PaddlePaddle/Paddle/pull/52918), [#52572](https://github.com/PaddlePaddle/Paddle/pull/52572), [#52683](https://github.com/PaddlePaddle/Paddle/pull/52683), [#52956](https://github.com/PaddlePaddle/Paddle/pull/52956), [#52963](https://github.com/PaddlePaddle/Paddle/pull/52963), [#52954](https://github.com/PaddlePaddle/Paddle/pull/52954), [#52444](https://github.com/PaddlePaddle/Paddle/pull/52444), [#52314](https://github.com/PaddlePaddle/Paddle/pull/52314), [#52887](https://github.com/PaddlePaddle/Paddle/pull/52887), [#52195](https://github.com/PaddlePaddle/Paddle/pull/52195), [#53100](https://github.com/PaddlePaddle/Paddle/pull/53100), [#52961](https://github.com/PaddlePaddle/Paddle/pull/52961), [#52953](https://github.com/PaddlePaddle/Paddle/pull/52953), [#53111](https://github.com/PaddlePaddle/Paddle/pull/53111), [#53549](https://github.com/PaddlePaddle/Paddle/pull/53549), [#53736](https://github.com/PaddlePaddle/Paddle/pull/53736), [#52920](https://github.com/PaddlePaddle/Paddle/pull/52920), [#53195](https://github.com/PaddlePaddle/Paddle/pull/53195), [#53535](https://github.com/PaddlePaddle/Paddle/pull/53535), [#53876](https://github.com/PaddlePaddle/Paddle/pull/53876), [#53785](https://github.com/PaddlePaddle/Paddle/pull/53785), [#53722](https://github.com/PaddlePaddle/Paddle/pull/53722), [#54285](https://github.com/PaddlePaddle/Paddle/pull/54285), [#54232](https://github.com/PaddlePaddle/Paddle/pull/54232), [#53922](https://github.com/PaddlePaddle/Paddle/pull/53922), [#47277](https://github.com/PaddlePaddle/Paddle/pull/47277), [#50811](https://github.com/PaddlePaddle/Paddle/pull/50811), [#54571](https://github.com/PaddlePaddle/Paddle/pull/54571), [#50129](https://github.com/PaddlePaddle/Paddle/pull/50129), [#50340](https://github.com/PaddlePaddle/Paddle/pull/50340), [#50848](https://github.com/PaddlePaddle/Paddle/pull/50848), [#50849](https://github.com/PaddlePaddle/Paddle/pull/50849), [#50868](https://github.com/PaddlePaddle/Paddle/pull/50868), [#50878](https://github.com/PaddlePaddle/Paddle/pull/50878), [#50929](https://github.com/PaddlePaddle/Paddle/pull/50929), [#50939](https://github.com/PaddlePaddle/Paddle/pull/50939), [#50973](https://github.com/PaddlePaddle/Paddle/pull/50973), [#50913](https://github.com/PaddlePaddle/Paddle/pull/50913), [#51145](https://github.com/PaddlePaddle/Paddle/pull/51145), [#51090](https://github.com/PaddlePaddle/Paddle/pull/51090), [#51098](https://github.com/PaddlePaddle/Paddle/pull/51098), [#51094](https://github.com/PaddlePaddle/Paddle/pull/51094), [#51216](https://github.com/PaddlePaddle/Paddle/pull/51216), [#51736](https://github.com/PaddlePaddle/Paddle/pull/51736), [#51684](https://github.com/PaddlePaddle/Paddle/pull/51684), [#51925](https://github.com/PaddlePaddle/Paddle/pull/51925), [#54030](https://github.com/PaddlePaddle/Paddle/pull/54030), [#50700](https://github.com/PaddlePaddle/Paddle/pull/50700), [#52264](https://github.com/PaddlePaddle/Paddle/pull/52264), [#51069](https://github.com/PaddlePaddle/Paddle/pull/51069), [#51101](https://github.com/PaddlePaddle/Paddle/pull/51101), [#51286](https://github.com/PaddlePaddle/Paddle/pull/51286), [#53582](https://github.com/PaddlePaddle/Paddle/pull/53582),[#49869](https://github.com/PaddlePaddle/Paddle/pull/49869))) -- AMP optimization: Comprehensively upgrade and optimize ease of use, accuracy stability and debuggability of AMP training, to better support acceleration of large model training. In terms of ease of use, unify the API for dynamic and static graphs. Add new conversion interfaces such as model.float(), model.float16() and model.bfloat16(). In terms of accuracy stability, enhance automatic adjustment of the strategy for BF16 type. Optimize blacklist settings. Enhance support of the multi_precision function by optimizer operators Adagrad, Adamax, Adadelta, and RMSProp. In the O2 mode, improve master grad mechanism, add type promotion mechanism and a new parameter for the specific module to use float32 computation to guarantee accuracy. In terms of debuggability, add the paddle.amp.debugging module to provide operator statistics, outlier detection, and accuracy comparison. ( [#50132](https://github.com/PaddlePaddle/Paddle/pull/50132), [#50078](https://github.com/PaddlePaddle/Paddle/pull/50078), [#50131](https://github.com/PaddlePaddle/Paddle/pull/50131), [#49705](https://github.com/PaddlePaddle/Paddle/pull/49705), [#52936](https://github.com/PaddlePaddle/Paddle/pull/52936), [#52871](https://github.com/PaddlePaddle/Paddle/pull/52871), [#53289](https://github.com/PaddlePaddle/Paddle/pull/53289), [#53362](https://github.com/PaddlePaddle/Paddle/pull/53362), [#54240](https://github.com/PaddlePaddle/Paddle/pull/54240), [#53768](https://github.com/PaddlePaddle/Paddle/pull/53768), [#48041](https://github.com/PaddlePaddle/Paddle/pull/48041), [#47672](https://github.com/PaddlePaddle/Paddle/pull/47672), [#48843](https://github.com/PaddlePaddle/Paddle/pull/48843), [#49391](https://github.com/PaddlePaddle/Paddle/pull/49391), [#51635](https://github.com/PaddlePaddle/Paddle/pull/51635), [#45541](https://github.com/PaddlePaddle/Paddle/pull/45541), [#53742](https://github.com/PaddlePaddle/Paddle/pull/53742), [#51020](https://github.com/PaddlePaddle/Paddle/pull/51020), [#51063](https://github.com/PaddlePaddle/Paddle/pull/51063), [#52514](https://github.com/PaddlePaddle/Paddle/pull/52514), [#50940](https://github.com/PaddlePaddle/Paddle/pull/50940), [#52936](https://github.com/PaddlePaddle/Paddle/pull/52936), [#53439](https://github.com/PaddlePaddle/Paddle/pull/53439), [#53712](https://github.com/PaddlePaddle/Paddle/pull/53712), [#48238](https://github.com/PaddlePaddle/Paddle/pull/48238), [#52215](https://github.com/PaddlePaddle/Paddle/pull/52215), [#53012](https://github.com/PaddlePaddle/Paddle/pull/53012), [#52918](https://github.com/PaddlePaddle/Paddle/pull/52918), [#54571](https://github.com/PaddlePaddle/Paddle/pull/54571)) -- For GroupNorm operator, add support for NHWC data format. ([#47533](https://github.com/PaddlePaddle/Paddle/pull/47533)) -- For index_put operator, add support for mixed data types of bool and int. ([#54195](https://github.com/PaddlePaddle/Paddle/pull/54195)) -- Add sparse.is_nan API for determining whether a sparse tensor contains a NaN element. ([#51513](https://github.com/PaddlePaddle/Paddle/pull/51513)) - -#### bug fix -- Fix bugs of computation errors of several operators such as trace, roll, dropout_nd, and log_softmax, stack overflow, and some unit test error. ([#50243](https://github.com/PaddlePaddle/Paddle/pull/50243), [#52012](https://github.com/PaddlePaddle/Paddle/pull/52012), [#53795](https://github.com/PaddlePaddle/Paddle/pull/53795), [#53149](https://github.com/PaddlePaddle/Paddle/pull/53149), [#53654](https://github.com/PaddlePaddle/Paddle/pull/53654), [#51054](https://github.com/PaddlePaddle/Paddle/pull/51054), [#49373](https://github.com/PaddlePaddle/Paddle/pull/49373), [#53038](https://github.com/PaddlePaddle/Paddle/pull/53038)) -- Fix the problem that conv operator exhaustive search does not work in some scenarios. ([#47065](https://github.com/PaddlePaddle/Paddle/pull/47065)) -- Fix timeout problem of collective_reduce_scatter and other operators on A100. ([#54513](https://github.com/PaddlePaddle/Paddle/pull/54513)) -- Fix the problem of attribute error in FusedLinear unit test. ([#50359](https://github.com/PaddlePaddle/Paddle/pull/50359)) -- Fix the OOM problem that may occur when using Profiler. ([#46089](https://github.com/PaddlePaddle/Paddle/pull/46089)) - -#### Performance optimization -- Further optimize GPU Kernel and eigen implementations of the framework's large number of operators, including max_pool3d, dropout, adaptive_pooling, depthwise_conv2d, transpose, eigh, broadcast class computations, reduce class computations, prelu, logsumexp, and sparse, to achieve better performance in more configuration scenarios. ([#45820](https://github.com/PaddlePaddle/Paddle/pull/45820), [#45959](https://github.com/PaddlePaddle/Paddle/pull/45959), [#45934](https://github.com/PaddlePaddle/Paddle/pull/45934), [#46332](https://github.com/PaddlePaddle/Paddle/pull/46332), [#46287](https://github.com/PaddlePaddle/Paddle/pull/46287), [#47233](https://github.com/PaddlePaddle/Paddle/pull/47233), [#48855](https://github.com/PaddlePaddle/Paddle/pull/48855), [#48560](https://github.com/PaddlePaddle/Paddle/pull/48560), [#49419](https://github.com/PaddlePaddle/Paddle/pull/49419), [#49748](https://github.com/PaddlePaddle/Paddle/pull/49748), [#50348](https://github.com/PaddlePaddle/Paddle/pull/50348), [#52401](https://github.com/PaddlePaddle/Paddle/pull/52401), [#51131](https://github.com/PaddlePaddle/Paddle/pull/51131), [#51141](https://github.com/PaddlePaddle/Paddle/pull/51141), [#51479](https://github.com/PaddlePaddle/Paddle/pull/51479), [#51835](https://github.com/PaddlePaddle/Paddle/pull/51835), [#52509](https://github.com/PaddlePaddle/Paddle/pull/52509), [#52482](https://github.com/PaddlePaddle/Paddle/pull/52482), [#52700](https://github.com/PaddlePaddle/Paddle/pull/52700), [#53112](https://github.com/PaddlePaddle/Paddle/pull/53112), [#53659](https://github.com/PaddlePaddle/Paddle/pull/53659), [#53658](https://github.com/PaddlePaddle/Paddle/pull/53658), [#53154](https://github.com/PaddlePaddle/Paddle/pull/53154), [#54071](https://github.com/PaddlePaddle/Paddle/pull/54071), [#53622](https://github.com/PaddlePaddle/Paddle/pull/53622), [#52952](https://github.com/PaddlePaddle/Paddle/pull/52952), [#46046](https://github.com/PaddlePaddle/Paddle/pull/46046), [#46119](https://github.com/PaddlePaddle/Paddle/pull/46119), [#45946](https://github.com/PaddlePaddle/Paddle/pull/45946), [#47212](https://github.com/PaddlePaddle/Paddle/pull/47212), [#47791](https://github.com/PaddlePaddle/Paddle/pull/47791), [#47454](https://github.com/PaddlePaddle/Paddle/pull/47454), [#45230](https://github.com/PaddlePaddle/Paddle/pull/45230), [#48899](https://github.com/PaddlePaddle/Paddle/pull/48899), [#33051](https://github.com/PaddlePaddle/Paddle/pull/33051), [#49040](https://github.com/PaddlePaddle/Paddle/pull/49040), [#48992](https://github.com/PaddlePaddle/Paddle/pull/48992), [#49086](https://github.com/PaddlePaddle/Paddle/pull/49086), [#50808](https://github.com/PaddlePaddle/Paddle/pull/50808), [#46431](https://github.com/PaddlePaddle/Paddle/pull/46431), [#50931](https://github.com/PaddlePaddle/Paddle/pull/50931), [#48056](https://github.com/PaddlePaddle/Paddle/pull/48056), [#46071](https://github.com/PaddlePaddle/Paddle/pull/46071), [#49231](https://github.com/PaddlePaddle/Paddle/pull/49231), [#38660](https://github.com/PaddlePaddle/Paddle/pull/38660), [#50287](https://github.com/PaddlePaddle/Paddle/pull/50287), [#46111](https://github.com/PaddlePaddle/Paddle/pull/46111), [#46997](https://github.com/PaddlePaddle/Paddle/pull/46997), [#45854](https://github.com/PaddlePaddle/Paddle/pull/45854), [#47738](https://github.com/PaddlePaddle/Paddle/pull/47738), [#48635](https://github.com/PaddlePaddle/Paddle/pull/48635), [#50353](https://github.com/PaddlePaddle/Paddle/pull/50353), [#50362](https://github.com/PaddlePaddle/Paddle/pull/50362), [#51934](https://github.com/PaddlePaddle/Paddle/pull/51934), [#54045](https://github.com/PaddlePaddle/Paddle/pull/54045), [#46679](https://github.com/PaddlePaddle/Paddle/pull/46679), [#52093](https://github.com/PaddlePaddle/Paddle/pull/52093), [#52969](https://github.com/PaddlePaddle/Paddle/pull/52969)) -- Provide more fusion implementations and related fusion pass, such as fused_feed_forward, gather-gemm-scatter, matmul + bias, layernorm_shift_partition + element_add, and elementwise class fusion, to further improve performance of models that use the mode. ( [#50423](https://github.com/PaddlePaddle/Paddle/pull/50423), [#50091](https://github.com/PaddlePaddle/Paddle/pull/50091), [#50364](https://github.com/PaddlePaddle/Paddle/pull/50364), [#53017](https://github.com/PaddlePaddle/Paddle/pull/53017), [#50755](https://github.com/PaddlePaddle/Paddle/pull/50755), [#50050](https://github.com/PaddlePaddle/Paddle/pull/50050), [#47099](https://github.com/PaddlePaddle/Paddle/pull/47099), [#48848](https://github.com/PaddlePaddle/Paddle/pull/48848), [#49383](https://github.com/PaddlePaddle/Paddle/pull/49383), [#50809](https://github.com/PaddlePaddle/Paddle/pull/50809), [#52361](https://github.com/PaddlePaddle/Paddle/pull/52361), [#52028](https://github.com/PaddlePaddle/Paddle/pull/52028), [#48439](https://github.com/PaddlePaddle/Paddle/pull/48439), [#49009](https://github.com/PaddlePaddle/Paddle/pull/49009), [#51427](https://github.com/PaddlePaddle/Paddle/pull/51427), [#52731](https://github.com/PaddlePaddle/Paddle/pull/52731), [#51805](https://github.com/PaddlePaddle/Paddle/pull/51805)) - -### Intermediate Representation -In order to guarantee stability and reduce R&D cost of the IR system, we have developed a new IR system for PaddlePaddle. Complete basic data structure definition, operator definition generation, and execution system adaptation. In order to better support higher-order requirements of scientific computing scenarios, complete higher-order adaptation of operators such as silu and cast. -- Complete the definition of IR data structure, including type system and operator definition. Implement execution adaptation with phi kernel. [#51112](https://github.com/PaddlePaddle/Paddle/pull/51112)， [#51992](https://github.com/PaddlePaddle/Paddle/pull/51992), [#50412](https://github.com/PaddlePaddle/Paddle/pull/50412), [#53557](https://github.com/PaddlePaddle/Paddle/pull/53557), [#53953](https://github.com/PaddlePaddle/Paddle/pull/53953), [#50959](https://github.com/PaddlePaddle/Paddle/pull/50959), [#54250](https://github.com/PaddlePaddle/Paddle/pull/54250), [#54197](https://github.com/PaddlePaddle/Paddle/pull/54197), [#54289](https://github.com/PaddlePaddle/Paddle/pull/54289), [#51636](https://github.com/PaddlePaddle/Paddle/pull/51636), [#52846](https://github.com/PaddlePaddle/Paddle/pull/52846), [#53988](https://github.com/PaddlePaddle/Paddle/pull/53988), [#54143](https://github.com/PaddlePaddle/Paddle/pull/54143), [#54035](https://github.com/PaddlePaddle/Paddle/pull/54035), [#54052](https://github.com/PaddlePaddle/Paddle/pull/54052), [#54340](https://github.com/PaddlePaddle/Paddle/pull/54340), [#54356](https://github.com/PaddlePaddle/Paddle/pull/54356), [#54068](https://github.com/PaddlePaddle/Paddle/pull/54068), [#53894](https://github.com/PaddlePaddle/Paddle/pull/53894), [#53707](https://github.com/PaddlePaddle/Paddle/pull/53707), [#54185](https://github.com/PaddlePaddle/Paddle/pull/54185), [#54031](https://github.com/PaddlePaddle/Paddle/pull/54031), [#54220](https://github.com/PaddlePaddle/Paddle/pull/54220), [#54275](https://github.com/PaddlePaddle/Paddle/pull/54275), [#54281](https://github.com/PaddlePaddle/Paddle/pull/54281), [#54186](https://github.com/PaddlePaddle/Paddle/pull/54186), [#54259](https://github.com/PaddlePaddle/Paddle/pull/54259), [#54124](https://github.com/PaddlePaddle/Paddle/pull/54124), [#54292](https://github.com/PaddlePaddle/Paddle/pull/54292), [#48068](https://github.com/PaddlePaddle/Paddle/pull/48068), [#53978](https://github.com/PaddlePaddle/Paddle/pull/53978) -- Improve the basic pass setup, including basic pass definition, pass registration management. [#54023](https://github.com/PaddlePaddle/Paddle/pull/54023),[#54170](https://github.com/PaddlePaddle/Paddle/pull/54170), [#54170](https://github.com/PaddlePaddle/Paddle/pull/54170), [#54308](https://github.com/PaddlePaddle/Paddle/pull/54308), [#54348](https://github.com/PaddlePaddle/Paddle/pull/54348), [#54385](https://github.com/PaddlePaddle/Paddle/pull/54385) -- Improve adaptation of high-level arithmetic, including modification of the basic module and adaptation of silu and cast arithmetic. [#52005](https://github.com/PaddlePaddle/Paddle/pull/52005), [#53425](https://github.com/PaddlePaddle/Paddle/pull/53425), [#53417](https://github.com/PaddlePaddle/Paddle/pull/53417), [#53417](https://github.com/PaddlePaddle/Paddle/pull/53417), [#53498](https://github.com/PaddlePaddle/Paddle/pull/53498), [#53171](https://github.com/PaddlePaddle/Paddle/pull/53171), [#53632](https://github.com/PaddlePaddle/Paddle/pull/53632), [#53605](https://github.com/PaddlePaddle/Paddle/pull/53605), [#53746](https://github.com/PaddlePaddle/Paddle/pull/53746), [#53874](https://github.com/PaddlePaddle/Paddle/pull/53874), [#54164](https://github.com/PaddlePaddle/Paddle/pull/54164), [#45888](https://github.com/PaddlePaddle/Paddle/pull/45888), [#46024](https://github.com/PaddlePaddle/Paddle/pull/46024), [#46446](https://github.com/PaddlePaddle/Paddle/pull/46446), [#46960](https://github.com/PaddlePaddle/Paddle/pull/46960) - -### CINN compiler -#### New features -- Add CINN support for 0D-Tensor. At present, in order to cooperate with the upgrade of the main framework, it is supported by adding pass temporarily. We will replace and upgrade the solution later. ([#53382](https://github.com/PaddlePaddle/Paddle/pull/53382), [#53955](https://github.com/PaddlePaddle/Paddle/pull/53955), [#54064](https://github.com/PaddlePaddle/Paddle/pull/54064), [#54118](https://github.com/PaddlePaddle/Paddle/pull/54118), [#54216](https://github.com/PaddlePaddle/Paddle/pull/54216), [#53454](https://github.com/PaddlePaddle/Paddle/pull/53454)) -- Add CINN support for int8/uint8/int16/uint16/bf16 data types. ([#50566](https://github.com/PaddlePaddle/Paddle/pull/50566), [#53637](https://github.com/PaddlePaddle/Paddle/pull/53637)) -- Add support for the CINN expand operator. ([#46776](https://github.com/PaddlePaddle/Paddle/pull/46776)) -- Add CINN support for PaddleInference. ([#45009](https://github.com/PaddlePaddle/Paddle/pull/45009)) - -#### Improvements -- For CINN compiler, pass skip_gc_vars attribute to CINN subgraph. CINN adds fetch operator for skip_gc_vars. [#49471](https://github.com/PaddlePaddle/Paddle/pull/49471), [#49553](https://github.com/PaddlePaddle/Paddle/pull/49553) -- For CINN compiler, conv2d and conv2d_grad do not use cinn operator by default. [#51645](https://github.com/PaddlePaddle/Paddle/pull/51645) -- Add build_cinn_pass to BuildStrategy for use in dynamic-to-static ([#49496](https://github.com/PaddlePaddle/Paddle/pull/49496)) -- Add reshape operator to perform unit test under combinator mechanism. ([#51276](https://github.com/PaddlePaddle/Paddle/pull/51276)) -- Change version of the main framework binding CINN from fixed commit to develop. ([#49775](https://github.com/PaddlePaddle/Paddle/pull/49775)) -- Set default Target parameter for CINN. ([#50182](https://github.com/PaddlePaddle/Paddle/pull/50182)) - -#### bug fix -- Fix the problem of inconsistent operator order after topology sorting during CINN symbolization. ([#52556](https://github.com/PaddlePaddle/Paddle/pull/52556)) -- Fix some operator computation errors, accuracy degradation, and unit test related problems. ([#53859](https://github.com/PaddlePaddle/Paddle/pull/53859), [#54261](https://github.com/PaddlePaddle/Paddle/pull/54261), [#46801](https://github.com/PaddlePaddle/Paddle/pull/46801), [#53676](https://github.com/PaddlePaddle/Paddle/pull/53676), [#53772](https://github.com/PaddlePaddle/Paddle/pull/53772)) -- Fix the problem of CINN support for float16 type. ([#48249](https://github.com/PaddlePaddle/Paddle/pull/48249)) -- Fix the problem in build_cinn_pass. ([#46843](https://github.com/PaddlePaddle/Paddle/pull/46843)) -- Fix the problem of no data area due to incorrect GC when CINN is turned on during combinator + dynamic-to-static. ([#50116](https://github.com/PaddlePaddle/Paddle/pull/50116)) -- Fix the problems of compiler dropout amp error, combinator resnet error, and inplace variable not found [#51688](https://github.com/PaddlePaddle/Paddle/pull/51688), [#52813](https://github.com/PaddlePaddle/Paddle/pull/52813), [#51769](https://github.com/PaddlePaddle/Paddle/pull/51769) - -#### Performance optimization -- Optimize reshape related fusion strategy ([#53066](https://github.com/PaddlePaddle/Paddle/pull/53066)) -- Optimize performance of BuildCINNPass. ([#49696](https://github.com/PaddlePaddle/Paddle/pull/49696)) -- Optimize performance of subgraph detection module. ([#45040](https://github.com/PaddlePaddle/Paddle/pull/45040), [#46937](https://github.com/PaddlePaddle/Paddle/pull/46937)) - -### Hardware support -#### CustomDevice -- Add support for the distributed strategy MP/Sharding/PP/MoE and recompute on the training side. Add support for the distributed strategy MP on the inference side. Support for hardware Ascend NPU and Cambricon MLU accessed through CustomDevice, without changing any codes, to automatically inherit all new distributed strategies added by CustomDevice. [#52872](https://github.com/PaddlePaddle/Paddle/pull/52872), [#54384](https://github.com/PaddlePaddle/Paddle/pull/54384), [#53220](https://github.com/PaddlePaddle/Paddle/pull/53220), [#54572](https://github.com/PaddlePaddle/Paddle/pull/54572), [#54573](https://github.com/PaddlePaddle/Paddle/pull/54573), [#54676](https://github.com/PaddlePaddle/Paddle/pull/54676), [#53044](https://github.com/PaddlePaddle/Paddle/pull/53044), [#53719](https://github.com/PaddlePaddle/Paddle/pull/53719), [#53701](https://github.com/PaddlePaddle/Paddle/pull/53701), [#53702](https://github.com/PaddlePaddle/Paddle/pull/53702), [#53703](https://github.com/PaddlePaddle/Paddle/pull/53703) -- Add API paddle.device.is_compiled_with_custom_device. It is convenient for users to judge whether the current environment supports the plug-in device backend of a certain hardware. [#49271](https://github.com/PaddlePaddle/Paddle/pull/49721) -- Add environment variable CUSTOM_DEVICE_BLACK_LIST setting, to support automatic heterogeneous operation on CPU of blacklisted operators. [#50409](https://github.com/PaddlePaddle/Paddle/pull/50409), [#50666](https://github.com/PaddlePaddle/Paddle/pull/50666) -- Optimize CustomDevice performance by reducing number of calls to get_device_count interface in runtime. [#46963](https://github.com/PaddlePaddle/Paddle/pull/46963) - -#### KUNLUNXIN XPU -- For the training side, use a new version of dynamic graph, with adding support for distributed strategy MP/Sharding/PP and recompute function, and communication library. For the inference side, add support for distributed strategy MP and support for XPU FasterTransformer operator acceleration library. [#49531](https://github.com/PaddlePaddle/Paddle/pull/49531), [#49815](https://github.com/PaddlePaddle/Paddle/pull/49815), [#48897](https://github.com/PaddlePaddle/Paddle/pull/48897), [#50717](https://github.com/PaddlePaddle/Paddle/pull/50717), [#51082](https://github.com/PaddlePaddle/Paddle/pull/51082), [#49757](https://github.com/PaddlePaddle/Paddle/pull/49757), [#51399](https://github.com/PaddlePaddle/Paddle/pull/51399), [#50329](https://github.com/PaddlePaddle/Paddle/pull/50329), [#48369](https://github.com/PaddlePaddle/Paddle/pull/48369), [#47838](https://github.com/PaddlePaddle/Paddle/pull/47838),[#48076](https://github.com/PaddlePaddle/Paddle/pull/48076),[#47882](https://github.com/PaddlePaddle/Paddle/pull/47882),[#48961](https://github.com/PaddlePaddle/Paddle/pull/48961),[#49043](https://github.com/PaddlePaddle/Paddle/pull/49043),[#49749](https://github.com/PaddlePaddle/Paddle/pull/49749),[#49806](https://github.com/PaddlePaddle/Paddle/pull/49806),[#53427](https://github.com/PaddlePaddle/Paddle/pull/53427),[#48470](https://github.com/PaddlePaddle/Paddle/pull/48470),[#49207](https://github.com/PaddlePaddle/Paddle/pull/49207),[#52296](https://github.com/PaddlePaddle/Paddle/pull/52296),[#51785](https://github.com/PaddlePaddle/Paddle/pull/51785),[#47168](https://github.com/PaddlePaddle/Paddle/pull/47168),[#47445](https://github.com/PaddlePaddle/Paddle/pull/47445),[#50200](https://github.com/PaddlePaddle/Paddle/pull/50200),[#49934](https://github.com/PaddlePaddle/Paddle/pull/49934),[#50792](https://github.com/PaddlePaddle/Paddle/pull/50792),[#52228](https://github.com/PaddlePaddle/Paddle/pull/52228),[#53337](https://github.com/PaddlePaddle/Paddle/pull/53337),[#53389](https://github.com/PaddlePaddle/Paddle/pull/53389),[#53496](https://github.com/PaddlePaddle/Paddle/pull/53496),[#53609](https://github.com/PaddlePaddle/Paddle/pull/53609),[#53697](https://github.com/PaddlePaddle/Paddle/pull/53697),[#53496](https://github.com/PaddlePaddle/Paddle/pull/53496),[#53720](https://github.com/PaddlePaddle/Paddle/pull/53720),[#53734](https://github.com/PaddlePaddle/Paddle/pull/53734),[#54172](https://github.com/PaddlePaddle/Paddle/pull/54172),[PR46227](https://github.com/PaddlePaddle/Paddle/pull/46227) - -## 4. Deployment Direction（Paddle Inference） -### New features -- Support Paddle TensorRT multiple subgraph TensorRT engine or TensorRT engine between different Predictors to share video memory in order to save video memory. [#45842](https://github.com/PaddlePaddle/Paddle/pull/45842) [#47631](https://github.com/PaddlePaddle/Paddle/pull/47631) -- For the C++ API, add Shape and data type API to obtain the input Tensor, and add Shape and data type API to obtain the output Tensor. For the C API, add SetExecStream, EnableMkldnnInt8 and other C++ existing APIs for serviced deployment. [#49758](https://github.com/PaddlePaddle/Paddle/pull/49758) -- Add paddle.inference.Predictor.register_output_hook() API. Support printing of the output of each layer under GPU inference in case of debugging. Support use in control flow models such as While. It should be noted the API does not support Paddle-TensorRT. [#54433](https://github.com/PaddlePaddle/Paddle/pull/54433) ，[#47050](https://github.com/PaddlePaddle/Paddle/pull/47050) ， [#54254](https://github.com/PaddlePaddle/Paddle/pull/54254) 。 -- Paddle Inference Predictor API supports paddle::Tensor as input and output, so users can directly reuse the PaddlePaddle dynamics graph for pre-inference and post-inference processing. ([#50445](https://github.com/PaddlePaddle/Paddle/pull/50445)) -- Enhance Paddle TensorRT dynamic shape running ability, config.enable_tuned_tensorrt_dynamic_shape() API to build TensorRT Engine at runtime without passing any parameters. It is unnecessary to collect shape information before running. To avoid rebuilding at runtime, it is necessary to overwrite minimum and maximum Shape in first operations for several times. [#52162](https://github.com/PaddlePaddle/Paddle/pull/52162) 。 -- Paddle-TensorRT supports model input in NHWC format. [#49633](https://github.com/PaddlePaddle/Paddle/pull/49633) 。 -- Extend config.Exp_DisableTensorRtOPs API to disable access to TensorRT by specifying the name of the Tensor variable. [#49497](https://github.com/PaddlePaddle/Paddle/pull/49497) 。 - -### Improvements -- Enhance GPU mixed-precision inference (non-Paddle TensorRT scenarios). For the Config.enable_use_gpu enhancement, you can set precision type. [#47993](https://github.com/PaddlePaddle/Paddle/pull/47993) -- Support double type input for inference. [#51786](https://github.com/PaddlePaddle/Paddle/pull/51786) 。 -- Since the TensorRT operator does not support the INT64 type, leading to running failure of INT64 data type in the model. Paddle-TensorRT has been enhanced to automatically convert, with reducing the model to run in the INT32 type when model contains INT64 data type. [#45547](https://github.com/PaddlePaddle/Paddle/pull/45547) -- Paddle-TensorRT supports more operators into TensorRT inference, including: - - expand_v2，gather_nd，rsqrt，sign，not，onehot，arg_min，temporal_shift，expend_as_v2，setvalue，index_select，round，acosh，square，reduce_max，not_equal，reduce_min，reduce_prod，grid_sampler，elementwise_mod，pad3d ，greater_equal，bitwise，cumsum，matmul_v2，reciprocal，where，bmm，take_along_axis，less_than，greater_than， logical_or， logical_xor， logical_and， less_equal，range，reduce_all，reduce_any ，fill_any_like ，pow - - [#47002](https://github.com/PaddlePaddle/Paddle/pull/47002) , [#47589](https://github.com/PaddlePaddle/Paddle/pull/47589) ，[#48223](https://github.com/PaddlePaddle/Paddle/pull/48223) ，[#48557](https://github.com/PaddlePaddle/Paddle/pull/48557) ， [#48655](https://github.com/PaddlePaddle/Paddle/pull/48655) ， [#49113](https://github.com/PaddlePaddle/Paddle/pull/49113) ， [#51207](https://github.com/PaddlePaddle/Paddle/pull/51207) ，[#51028](https://github.com/PaddlePaddle/Paddle/pull/51028) ，[#50341](https://github.com/PaddlePaddle/Paddle/pull/50341) ，[#51498](https://github.com/PaddlePaddle/Paddle/pull/51498) ，[#48534](https://github.com/PaddlePaddle/Paddle/pull/48534) ，[#48684](https://github.com/PaddlePaddle/Paddle/pull/48684) ， [#49393](https://github.com/PaddlePaddle/Paddle/pull/49393) ， [#49615](https://github.com/PaddlePaddle/Paddle/pull/49615) ，[#50934](https://github.com/PaddlePaddle/Paddle/pull/50934) ，[#50974](https://github.com/PaddlePaddle/Paddle/pull/50974)，[#50986](https://github.com/PaddlePaddle/Paddle/pull/50986) ， [#52000](https://github.com/PaddlePaddle/Paddle/pull/52000) ，[#51971](https://github.com/PaddlePaddle/Paddle/pull/51971) ， [#52518](https://github.com/PaddlePaddle/Paddle/pull/52518) ，[#44918](https://github.com/PaddlePaddle/Paddle/pull/44918) ，[#48230](https://github.com/PaddlePaddle/Paddle/pull/48230) ，[#47820](https://github.com/PaddlePaddle/Paddle/pull/47820) ， [#46877](https://github.com/PaddlePaddle/Paddle/pull/46877) ， [#48358](https://github.com/PaddlePaddle/Paddle/pull/48358) ， [#48592](https://github.com/PaddlePaddle/Paddle/pull/48592) ，[#48697](https://github.com/PaddlePaddle/Paddle/pull/48697) , [#53088](https://github.com/PaddlePaddle/Paddle/pull/53088) ， [#47974](https://github.com/PaddlePaddle/Paddle/pull/47974) ， [#53462](https://github.com/PaddlePaddle/Paddle/pull/53462) -- Enhance Paddle-TensorRT mapping operators strided_slice, instance_norm, prelu, argmax, cast, nearest_interp_v2, elementwise, bilinear. [#46819](https://github.com/PaddlePaddle/Paddle/pull/46819) ，[#47998](https://github.com/PaddlePaddle/Paddle/pull/47998) ，[#48043](https://github.com/PaddlePaddle/Paddle/pull/48043) ，[#48998](https://github.com/PaddlePaddle/Paddle/pull/48998) ， [#49675](https://github.com/PaddlePaddle/Paddle/pull/49675) , [#47495](https://github.com/PaddlePaddle/Paddle/pull/47495) -- Paddle-TensorRT partial operators (scale, square, sum, swish, expand_as_v2, prelu, gelu, hard_swish, hard_sigmoid, leaky_relu,softmax, stack, clip, cast, flatten_contiguous_range, unary, equal, elementwise_op). Support 0-dimensional Tensor. [#53660](https://github.com/PaddlePaddle/Paddle/pull/53660) ，[#53627](https://github.com/PaddlePaddle/Paddle/pull/53627) ， [#53634](https://github.com/PaddlePaddle/Paddle/pull/53634) ， [#53714](https://github.com/PaddlePaddle/Paddle/pull/53714) ， [#53729](https://github.com/PaddlePaddle/Paddle/pull/53729) ，[#53769](https://github.com/PaddlePaddle/Paddle/pull/53769) ，[#53506](https://github.com/PaddlePaddle/Paddle/pull/53506) ，[#53704](https://github.com/PaddlePaddle/Paddle/pull/53704) -- Support compilation for versions earlier than GCC12 + CUDA 12.0. [#50106](https://github.com/PaddlePaddle/Paddle/pull/50106) -- Paddle-TensorRT's DeformableConv plugin supports dynamic Shape input. [#50698](https://github.com/PaddlePaddle/Paddle/pull/50698) -- For Paddle-TensorRT, add plugin support for lookup_table operator. [#46613](https://github.com/PaddlePaddle/Paddle/pull/46613) -- Add config.enable_low_precision_io() API to support low-precision type input in Paddle-TensorRT scenario. [#52485](https://github.com/PaddlePaddle/Paddle/pull/52485) -- Paddle-TensorRT's LayerNorm plugin supports FP16 computation. [#45043](https://github.com/PaddlePaddle/Paddle/pull/45043) -- Predictor's input data paddle_infer::Tensor supports bool type. [#49388](https://github.com/PaddlePaddle/Paddle/pull/49388) -- Paddle-TensorRT enhanced Convolution implementation uses ConvolutionNd. [#47653](https://github.com/PaddlePaddle/Paddle/pull/47653) -- conv2d_fusion operator supports NHWC format. [#49047](https://github.com/PaddlePaddle/Paddle/pull/49047) -- Adjust the directory structure related to Phi operators under C++ inference library. [#53091](https://github.com/PaddlePaddle/Paddle/pull/53091) -- Support rebuilding TensorRT Engine instead of reporting errors when TensorRT serialization and loading versions do not match. [#50775](https://github.com/PaddlePaddle/Paddle/pull/50775) 。 -- Optimize Paddle-TensorRT runtime to print log messages. [#50181](https://github.com/PaddlePaddle/Paddle/pull/50181) -- Support elementwise 0-dimensional Tensor inputs for oneDNN-based CPU inference. [#51656](https://github.com/PaddlePaddle/Paddle/pull/51656) -- Clean up and normalize support for Paddle-TensorRT's FC, matmul, matmul_v2 operators, and unify and upgrade to use TensorRT's IMatrixMultiplyLayer for support. [#52222](https://github.com/PaddlePaddle/Paddle/pull/52222) - -### Performance optimization -- Support multiple lookup_tables into Paddle-TensorRT's Embedding+Eltwise+LayerNorm fusion. [#46243](https://github.com/PaddlePaddle/Paddle/pull/46243) ，[#46230](https://github.com/PaddlePaddle/Paddle/pull/46230) -- Add MoE fusion Phi operator to improve inference performance of MoE model. [#48703](https://github.com/PaddlePaddle/Paddle/pull/48703) -- In the scenario of INT8 quantized inference, Paddle-TensorRT plugin can fall back to FP16 computation, instead of FP32 computation. [#50554](https://github.com/PaddlePaddle/Paddle/pull/50554) -- Optimize memory and video memory in case of inference. [#49051](https://github.com/PaddlePaddle/Paddle/pull/49051) ， [#49046](https://github.com/PaddlePaddle/Paddle/pull/49046) ，[#53930](https://github.com/PaddlePaddle/Paddle/pull/53930) -- Optimize Layout and enhance Pass. [#52997](https://github.com/PaddlePaddle/Paddle/pull/52997) -- Support caching of operator Shape inferences to improve model inference performance. [#48312](https://github.com/PaddlePaddle/Paddle/pull/48312) -- Optimize bias+add+relu fusion using half2 instructions. [#49048](https://github.com/PaddlePaddle/Paddle/pull/49048) -- Optimize Concat Kernel for multiple inputs using vectorization operations. [#49540](https://github.com/PaddlePaddle/Paddle/pull/49540) -- Implement Convolution, Depthwise Convolution and related fusion operators based on CUTLASS to improve inference speed. [#47989](https://github.com/PaddlePaddle/Paddle/pull/47989) ，[#50603](https://github.com/PaddlePaddle/Paddle/pull/50603) ，[#51792](https://github.com/PaddlePaddle/Paddle/pull/51792) ，[#50603](https://github.com/PaddlePaddle/Paddle/pull/50603) -- Paddle-TensorRT supports FlashAttention’s plugin, to improve inference speed of models such as StableDiffusion. [#49438](https://github.com/PaddlePaddle/Paddle/pull/49438) 。 -- Add Transpose+LayerNorm fusion PASS, to improve inference speed of models such as StableDiffusion. [#50082](https://github.com/PaddlePaddle/Paddle/pull/50082) 。 -- Add Elementwise+Transpose fusion. [#50081](https://github.com/PaddlePaddle/Paddle/pull/50081) -- Optimize Paddle-TensorRT Group Norm plugin implementation. [#49160](https://github.com/PaddlePaddle/Paddle/pull/49160) -- For Config.EnableTensorRtEngine() API, add use_cuda_graph parameter. You can enable CUDA Graph. It should be noted you need to ensure the model input shape remains unchanged during usage, to reduce runtime consumption. [#53406](https://github.com/PaddlePaddle/Paddle/pull/53406) -- Support inplace operation of Reshape, to reduce copying time of the model at runtime. [#49146](https://github.com/PaddlePaddle/Paddle/pull/49146) -- Optimize LayerNorm kernel implementation based on oneDNN. [#47782](https://github.com/PaddlePaddle/Paddle/pull/47782) -- Support fusion of quantize+transpose and transpose+dequantize based on oneDNN. [#49509](https://github.com/PaddlePaddle/Paddle/pull/49509) -- When MKLDNN is turned on in CPU inference, FC-related fusion pass is enabled by default, to improve performance. [#45704](https://github.com/PaddlePaddle/Paddle/pull/45704) -- CPU OneDNN inference supports suqeeze2 + transpose2 fusion. [#47592](https://github.com/PaddlePaddle/Paddle/pull/47592) - -### XPU inference enhancement and performance optimization -- Add ExpRunWithRuntimeConfig API and XpuRuntimeConfig, to allow settings of parameters such as external streams, and L3 cache during inference. GetExecStream API supports obtaining Kunlun external stream objects. Input and output support Kunlun device memory, to reduce D2H and H2D overheads. [#53334](https://github.com/PaddlePaddle/Paddle/pull/53334)、 [#52466](https://github.com/PaddlePaddle/Paddle/pull/52466)、 [#53240](https://github.com/PaddlePaddle/Paddle/pull/53240) -- Add multi-encoder, fused_multi_transformer and fusion pass, to improve performance of ERNIE and Transformer class models. [#50570](https://github.com/PaddlePaddle/Paddle/pull/50570)、[#51346](https://github.com/PaddlePaddle/Paddle/pull/51346)、 [#50499](https://github.com/PaddlePaddle/Paddle/pull/50499)、[#53982](https://github.com/PaddlePaddle/Paddle/pull/53982)、[#50759](https://github.com/PaddlePaddle/Paddle/pull/50759)、[#51571](https://github.com/PaddlePaddle/Paddle/pull/51571)、 [#53144](https://github.com/PaddlePaddle/Paddle/pull/53144)、[#53306](https://github.com/PaddlePaddle/Paddle/pull/53306) -- Optimize BeamSearch performance. Transform, remove and fuse fine-grained operators such as write_read_array and gather, to improve model performance when beam_size=1. [#53130](https://github.com/PaddlePaddle/Paddle/pull/53130) -- Transform multiple stack operators with the same input into unsqueeze operators that support broadcast. Unsquee/squeeze supports inplace computation. [#52099](https://github.com/PaddlePaddle/Paddle/pull/52099) -- Add support for exporting multi-card inference models for Kunlunxin. [#50490](https://github.com/PaddlePaddle/Paddle/pull/50490) -- Add embedding_with_eltwise_add fusion pass and operator phi kernel, to reduce video memory usage and improve inference performance. [#50590](https://github.com/PaddlePaddle/Paddle/pull/50590) -- interpolate class operator phi kernel supports FP16. [#52358](https://github.com/PaddlePaddle/Paddle/pull/52358) -- argmax operator supports INT32 type output. [#51303](https://github.com/PaddlePaddle/Paddle/pull/51303) -- Fix the error of only model file when saving serialized model after turning on mixed-precision inference mode. [#52994](https://github.com/PaddlePaddle/Paddle/pull/52994) -- Fix segment error of instance_norm when scale and bias are empty. [#52627](https://github.com/PaddlePaddle/Paddle/pull/52627) -- conv_transpose operator supports FP16. [#53626](https://github.com/PaddlePaddle/Paddle/pull/53626) -- Add yolo_box_xpu fusion pass and operator phi kernel, to optimize YOLO model generic substructure. [#54163](https://github.com/PaddlePaddle/Paddle/pull/54163) -- Add conv2d_xpu fusion pass and operator phi kernel, and support FP16 inference, to optimize convolution operation inference consumption time. [#52247](https://github.com/PaddlePaddle/Paddle/pull/52247) ，[#53626](https://github.com/PaddlePaddle/Paddle/pull/53626) -- Add sigmoid_elementmul generic fusion pass, to fuse to swish operator to match conv2d_fusion pass to improve YOLO model inference performance. [#53580](https://github.com/PaddlePaddle/Paddle/pull/53580) -- Add act_add fusion pass and operator phi kernel to improve inference performance. [#53965](https://github.com/PaddlePaddle/Paddle/pull/53965) -- Add fold_interp_outsize fusion pass, to improve inference performance. [#54245](https://github.com/PaddlePaddle/Paddle/pull/54245) -- Solve the problem of incorrect results due to duplicate fusion when there is shared weight in FC. [#51108](https://github.com/PaddlePaddle/Paddle/pull/51108)、[#51039](https://github.com/PaddlePaddle/Paddle/pull/51039) -- Remove op_device attribute where operator is only used for training, to prevent wrong choice of place for training during inference. [#51029](https://github.com/PaddlePaddle/Paddle/pull/51029) -- Support saving of optimized models, allowing PASS optimization to be skipped in case of re-inference, to reduce first time inference time. [#53696](https://github.com/PaddlePaddle/Paddle/pull/53696) -- Solve the problem of computation error caused by the CPUPlace input of operator Kernel being forced to copy to XPU. [#51306](https://github.com/PaddlePaddle/Paddle/pull/51306) -- subblock supports early copying of H2D parameters to improve inference performance. [#51876](https://github.com/PaddlePaddle/Paddle/pull/51876) -- Fix scale memory size of the output activation of Kunlunxin 2nd generation chip. [#53505](https://github.com/PaddlePaddle/Paddle/pull/53505) -- In new executor Kunlunxin D2D copy, support asynchronous execution. [#51876](https://github.com/PaddlePaddle/Paddle/pull/51876) -- Remove concat operator with only one input. [#52304](https://github.com/PaddlePaddle/Paddle/pull/52304) -- lookup_table_v2 supports FP16 to remove redundant cast operator. [#52888](https://github.com/PaddlePaddle/Paddle/pull/52888) -- Control flow While operator supports caching scope, to reduce overhead of creating new scope every time. [#52628](https://github.com/PaddlePaddle/Paddle/pull/52628) -- Scatter newly supports FP16, to remove redundant cast operators and elementwise_mul operators with an input of 1. [#52831](https://github.com/PaddlePaddle/Paddle/pull/52831) - -### Model quantization -- Upgrade of dynamic graph quantization function. - - Add a new API for quantization training of dynamic graph models: ```paddle.quantization.QAT ```. Support passing quantization-related parameters through configuration, simplifying quantization training process and difficulty of secondary development. ([#49398](https://github.com/PaddlePaddle/Paddle/pull/49398)) - - Add a new offline quantization API: ```paddle.quantization.PTQ ```. Support exporting quantization model to model format supported by inference. ([#50107](https://github.com/PaddlePaddle/Paddle/pull/50107)) - - Add STUB operator to simulate actual quantization operation during training process. ([#50510](https://github.com/PaddlePaddle/Paddle/pull/50510)) -- Support quantization training model to load parameters of offline quantization model. Support more operators for quantization, including matmul, scale, and conv1d. [#47892](https://github.com/PaddlePaddle/Paddle/pull/47892)， [#45911](https://github.com/PaddlePaddle/Paddle/pull/45911)，[#48912](https://github.com/PaddlePaddle/Paddle/pull/48912) -- Support hybrid parallel training of static graph quantization training. [#52219](https://github.com/PaddlePaddle/Paddle/pull/52219) -- Fix the problem in the process of dynamic graph quantization: - - Repeat insertion of quantization nodes when exporting quantization training models. [#48751](https://github.com/PaddlePaddle/Paddle/pull/48751) - - Fix the problem of inserting quantization nodes into model input. [#49926](https://github.com/PaddlePaddle/Paddle/pull/49926) - -## 5. Environment Adaptation -Improve efficiency of source code compilation, and promote setuptools + ninja compilation method to increase development efficiency: In CPU scenarios, full amount of compilation time is reduced by 20 min, and compilation speed is increased by 24.52%. In GPU scenario, full amount of compilation time is reduced by 22 min, and compilation speed is increased by 29.31%. In order to adapt to mainstream development environments, PaddlePaddle supports gcc12 compilation and C++17 in the source code, and adapts to the latest CUDA12. In terms of code quality, complete cleanup of compilation warnings, to improve compilation experience. At the third-party dependency level, we have upgraded the version of underlying protobuf to reduce dependency, cleaned up deprecated attributes of some earlier versions of dependency libraries and old code formats, and removed support for Python 2.x. -- ninja compilation adaptation to improve compilation speed. [#52433](https://github.com/PaddlePaddle/Paddle/pull/52433),[#48932](https://github.com/PaddlePaddle/Paddle/pull/48932),[#49420](https://github.com/PaddlePaddle/Paddle/pull/49420),[#48435](https://github.com/PaddlePaddle/Paddle/pull/48435),[#49303](https://github.com/PaddlePaddle/Paddle/pull/49303),[#49448](https://github.com/PaddlePaddle/Paddle/pull/49448),[#49838](https://github.com/PaddlePaddle/Paddle/pull/49838),[#50067](https://github.com/PaddlePaddle/Paddle/pull/50067),[#52796](https://github.com/PaddlePaddle/Paddle/pull/52796),[#50431](https://github.com/PaddlePaddle/Paddle/pull/50431),[#49181](https://github.com/PaddlePaddle/Paddle/pull/49181),[#48867](https://github.com/PaddlePaddle/Paddle/pull/48867),[#48490](https://github.com/PaddlePaddle/Paddle/pull/48490),[#48211](https://github.com/PaddlePaddle/Paddle/pull/48211),[#49499](https://github.com/PaddlePaddle/Paddle/pull/49499),[#53076](https://github.com/PaddlePaddle/Paddle/pull/53076) -- setuptools compilation and package all-in-one adaptation. [#48770](https://github.com/PaddlePaddle/Paddle/pull/48770),[#46957](https://github.com/PaddlePaddle/Paddle/pull/46957),[#49583](https://github.com/PaddlePaddle/Paddle/pull/49583),[#47602](https://github.com/PaddlePaddle/Paddle/pull/47602),[#48301](https://github.com/PaddlePaddle/Paddle/pull/48301),[#50800](https://github.com/PaddlePaddle/Paddle/pull/50800),[#42575](https://github.com/PaddlePaddle/Paddle/pull/42575)),[#49826](https://github.com/PaddlePaddle/Paddle/pull/49826),[#49002](https://github.com/PaddlePaddle/Paddle/pull/49002),[#51443](https://github.com/PaddlePaddle/Paddle/pull/51443),[#51528](https://github.com/PaddlePaddle/Paddle/pull/51528),[#52621](https://github.com/PaddlePaddle/Paddle/pull/52621),[#52465](https://github.com/PaddlePaddle/Paddle/pull/52465) -- gcc12 support. [#52960](https://github.com/PaddlePaddle/Paddle/pull/52960),[#52265](https://github.com/PaddlePaddle/Paddle/pull/52265),[#46546](https://github.com/PaddlePaddle/Paddle/pull/46546),[#52318](https://github.com/PaddlePaddle/Paddle/pull/52318),[#46808](https://github.com/PaddlePaddle/Paddle/pull/46808),[#47466](https://github.com/PaddlePaddle/Paddle/pull/47466),[#52083](https://github.com/PaddlePaddle/Paddle/pull/52083),[#48176](https://github.com/PaddlePaddle/Paddle/pull/48176),[#49423](https://github.com/PaddlePaddle/Paddle/pull/49423),[#49452](https://github.com/PaddlePaddle/Paddle/pull/49452),[#51037](https://github.com/PaddlePaddle/Paddle/pull/51037),[#52007](https://github.com/PaddlePaddle/Paddle/pull/52007),[#52441](https://github.com/PaddlePaddle/Paddle/pull/52441),[#52085](https://github.com/PaddlePaddle/Paddle/pull/52085),[#50817](https://github.com/PaddlePaddle/Paddle/pull/50817),[#52646](https://github.com/PaddlePaddle/Paddle/pull/52646),[#50777](https://github.com/PaddlePaddle/Paddle/pull/50777),[#53288](https://github.com/PaddlePaddle/Paddle/pull/53288),[#54009](https://github.com/PaddlePaddle/Paddle/pull/54009) -- c++17 standard support. [#53345](https://github.com/PaddlePaddle/Paddle/pull/53345),[#53892](https://github.com/PaddlePaddle/Paddle/pull/53892),[#54282](https://github.com/PaddlePaddle/Paddle/pull/54282),[#49017](https://github.com/PaddlePaddle/Paddle/pull/49017),[#47635](https://github.com/PaddlePaddle/Paddle/pull/47635),[#54258](https://github.com/PaddlePaddle/Paddle/pull/54258) -- cuda12 support. [#52285](https://github.com/PaddlePaddle/Paddle/pull/52285),[#49592](https://github.com/PaddlePaddle/Paddle/pull/49592),[#52232](https://github.com/PaddlePaddle/Paddle/pull/52232),[#52654](https://github.com/PaddlePaddle/Paddle/pull/52654),[#54641](https://github.com/PaddlePaddle/Paddle/pull/54641) -- CodeStyle。[#45909](https://github.com/PaddlePaddle/Paddle/pull/45909),[#47772](https://github.com/PaddlePaddle/Paddle/pull/47772),[#48538](https://github.com/PaddlePaddle/Paddle/pull/48538),[#49522](https://github.com/PaddlePaddle/Paddle/pull/49522),[#47264](https://github.com/PaddlePaddle/Paddle/pull/47264),[#49558](https://github.com/PaddlePaddle/Paddle/pull/49558) -- Compilation Warning is removed. [#47163](https://github.com/PaddlePaddle/Paddle/pull/47163),[#47216](https://github.com/PaddlePaddle/Paddle/pull/47216),[#47309](https://github.com/PaddlePaddle/Paddle/pull/47309)，[#47252](https://github.com/PaddlePaddle/Paddle/pull/47252)，[#47341](https://github.com/PaddlePaddle/Paddle/pull/47341)，[#47399](https://github.com/PaddlePaddle/Paddle/pull/47399)，[#47513](https://github.com/PaddlePaddle/Paddle/pull/47513)，[#47558](https://github.com/PaddlePaddle/Paddle/pull/47558)，[#47706](https://github.com/PaddlePaddle/Paddle/pull/47706)，[#52717](https://github.com/PaddlePaddle/Paddle/pull/52717)，[#51203](https://github.com/PaddlePaddle/Paddle/pull/51203)，[#51336](https://github.com/PaddlePaddle/Paddle/pull/51336)，[#51608](https://github.com/PaddlePaddle/Paddle/pull/51608)，[#51633](https://github.com/PaddlePaddle/Paddle/pull/51633),[#46644](https://github.com/PaddlePaddle/Paddle/pull/46644),[#53092](https://github.com/PaddlePaddle/Paddle/pull/53092),[#53185](https://github.com/PaddlePaddle/Paddle/pull/53185),[#53246](https://github.com/PaddlePaddle/Paddle/pull/53246),[#53650](https://github.com/PaddlePaddle/Paddle/pull/53650),[#53683](https://github.com/PaddlePaddle/Paddle/pull/53683),[#53687](https://github.com/PaddlePaddle/Paddle/pull/53687),[#53886](https://github.com/PaddlePaddle/Paddle/pull/53886),[#53689](https://github.com/PaddlePaddle/Paddle/pull/53689),[#53679](https://github.com/PaddlePaddle/Paddle/pull/53679),[#53681](https://github.com/PaddlePaddle/Paddle/pull/53681),[#53532](https://github.com/PaddlePaddle/Paddle/pull/53532),[#47137](https://github.com/PaddlePaddle/Paddle/pull/47137),[#47045](https://github.com/PaddlePaddle/Paddle/pull/47045),[#52186](https://github.com/PaddlePaddle/Paddle/pull/52186),[#52490](https://github.com/PaddlePaddle/Paddle/pull/52490),[#53924](https://github.com/PaddlePaddle/Paddle/pull/53924),[#53938](https://github.com/PaddlePaddle/Paddle/pull/53938),[#53945](https://github.com/PaddlePaddle/Paddle/pull/53945),[#53851](https://github.com/PaddlePaddle/Paddle/pull/53851),[#53847](https://github.com/PaddlePaddle/Paddle/pull/53847),[#53818](https://github.com/PaddlePaddle/Paddle/pull/53818),[#53931](https://github.com/PaddlePaddle/Paddle/pull/53931) -- Support protobuf upgrade. [#49875](https://github.com/PaddlePaddle/Paddle/pull/49875),[#48495](https://github.com/PaddlePaddle/Paddle/pull/48495),[#49673](https://github.com/PaddlePaddle/Paddle/pull/49673),[#52499](https://github.com/PaddlePaddle/Paddle/pull/52499),[#51161](https://github.com/PaddlePaddle/Paddle/pull/51161),[#49168](https://github.com/PaddlePaddle/Paddle/pull/49168) -- Support offline compilation of third-party libraries. [#54326](https://github.com/PaddlePaddle/Paddle/pull/54326),[#54370](https://github.com/PaddlePaddle/Paddle/pull/54370),[#54335](https://github.com/PaddlePaddle/Paddle/pull/54335),[#54346](https://github.com/PaddlePaddle/Paddle/pull/54346),[#53744](https://github.com/PaddlePaddle/Paddle/pull/53744),[#54319](https://github.com/PaddlePaddle/Paddle/pull/54319),[#53915](https://github.com/PaddlePaddle/Paddle/pull/53915) -- Phi independent compilation header file dependency decoupling. [#50456](https://github.com/PaddlePaddle/Paddle/pull/50456),[#47088](https://github.com/PaddlePaddle/Paddle/pull/47088),[#52573](https://github.com/PaddlePaddle/Paddle/pull/52573),[#52651](https://github.com/PaddlePaddle/Paddle/pull/52651) -- Python2.x decommissioning. [#48685](https://github.com/PaddlePaddle/Paddle/pull/48685) - -## 6. Security -- Fix bugs such as null pointer usage, illegal address access, memory out of bounds, divide by 0, and Python IndexError [PR49976](https://github.com/PaddlePaddle/Paddle/pull/49976), [ PR49993](https://github.com/PaddlePaddle/Paddle/pull/49993)[, PR49942](https://github.com/PaddlePaddle/Paddle/pull/49942), [PR49965](https://github.com/PaddlePaddle/Paddle/pull/49965)[, PR50000](https://github.com/PaddlePaddle/Paddle/pull/50000)[, PR50005](https://github.com/PaddlePaddle/Paddle/pull/50005)[, PR49953](https://github.com/PaddlePaddle/Paddle/pull/49953)[, PR49995](https://github.com/PaddlePaddle/Paddle/pull/49995)[, PR49974](https://github.com/PaddlePaddle/Paddle/pull/49974)[, PR50015](https://github.com/PaddlePaddle/Paddle/pull/50015)[, PR50010](https://github.com/PaddlePaddle/Paddle/pull/50010), [PR49979](https://github.com/PaddlePaddle/Paddle/pull/49979), [PR49994](https://github.com/PaddlePaddle/Paddle/pull/49994), [PR49977](https://github.com/PaddlePaddle/Paddle/pull/49977)[, PR49968](https://github.com/PaddlePaddle/Paddle/pull/49968), [PR49984](https://github.com/PaddlePaddle/Paddle/pull/49984)[, PR49958](https://github.com/PaddlePaddle/Paddle/pull/49958)[, PR50008](https://github.com/PaddlePaddle/Paddle/pull/50008)[, PR51714](https://github.com/PaddlePaddle/Paddle/pull/51714), [PR51847](https://github.com/PaddlePaddle/Paddle/pull/51847), [PR51034](https://github.com/PaddlePaddle/Paddle/pull/51034)[, PR51088](https://github.com/PaddlePaddle/Paddle/pull/51088)[, PR51091](https://github.com/PaddlePaddle/Paddle/pull/51091)[, PR51092](https://github.com/PaddlePaddle/Paddle/pull/51092), [PR49966](https://github.com/PaddlePaddle/Paddle/pull/49966), [PR49656](https://github.com/PaddlePaddle/Paddle/pull/49656), [PR52161](https://github.com/PaddlePaddle/Paddle/pull/52161), [PR49548](https://github.com/PaddlePaddle/Paddle/pull/49548), [PR49546](https://github.com/PaddlePaddle/Paddle/pull/49546), [PR49547](https://github.com/PaddlePaddle/Paddle/pull/49547), [PR49549](https://github.com/PaddlePaddle/Paddle/pull/49549), [PR51850](https://github.com/PaddlePaddle/Paddle/pull/51850) - -## Thanks to our Contributors -This release contains contributions from: -1want2sleep, 201716010711, 404988613, 5u13, 6clc, Ackeraa, Aganlengzi, ahahahahahaha, Ainavo, Allen Guo, andyj, Asthestarsfalll, Aurelius84, Ayuan, BellaZYL, Bjmw3, Bo Zhang, bukejiyu, caozhou, carryyu, Ccc, ccrrong, ceci3, chalsliu, Chang Xu, CHANGer, Charles-hit, Chen Weihang, chenjian, Chenxiao Niu, chenxiao120660, chenxujun, Chitsing KUI, cifar10, co63oc, CollaborativeFiltering, csy0225, cxxly, cyber-pioneer, cyberslack_lee, czr-gc, Dandelight, danleifeng, Danyang Zhang, dasen, denglianbin, Difer, dongfangshenzhu, DrowFish19, duanboqiang, duanyanhui, engineer, engineer1109, Epsilon Luoo, feifei-111, Feiyu Chan, Feng Ni, feng_shuai, Fisher, FlyingQianMM, Frank Lin, Galaxy1458, GaoYuYang, gaoziyuan, gem5, GGBond8488, Ghost Screaming, gongenlei, gouzil, Guanghua Yu, Guo Sheng, Guoxia Wang, Hamid Zare, Hanchiao, handiz, Haohongxiang, haosicheng, haozi, Happyd99, heliqi, hellockx, hellolllw, heyanru, hg-1099255210, hh-qiao, hjyp, hong, HongyuJia, houj04, hua-zi, Huang Jiyi, Huang Zhengjie, huangjiyi, huangjun12, Hui Zhang, Huihuang Zheng, Hulek, hwa, HydrogenSulfate, Ikko Eltociear Ashimine, iLeGend, Infinity_lee, Infrared1029, Jacek Czaja, jakpiase, james, jameszhang, Jiabin Yang, jiahongyu, jiangcheng, jiangfan06, Jianghai, jiaqianjing, jingsongliu, JingZhuangzhuang, jjyaoao, joanna.wozna.intel, junxiu777, Jx-qi, JYChen, JZ-LIANG, jzhang533, Kai Song, Kai Xing, Kaipeng Deng, Kang Zhao, kangguangli, Kevin Wu Jiawen , Kim, Kim Yann, knamg, kuizhiqing, lanxianghit, Leding Li, Leo Chen, Leo Guo, levi131, Li Min, Li-fAngyU, Ligoml, lijialin03, lijin23, limingshu, Lin Manhui, LinearTemporalLogic, Linjie Chen, lishicheng1996, Little-chick, littleforest, liu zhengxi, liulinduo, liuruyan, liuzhenhai93, LiYuRio, lj970926, LokeZhou, LoneRanger, lubiu, Lucas, lugimzzz, Lux et Veritas, lxsbupt, LyndonKong, lzy, lzydev, Mahmoud Ashraf, Manan Goel, Maple Xie, Matsumoto Ruko, mayang002, MayYouBeProsperous, megemini, mengziheng, Meteor Liu, mhy, mhy-666, Ming-Xu Huang, ming1753, minghaoBD, mjxs, Moqim, Mountagha, Mr.Juice, mrcangye, NetPunk, Netpunk, nihao, niuliling123, Nyakku Shigure, OccupyMars2025, Ouyang Chao, pangengzheng, pangyoki, parap1uie-s, Paulina Gacek, Piotr Paturej, PommesPeter, PPGitub, PPPPzhang, PuQing, Qi Li, Qi Shao, QingshuChen, qipengh, qizhaoaoe, Rayman, RedContritio, RichardWooSJTU, risemeup1, Roc, ronnywang, Ruibiao Chen, Ruibin Cheung, RuohengMa, Ryan, SaltFish11, Sanbu, Scotty, scotty, seemingwang, Shaojie WANG, ShenLiang, shentanyue, Shijie, Shuangchi He, Siming Dai, Sing_chan, sneaxiy, Sonder, sprouteer, Sqhttwl, sunli, superwinner1, supplyout, SylarTiaNII, Sylwester Fraczek, Sławomir Siwek, taixiurong, Tao Luo, Taylor-Layrose, TeFeng Chen, Thomas Young, thunder95, Thunderbrook, Tian, Tian Zheng, tiancaishaonvjituizi, tianshuo78520a, tifa, Tinson Lai, Tomasz Socha, Tony Cao, ucsk, umiswing, ustiniankw, Vegetable dog, Vigi Zhang, Vvsmile, Wang Bojun, Wang Xin, Wang Xinyu, wangfengsheng1999, wangguanqun, wangguanzhong, wanghuancoder, wangna11BD, wangshengxiang, wangxiaoning, wangxinxin08, Wangzheee, WangZhen, wangzhen38, wasupandceacar, wawltor, Wei Shengyu, Weilong Wu, weishengying, Wen Sun, wenbin, wentao yu, wenzhe.wang, westfish, whisky-12, whs, Wilber, will-jl944, winter-wang, Winters Montagne, WJJ1995, wuhuachaocoding, wuyefeilin, wz1qqx, XiangGao, xiaoguoguo626807, xiaohemaikoo, xiaoluomi, xiaoting, xiaoxiaohehe001, Xiaoxu Chen, xiaoyuanzi914, Xinger, Xinyu Chen, xiongkun, xjmxyt, xu98bin, xysheng-baidu, yangguohao, yangjianfengo1, YangQun, YangZhou, yeliang2258, YepKong, Yichen Zhang, yikaikkk, Yiqun Liu, yjphhw, ykkk2333, Young-Flash, yu wentao, Yuang Liu, Yuanle Liu, YuanRisheng, yuchen202, yuehuayingxueluo, YuhangLi, Yulong Ao, YUNSHEN XIE, yunyaoXYY, YuRonan, zachary sun, ZeKai Zhou, Zenghui Yuan, zengshao0622, Zero Rains, Zhan Rongrui, Zhang Jun, Zhang Na, Zhang Ting, Zhang Zheng, zhangbo9674, ZhangDY-6483, zhangkaihuo, zhangxin81, zhangyikun02, zhangyingying520, zhangyuqin1998, zhaocaibei123, zhaoyingli, Zhen Wang, Zheng-Bicheng, Zhenghai Zhang, Zheng_Bicheng, zhenyun, Zhibao Li, zhiboniu, Zhong Hui, Zhou Wei, ZhouMengLei1999, zhoutianzi666, zhouzj, zhupengyang, zhurou603, zhuyipin, zhwesky2010, ziyoujiyi, zlsh80826, Zman, zmxdream, zqw_1997, Zuza Gawrysiak, zxcd, zyfncg, ZZK, zzk0, Ding Yi, Fu Jianhan, Liu Ge Gu Tou, Lu Lin, Zhou Zhouzhou, Jiang Yongyong, Xue Zhawu, Zhang Chunqiao, Zhang Zhenghai, Ning Meng Wei, Wang Mingdong, Shi Xiaowei, Chao Ji Ma Niu, Chen Cangye, Qi Ma Xiao Mao - -# 2.4.2 Release Note - - V2.4.2 fixed known bugs, and added a tiny set of features. - -## Training Framework (distributed included) - - - Fix the problem while using paddle.utils.dlpack.to_dlpack API to create dlpack objects multiple times in the for loop, and fix the bug that the reference counting error causes the memory actually pointed by dlpack to be destructed unexpectedly. [#50138](https://github.com/PaddlePaddle/Paddle/pull/50138) - - Fixed the issue of out-of-bounds memory access when the input tensor is multi-dimensional in paddle.multiplex API. [#49368](https://github.com/PaddlePaddle/Paddle/pull/49368) - - Fix the occasional compilation error caused by incorrect referencing of the Eigen header file. [#48157](https://github.com/PaddlePaddle/Paddle/pull/48157) - - Fixed the bug that the output value of the backward operator may be None when the output gradient parameter order of the custom operator is not continuous.[#48656](https://github.com/PaddlePaddle/Paddle/pull/48656) - - Add cutlass and implement the fusion kernel of gather+gemm+scatter; Optimize training and inference performance of sparse convolution; Optimize inference performance of batch_norm under 1D input data.[#50118](https://github.com/PaddlePaddle/Paddle/pull/50118) - - Fix compilation failure in gcc54 environment caused by using constexpr. [#50421](https://github.com/PaddlePaddle/Paddle/pull/50421) - - Move sum op kernel to PHI and fix bug that can't get correct SelectedRows' dims when run infermeta.[#49342](https://github.com/PaddlePaddle/Paddle/pull/49342) - - Fixed the issue that the fold operator accesses memory out of bounds under large bs input.[#49491](https://github.com/PaddlePaddle/Paddle/pull/49491) - - Fix the problem that no parameter Layer cannot call backward under dynamic to static mode.[#49812](https://github.com/PaddlePaddle/Paddle/pull/49812) - - Fix the compile problem of CUDA11.8 on windows platform.[#50205](https://github.com/PaddlePaddle/Paddle/pull/50205) - - Fix the unsupported error for `FusedDropoutActBiasGrad` on H100.[#47285](https://github.com/PaddlePaddle/Paddle/pull/47285) - - Add `debug_graphviz_path` option into `build_strategy`.[#46531](https://github.com/PaddlePaddle/Paddle/pull/46531) - - Fix the not closed `popen` object.[#47053](https://github.com/PaddlePaddle/Paddle/pull/47053) - -## Deployment Direction (Paddle Inference) - - - Improve the functionality and stability of mixed-precision inference. Reconstruct the implementation of interface convert_to_mixed_precision and add parameter precision to interface enable_use_gpu.[#49077](https://github.com/PaddlePaddle/Paddle/pull/49077)、[#49239](https://github.com/PaddlePaddle/Paddle/pull/49239)、[#49477](https://github.com/PaddlePaddle/Paddle/pull/49477) - - Support compilation under jetson ampere architecture.[#49364](https://github.com/PaddlePaddle/Paddle/pull/49364) - - Fixed fc kernel diff.[#49781](https://github.com/PaddlePaddle/Paddle/pull/49781) - - Fixed the error of trt workspace parameter type under CAPI. [#48350](https://github.com/PaddlePaddle/Paddle/pull/48350) - - Fixed the error caused by arg_max/arg_min without flatten dtype parameter in Paddle 1.x version. [#49771](https://github.com/PaddlePaddle/Paddle/pull/49771) - - Fixed the bug of missing information about lod logic after split infermeta's refactoring. [#49745](https://github.com/PaddlePaddle/Paddle/pull/49745) - - Fixed the bug of the constant-folding pass, which causes the conv2d weight to be non-persistent after folding and not enter the TensorRT engine. [#50105](https://github.com/PaddlePaddle/Paddle/pull/50105) - -# 2.4.1 Release Note - - -Remove the dependence of the Paddle on python.so, and fix the bug that fails to execute due to the inability to find python.so in specific environments, including conda. - - -# 2.4.0 Release Note - -## 1. Important Updates - -- **New dynamic graph architecture is officially effective**: The new dynamic graph framework has significantly improved the scheduling performance. The scheduling performance of more than 90% APIs is improved by over 50%, and the model performance of more than 50% kits is improved by over 5%. The functional architecture is clearer, and the secondary development capability and experience are significantly enhanced. - -- **Comprehensive improvement of the dynamic-static unification ability of the PaddlePaddle**: The dynamic-to-static function is provided with richer Python syntax support. The Python syntax coverage of the PaddlePaddle reaches 90%. The syntax transcription logic is mainly optimized to completely support the control flow syntax, with providing smooth dynamic-to-static graph experiences by pressing one key. With the newly upgraded static graph executor, the dynamic-to-static training has better acceleration capability, and the key model test shows that it is close to the best level of the static graph. The dynamic-to-static scalability is improved, with newly supporting multi-function merge export and inference. Users can use the PHI operator library for secondary development and flexible deployment. This can effectively support the custom decoding of U2++ featured models in the speech domain. - -- **Add sparse computing APIs**: Add 55 sparse APIs `paddle.sparse.*` and support mainstream sparse computing scenarios. The APIs have been applied to sparse training and inference deployment for 3D point cloud target detection, Sparse Transformers, and other tasks, with a speedup of 105.75% compared to DenseTensor in high sparse scenarios. In contrast to similar products, the speed of sparse computing is increased by 4.01%-58.55%. Support the computing of a variety of sparse Tensors (SparseCoo and SparseCsr). This is the ultimate saving of video memory. Meanwhile, it maintains a consistent usage experience, with the same usage method of the dense Tensor API. - -- **Large-scale graph neural network GPU training engine**: Through the heterogeneous hierarchical storage technology of SSD, memory, and video memory, it breaks through the video memory bottleneck and supports all-GPU storage and training of super-large-scale graphs. It realizes the all-GPU integrated solution of walk, sampling and training. This can increase the training speed by more than 10x under the same costs, compared to the traditional distributed CPU solution. - -- **Environment adaptation**: Add pre-compiled installer adapted to CUDA version 11.7. It newly supports the running in Ubuntu 22.04 or later. - -### Forward-looking forecast - -- PaddlePaddle Framework will deprecate support for python 3.6 in version 2.5. -- The PaddlePaddle framework will gradually deprecate the API under the `paddle.fluild` namespace on the python side, and some of the APIs under this namespace will be directly removed in version 2.5. - -## 2. Incompatibility upgrade - -- The pre-compiled installer for CUDA version 10.1 is cancelled. -- The -Tensor.clear_gradient(bool set_to_zero) interface will not take the value passed by kwargs, and will have to pass the bool variable of set_to_zero through args. -- In order to improve the utilization efficiency of video memory, only the gradients of forward leaf node variables, such as the gradients of network parameters in training, are retained in the dynamic graph by default, instead of the gradients of non-leaf nodes. If you need to preserve a specific Tensor gradient, you can call the Tensor.retain_grads() interface before reverse execution. -- paddle.autograd. PyLayer will no longer support the case where the input is tuple, pass in a list of Tensor if you want a group of them. - -## 3. Training framework (including the distributed feature) - -### （1）New APIs and enhanced API functions -- **Add the sparse computing class API**：paddle.sparse - - Add 55 sparse APIs and support mainstream sparse computing scenarios. The APIs have been applied to sparse training and inference deployment for 3D point cloud target detection, Sparse Transformers, and other tasks, with a speedup of 105.75% compared to DenseTensor in high sparse scenarios. In contrast to similar products, the speed of sparse computing is increased by 4.01%-58.55%. Support the computing of a variety of sparse Tensors (SparseCoo and SparseCsr). This is the ultimate saving of video memory. Meanwhile, it maintains a consistent usage experience, with the same usage method of the dense Tensor API.[#45849](https://github.com/PaddlePaddle/Paddle/pull/45849), [#46694](https://github.com/PaddlePaddle/Paddle/pull/46694), [#45086](https://github.com/PaddlePaddle/Paddle/pull/45086), [#41857](https://github.com/PaddlePaddle/Paddle/pull/41857), [#42935](https://github.com/PaddlePaddle/Paddle/pull/42935), [#43475](https://github.com/PaddlePaddle/Paddle/pull/43475), [#43668](https://github.com/PaddlePaddle/Paddle/pull/43668), [#43966](https://github.com/PaddlePaddle/Paddle/pull/43966), [#44022](https://github.com/PaddlePaddle/Paddle/pull/44022), [#44346](https://github.com/PaddlePaddle/Paddle/pull/44346), [#44432](https://github.com/PaddlePaddle/Paddle/pull/44432), [#44451](https://github.com/PaddlePaddle/Paddle/pull/44451), [#44743](https://github.com/PaddlePaddle/Paddle/pull/44743), [#42013](https://github.com/PaddlePaddle/Paddle/pull/42013), [#43520](https://github.com/PaddlePaddle/Paddle/pull/43520), [#41434](https://github.com/PaddlePaddle/Paddle/pull/41434), [#42130](https://github.com/PaddlePaddle/Paddle/pull/42130), [#41276](https://github.com/PaddlePaddle/Paddle/pull/41276), [#41857](https://github.com/PaddlePaddle/Paddle/pull/41857), [#41356](https://github.com/PaddlePaddle/Paddle/pull/41356) -- **Add the audio field API：** paddle.audio - - Add the feature extraction APIs such as MFCC, Spectrogram, and LogMelSpectrogram. Support the GPU computing. The performance increases by more than 15x compared to the CPU. This can significantly improve the GPU utilization in speech model training.[#45424](https://github.com/PaddlePaddle/Paddle/pull/45424) - - Add the feature extraction basic APIs such as Window Function and Discrete Cosine Transform. This can facilitate users to customize the speech feature extraction.[#45424](https://github.com/PaddlePaddle/Paddle/pull/45424) - - Add the speech I/O module. It provides 2 types of audio I/O backend and supports 6 types of codecs for convenient loading of speech data. [#45939](https://github.com/PaddlePaddle/Paddle/pull/45939) - - Add TESS and ESC50 speech classification datasets. It is convenient for users to complete the classical speech classification model.[#45939](https://github.com/PaddlePaddle/Paddle/pull/45939) -- **Add the graph learning domain API:** paddle.geometric - - Graph learning is gradually becoming a key technology in the field of machine learning. The new paddle.geometric module of PaddlePaddle provides a better modeling and training development experience of graph learning. - - Message passing: The message passing mechanism of the graph learning is the basis of graph modeling. We add 7 graph learning message passing APIs to make it more convenient to complete the modeling of the graph learning. Among them, 3 newly added message passing fusion operators can significantly reduce the GPU memory consumption in the GNN model training. In the dense graph scenarios, more than 50% of GPU memory can be saved in the models of GCN series, and the training speed can increase by more than 20%.[#44848](https://github.com/PaddlePaddle/Paddle/pull/44848), [#44580](https://github.com/PaddlePaddle/Paddle/pull/44580), [#43174](https://github.com/PaddlePaddle/Paddle/pull/43174), [#44970](https://github.com/PaddlePaddle/Paddle/pull/44970) - - Graph sampling: Graph sampling is the performance bottleneck of GNN model training. This newly added high-performance graph sampling operator supports high concurrent graph sampling. It can increase the sampling speed of GraphSage by more than 32 times and the model training speed by more than 12 times.[#44970](https://github.com/PaddlePaddle/Paddle/pull/44970) -- **Add the vision domain API** - - The paddle.vision is added with target detection domain operators.([#43736](https://github.com/PaddlePaddle/Paddle/pull/43736)), paddle.vision.generate_proposals([#43611](https://github.com/PaddlePaddle/Paddle/pull/43611)), paddle.vision.matrix_nms([#44357](https://github.com/PaddlePaddle/Paddle/pull/44357)), paddle.vision.prior_box 和 paddle.vision.box_coder( [#47282](https://github.com/PaddlePaddle/Paddle/pull/47282) ). - -- - **Add other API** - - Add the iinfo([#45321](https://github.com/PaddlePaddle/Paddle/pull/45321)), count_nonzero([#44169](https://github.com/PaddlePaddle/Paddle/pull/44169)), nanmedian([#42385](https://github.com/PaddlePaddle/Paddle/pull/42385)), remainder\_ ([#45266](https://github.com/PaddlePaddle/Paddle/pull/45266)), take([#44741](https://github.com/PaddlePaddle/Paddle/pull/44741)), triu_indices([#45168](https://github.com/PaddlePaddle/Paddle/pull/45168)), sgn([#44568](https://github.com/PaddlePaddle/Paddle/pull/44568)), bucketize([#44195](https://github.com/PaddlePaddle/Paddle/pull/44195)), nanquantile([#41343](https://github.com/PaddlePaddle/Paddle/pull/41343)), frac([#41226](https://github.com/PaddlePaddle/Paddle/pull/41226)), logcumsumexp([#42267](https://github.com/PaddlePaddle/Paddle/pull/42267)), pairwise_distance([#44161](https://github.com/PaddlePaddle/Paddle/pull/44161)), heaviside([#41872](https://github.com/PaddlePaddle/Paddle/pull/41872)), logspace([#41261](https://github.com/PaddlePaddle/Paddle/pull/41261)), corrcoef([#40690](https://github.com/PaddlePaddle/Paddle/pull/40690)) - - Add the RReLU([#41823](https://github.com/PaddlePaddle/Paddle/pull/41823)), CyclicLR([#40698](https://github.com/PaddlePaddle/Paddle/pull/40698)), OneCycleLR([#41825](https://github.com/PaddlePaddle/Paddle/pull/41825)), Softmax2D([#40910](https://github.com/PaddlePaddle/Paddle/pull/40910)), SoftMarginLoss([#42364](https://github.com/PaddlePaddle/Paddle/pull/42364)), MultiLabelSoftMarginLoss([#41183](https://github.com/PaddlePaddle/Paddle/pull/41183)), TripletMarginLoss([#40487](https://github.com/PaddlePaddle/Paddle/pull/40487)), TripletMarginWithDistanceLoss([#40545](https://github.com/PaddlePaddle/Paddle/pull/40545)), CosineEmbeddingLoss 和 cosine_embedding_loss([#41680](https://github.com/PaddlePaddle/Paddle/pull/41680)), PixelUnshuffle([#40728](https://github.com/PaddlePaddle/Paddle/pull/40728)), ChannelShuffle([#40743](https://github.com/PaddlePaddle/Paddle/pull/40743)) -- **Enhanced API functions** - - Add the large batch_size calculation function of BatchNorm1D [#43072](https://github.com/PaddlePaddle/Paddle/pull/43072) -- **Optimize the collective communications distributed training API** - - Optimize the `fleet.init` function, and add the `log_level` parameter to facilitate users to view logs during operation [#45909](https://github.com/PaddlePaddle/Paddle/pull/45909) - - Add the `paddle.distributed.fleet.recompute_sequential paddle.distributed.fleet.recompute_hybrid` interface. It is convenient for users to use the recompute function [#45348](https://github.com/PaddlePaddle/Paddle/pull/45348) - - Add the `paddle.distributed.fleet.layers.mpu` package. It is convenient for users to use tensor parallel function [#45803](https://github.com/PaddlePaddle/Paddle/pull/45803) - - Add the communication API `paddle.distributed.destroy_process_group paddle.distributed.isend paddle.distributed.irecv paddle.distributed.all_to_all_single`. It improves the completeness and ease of use of communication [#43918](https://github.com/PaddlePaddle/Paddle/pull/43918) - - Add the `paddle.distributed.stream` package. The performance is increased by 5% to 10% compared to the base version[#46023](https://github.com/PaddlePaddle/Paddle/pull/46023) [#45282](https://github.com/PaddlePaddle/Paddle/pull/45282) - - The communication API is added with the support of multiple data types such as `Char/Byte/Bool`. It improves the completeness and ease of use of communication [#45574](https://github.com/PaddlePaddle/Paddle/pull/45574) [#45440](https://github.com/PaddlePaddle/Paddle/pull/45440) - - The communication API asynchronous parameter is changed from`use_calc_stream` to `sync_op`, It enhances the semantic readability of the interface [#46493](https://github.com/PaddlePaddle/Paddle/pull/46493) -- **Enhanced high-level API** - - The visual model ResNeXt in the high-level API implements the reuse of the ResNet code for refactoring. [#40588](https://github.com/PaddlePaddle/Paddle/pull/40588) - - The visual models Inceptionv3, MobileNetv1, MobileNetv2, and ShuffleNetv2 in the high level API are improved.[#40431](https://github.com/PaddlePaddle/Paddle/pull/40431) - -### （2）New functions and important upgrades - -- **The new dynamic graph architecture is officially launched**：The scheduling performance of the new dynamic graph framework is greatly improved. Compared with the original architecture, the scheduling performance is significantly enhanced. The scheduling performance of more than 90% APIs is improved by over 50%, and the model performance of more than 50% of kits is improved by over 5%. The new dynamic graph architecture is clear, and the coupling is low. The learning and development costs of extension modules such as Hook and PyLayer are significantly reduced based on the new architecture. [#37550](https://github.com/PaddlePaddle/Paddle/pull/37550) , [#37574](https://github.com/PaddlePaddle/Paddle/pull/37574) , [#37813](https://github.com/PaddlePaddle/Paddle/pull/37813) , [#37926](https://github.com/PaddlePaddle/Paddle/pull/37926) , [#39192](https://github.com/PaddlePaddle/Paddle/pull/39192) , [#37599](https://github.com/PaddlePaddle/Paddle/pull/37599) , [#37406](https://github.com/PaddlePaddle/Paddle/pull/37406) , [#37466](https://github.com/PaddlePaddle/Paddle/pull/37466) , [#37599](https://github.com/PaddlePaddle/Paddle/pull/37599) , [#40945](https://github.com/PaddlePaddle/Paddle/pull/40945) , [#39989](https://github.com/PaddlePaddle/Paddle/pull/39989) - -- **High-order auto-differentiation mechanism**：In order to better support scientific computing and other scenarios, the PaddlePaddle framework has been further improved and optimized for higher-order auto-differentiation capabilities. At present, the `paddle.incubate.autograd` directory has provided relevant trial functions and APIs for forward/reverse higher-order auto-differentiation (Currently they are in incubation, and related functions and API signatures may change).If you intend to implement related models and explore the auto-differentiation mechanism by yourself, please read the [usage and limitations of higher-order auto-differentiation](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/incubate/autograd/Overview_cn.html) carefully. Specific upgrades include： - 1. Static graph higher-order differentiation mechanism upgrade. Through the base operator system and program transformation, it supports higher-order forward and reverse differentiation, with the availability of the compiler and distributed functions.[#41919](https://github.com/PaddlePaddle/Paddle/pull/41919), [#41201](https://github.com/PaddlePaddle/Paddle/pull/41201) - 2. Add the forward and reverse higher-order auto-differentiation API, `paddle.incubate.autograd.forward_grad`, `paddle.incubate.autograd.grad`. [#43354](https://github.com/PaddlePaddle/Paddle/pull/43354) - 3. Add 18 higher-order auto-differentiation operators:`sin`, `cos`, `exp`, `erf`, `abs`, `log`, `cast`, `where`, `equal`, `not_equal`, `greater_than`, `greater_equal`, `elementwise_pow` `square`, `elementwise_max`, `gelu`, `reduce_mean`, `size`. [#46184](https://github.com/PaddlePaddle/Paddle/pull/46184), [#46024](https://github.com/PaddlePaddle/Paddle/pull/46024), [#45888](https://github.com/PaddlePaddle/Paddle/pull/45888), [#45338](https://github.com/PaddlePaddle/Paddle/pull/45338), [#44345](https://github.com/PaddlePaddle/Paddle/pull/44345) - 4. Fix the existing bugs of the operators such as`elementwise_div`, `reduce_sum`, `p_norm`. [#46514](https://github.com/PaddlePaddle/Paddle/pull/46514), [#46184](https://github.com/PaddlePaddle/Paddle/pull/46184) -- **Generic heterogeneous parameter server architecture**： - - Parameter server GPUGraph infrastructure upgraded to meet the implementation needs of large-scale applications: The storage and training of large-scale graph neural networks based on the traditional CPU feature high cost, low stability, and less performance. To overcome these problems, we have built a pure GPU graph training engine (PGLBox). Through the heterogeneous hierarchical storage technology of SSD, memory and video memory, it supports the training of ultra-large scale graph models. The training performance is improved by more than 10x compared with CPU graph training engine on the premise of equal cost. The task failure rate is extremely low.[#44594](https://github.com/PaddlePaddle/Paddle/pull/44594) - - Large-scale federation parameter server architecture: For large-scale personalized recommendation scenarios, the large-scale federation parameter server training is developed based on the heterogeneous PS infrastructure, to support horizontal and vertical federation under hundreds of billions of parameters. It includes two features: User private parameters updated locally and public parameters updated remotely. Users can flexibly configure the slicing policy for private and public parameters. A new central scheduling node Coordinator is added. Users can perform secondary development from the base class to customize the Client selection policy. [#42682](https://github.com/PaddlePaddle/Paddle/pull/42682) , [#44864](https://github.com/PaddlePaddle/Paddle/pull/44864) , [#44327](https://github.com/PaddlePaddle/Paddle/pull/44327) -- **Adaptive parallel** - - Design and launch a complete automatic parallelism interface system: Support automatic dynamic-to-static distributed training, automatic distributed data loading, automatic distributed saving and loading, automatic parameter conversion, custom slice marker and custom execution process. Users can easily obtain the automatic distributed training capability based on a single machine networking. It supports data parallel, model parallel, pipeline parallel, and hybrid parallel. [#45776](https://github.com/PaddlePaddle/Paddle/pull/45776) ，[#46552](https://github.com/PaddlePaddle/Paddle/pull/46552) , [#44202](https://github.com/PaddlePaddle/Paddle/pull/44202) , [#45840](https://github.com/PaddlePaddle/Paddle/pull/45840) , [#45518](https://github.com/PaddlePaddle/Paddle/pull/45518) , [#40528](https://github.com/PaddlePaddle/Paddle/pull/40528), [#42838](https://github.com/PaddlePaddle/Paddle/pull/42838), [#43093](https://github.com/PaddlePaddle/Paddle/pull/43093), [#43312](https://github.com/PaddlePaddle/Paddle/pull/43312), [#45053](https://github.com/PaddlePaddle/Paddle/pull/45053). - - Improve the underlying adaptive parallel mechanism, including the upgrade of the distributed costmodel design and implementation, to provide better evaluation of the slice policy. Add the native distributed properties to ProgramIR and enrich the Cluster functions. [#40457](https://github.com/PaddlePaddle/Paddle/pull/40457) , [#42601](https://github.com/PaddlePaddle/Paddle/pull/42601) , [#42727](https://github.com/PaddlePaddle/Paddle/pull/42727) , [#42874](https://github.com/PaddlePaddle/Paddle/pull/42784) , [#43114](https://github.com/PaddlePaddle/Paddle/pull/43114) , [#44095](https://github.com/PaddlePaddle/Paddle/pull/44095) , [#44146](https://github.com/PaddlePaddle/Paddle/pull/44146) , [#44701](https://github.com/PaddlePaddle/Paddle/pull/44701) , [#44973](https://github.com/PaddlePaddle/Paddle/pull/44973) , [#45002](https://github.com/PaddlePaddle/Paddle/pull/45002) , [#45118](https://github.com/PaddlePaddle/Paddle/pull/45118) , [#45237](https://github.com/PaddlePaddle/Paddle/pull/45237) , [#42576](https://github.com/PaddlePaddle/Paddle/pull/42576) , [#41722](https://github.com/PaddlePaddle/Paddle/pull/41722) , [#44150](https://github.com/PaddlePaddle/Paddle/pull/44150) , [#44989](https://github.com/PaddlePaddle/Paddle/pull/44989), [#44951](https://github.com/PaddlePaddle/Paddle/pull/44951), [#44963](https://github.com/PaddlePaddle/Paddle/pull/44963) . - - Add the Shardingstage1/2/3 AutoTuning feature under data parallel. This allows to automatically select the highest throughput Shardingstage policy while ensuring that the video memory constraints are met. [#43782](https://github.com/PaddlePaddle/Paddle/pull/43782) . - -- **Training hardware access - Plug-in solutions**：Add custom Runtime/Kernel/CCL/Graph/Pass solutions. The hardware vendors can choose which modules to implement on-demand based on hardware characteristics. - -- **ONNX format export** - - Support the quantized model export. The exported ONNX model uses TensorRT or ONNXRuntime to load inference. About 1.5~4 times inference acceleration can be obtained [#856](https://github.com/PaddlePaddle/Paddle2ONNX/pull/856), [#782](https://github.com/PaddlePaddle/Paddle2ONNX/pull/782) - - Add the export of a large model greater than 2GB [#942](https://github.com/PaddlePaddle/Paddle2ONNX/pull/942) - -### （3）Function optimization -- **Comprehensive increase of dynamic-to-static analysis conversion & extension capabilities** - - In order to improve the success rate and experience of model dynamic-to-static conversion, the transcription logic of control flow syntax is reconstructed. The core syntax has been upgraded to JIT (just-in-time) paradigm to achieve equivalent transcription with Python codes. The syntax functions such as break, return and continue are improved.[#43666](https://github.com/PaddlePaddle/Paddle/pull/43666) , [#43846](https://github.com/PaddlePaddle/Paddle/pull/43846) , [#43848](https://github.com/PaddlePaddle/Paddle/pull/43848) , [#43880](https://github.com/PaddlePaddle/Paddle/pull/43880) , [#43957](https://github.com/PaddlePaddle/Paddle/pull/43957) , [#43328](https://github.com/PaddlePaddle/Paddle/pull/43328) , [#43348](https://github.com/PaddlePaddle/Paddle/pull/43348) , [#43998](https://github.com/PaddlePaddle/Paddle/pull/43998) , [#44465](https://github.com/PaddlePaddle/Paddle/pull/44465) , [#44504](https://github.com/PaddlePaddle/Paddle/pull/44504) , [#43713](https://github.com/PaddlePaddle/Paddle/pull/43713) , [#43864](https://github.com/PaddlePaddle/Paddle/pull/43864) , [#43967](https://github.com/PaddlePaddle/Paddle/pull/43967) , [#44155](https://github.com/PaddlePaddle/Paddle/pull/44155) , [#44487](https://github.com/PaddlePaddle/Paddle/pull/44487) , [#44527](https://github.com/PaddlePaddle/Paddle/pull/44527) , [#45105](https://github.com/PaddlePaddle/Paddle/pull/45105) , [#45900](https://github.com/PaddlePaddle/Paddle/pull/45900) - - In order to support the voice custom decoding flexible deployment scenarios, the jit.save/load interface function is extended to support user multi-function merge and export. A new JITLayer component is added to support the invocation of class functions. Meanwhile, the custom inference deployment function is implemented with the PHI operator library C++ API. [#44283](https://github.com/PaddlePaddle/Paddle/pull/44283), [#41783](https://github.com/PaddlePaddle/Paddle/pull/41783), [#43607](https://github.com/PaddlePaddle/Paddle/pull/43607), [#43754](https://github.com/PaddlePaddle/Paddle/pull/43754), [#43758](https://github.com/PaddlePaddle/Paddle/pull/43758), [#43798](https://github.com/PaddlePaddle/Paddle/pull/43798), [#44010](https://github.com/PaddlePaddle/Paddle/pull/44010), [#44351](https://github.com/PaddlePaddle/Paddle/pull/44351), [#44465](https://github.com/PaddlePaddle/Paddle/pull/44465), [#44504](https://github.com/PaddlePaddle/Paddle/pull/44504), [#44597](https://github.com/PaddlePaddle/Paddle/pull/44597), [#44738](https://github.com/PaddlePaddle/Paddle/pull/44738), [#44984](https://github.com/PaddlePaddle/Paddle/pull/44984), [#46249](https://github.com/PaddlePaddle/Paddle/pull/46249) - - In order to unify API dynamic and static behaviors, 20 operators are upgraded to support variable attribute information of Op in static graphs, to ensure consistent dynamic and static behaviors and improve the success rate of dynamic-to-static conversion of models. Include `pad2d`,`depthwise_conv2d_transpose`,`conv2d_transpose`,`adaptive_avg_pool2d`,`reverse`,`bincount`,`multinomial`,`reduce_sum`,`reduce_mean`,`reduce_prod`,`reduce_min`,`reduce_max`,`uniform`,`squeeze`,`max_unpool2d`,`dropout`,`cumsum`,`eye`,`argmin`,`argmax`. [#44737](https://github.com/PaddlePaddle/Paddle/pull/44737), [#45084](https://github.com/PaddlePaddle/Paddle/pull/45084), [#45189](https://github.com/PaddlePaddle/Paddle/pull/45189), [#45391](https://github.com/PaddlePaddle/Paddle/pull/45391), [#45417](https://github.com/PaddlePaddle/Paddle/pull/45417), [#45427](https://github.com/PaddlePaddle/Paddle/pull/45427), [#45514](https://github.com/PaddlePaddle/Paddle/pull/45514), [#45525](https://github.com/PaddlePaddle/Paddle/pull/45525), [#45543](https://github.com/PaddlePaddle/Paddle/pull/45543), [#45660](https://github.com/PaddlePaddle/Paddle/pull/45660), [#46352](https://github.com/PaddlePaddle/Paddle/pull/46352/), [#46433](https://github.com/PaddlePaddle/Paddle/pull/46433), [#45078](https://github.com/PaddlePaddle/Paddle/pull/45078), [#45342](https://github.com/PaddlePaddle/Paddle/pull/45342), [#45372](https://github.com/PaddlePaddle/Paddle/pull/45372), [#45453](https://github.com/PaddlePaddle/Paddle/pull/45453), [#45522](https://github.com/PaddlePaddle/Paddle/pull/45522), [#45620](https://github.com/PaddlePaddle/Paddle/pull/45620) - - In order to solve the problem of occasional loss of error reporting stack for user dynamic-to-static, the logic of the error reporting module is optimized to improve the readability of the error reporting stack and the user debugging experience. [#44054](https://github.com/PaddlePaddle/Paddle/pull/44054), [#44083](https://github.com/PaddlePaddle/Paddle/pull/44083), [#44781](https://github.com/PaddlePaddle/Paddle/pull/44781), [#44996](https://github.com/PaddlePaddle/Paddle/pull/44996) - - Add the TypeHint syntax recognition and transcription module to fully support Python Type Hint syntax. [#47121](https://github.com/PaddlePaddle/Paddle/pull/47121) - -- **PHI operator library covers the full amount of arithmetic class operators**：Continuously build the highly reusable operator library PHI. The remaining PaddlePaddle 2.x arithmetic class PythonAPI-associated operators and related kernels are migrated to the PHI operators library and rewritten as functional expression. Add about 180 forward/reverse operator CPU&GPU kernels, and 170 Kunlun-specific arithmetic kernels. This further enhances the kernel function sets that can be reused when new operators are added. In addition, add more than 100 C++ arithmetic class APIs. These APIs can be used in the custom operators, further enhancing the ease of use for external extension development based on the PaddlePaddle. [#44577](https://github.com/PaddlePaddle/Paddle/pull/44577), [#44631](https://github.com/PaddlePaddle/Paddle/pull/44631), [#44434](https://github.com/PaddlePaddle/Paddle/pull/44434), [#44605](https://github.com/PaddlePaddle/Paddle/pull/44605), [#44676](https://github.com/PaddlePaddle/Paddle/pull/44676), [#44742](https://github.com/PaddlePaddle/Paddle/pull/44742), [#44436](https://github.com/PaddlePaddle/Paddle/pull/44436) , [#45887](https://github.com/PaddlePaddle/Paddle/pull/45887), [#45851](https://github.com/PaddlePaddle/Paddle/pull/45851), [#45623](https://github.com/PaddlePaddle/Paddle/pull/45623), [#45397](https://github.com/PaddlePaddle/Paddle/pull/45397), [#45863](https://github.com/PaddlePaddle/Paddle/pull/45863) - -- **Normalized operator definitions with significantly improving the model simplicity**：For the problems of many redundant parameters in the historical operator definitions of PaddlePaddle 1.x and the high cost of understanding the adaptation, the redundant parameters of about 150 high-frequency operators are cleaned up centrally. Basically, the mathematically irrelevant parameters are removed. After these redundant parameters are cleaned up, the amount of information in the inference model stored in the PaddlePaddle is significantly reduced. Generally, about 40% of the attribute variables are removed, significantly improving the clarity of the PaddlePaddle operator definition, and improving the experience of model analysis and debugging. Meanwhile, the size of the inference model stored in the PaddlePaddle is also significantly reduced by more than 70%. As a result, this can significantly improve the lightweight of the PaddlePaddle model. [#44310](https://github.com/PaddlePaddle/Paddle/pull/44310) , [#45613](https://github.com/PaddlePaddle/Paddle/pull/45613) , [#45684](https://github.com/PaddlePaddle/Paddle/pull/45684) , [#45708](https://github.com/PaddlePaddle/Paddle/pull/45708) , [#45758](https://github.com/PaddlePaddle/Paddle/pull/45758) , [#45786](https://github.com/PaddlePaddle/Paddle/pull/45786) , [#45772](https://github.com/PaddlePaddle/Paddle/pull/45772) , [#45845](https://github.com/PaddlePaddle/Paddle/pull/45845) , [#45984](https://github.com/PaddlePaddle/Paddle/pull/45984) , [#46218](https://github.com/PaddlePaddle/Paddle/pull/46218) , [#46553](https://github.com/PaddlePaddle/Paddle/pull/46553) - -### （4）Performance optimization - -- AMP performance and accuracy optimization - - More operators are added with the support of FP16 data types, including elementwise series operators, compare series operators, strided_slice, set_value, uniform_ramdom, etc.（[#45504](https://github.com/PaddlePaddle/Paddle/pull/45504) [#44405](https://github.com/PaddlePaddle/Paddle/pull/44405) [#45496](https://github.com/PaddlePaddle/Paddle/pull/45496) [#46641](https://github.com/PaddlePaddle/Paddle/pull/46641), [#46906](https://github.com/PaddlePaddle/Paddle/pull/46906) ） - - Optimize the implementation scheme of the hard_swish operator FP16 Kernel to guarantee the accuracy without loss. （ [35386](https://github.com/PaddlePaddle/Paddle/pull/35386) ） - - More operators are added with the support of BF16 data types, including fused_linear, empty, selu, pow, adam, clip, embedding, gelu, pad3d, pixel_shuffle, tile, where, etc. [#46364](https://github.com/PaddlePaddle/Paddle/pull/46364), [#47177](https://github.com/PaddlePaddle/Paddle/pull/47177) -- AutoTuning of single machine training performance - - Transpose OP supports automatic Kernel selection mechanism. This allows the automatic search for the best Kernel implementation for different model configurations, improving the model performance. [#43310](https://github.com/PaddlePaddle/Paddle/pull/43310) (Transpose Op access AutoTuning function) - - AMP Layout auto-switching supports the new dynamic graph mode. For the ResNet50, TSM, and DeepLabV3 models, the performance increases by 9%-21% by Layout AutoTuning in the new dynamic graph. ([#45409](https://github.com/PaddlePaddle/Paddle/pull/45409), [#45751](https://github.com/PaddlePaddle/Paddle/pull/45751), [#45826](https://github.com/PaddlePaddle/Paddle/pull/45826), [#46880](https://github.com/PaddlePaddle/Paddle/pull/46880)) -- Generic performance optimization of GPU single machine training - - Optimize the Cache scheme of the Conv operator cuDNN algorithm and Cache the results in all algorithm acquisition methods. This can significantly reduce the CPU overhead of the operator.（[#41891](https://github.com/PaddlePaddle/Paddle/pull/41891) [#47197](https://github.com/PaddlePaddle/Paddle/pull/47197) ） - - Further optimize the GPU Kernel and Python side performance of multiple operators, including dist, poisson, depthwise_conv2d, transpose, eigh, broadcast computation, reduce computation, layer_norm, cross_entropy, etc. This can achieve better performance in more configuration scenarios. （[#44946](https://github.com/PaddlePaddle/Paddle/pull/44946), [#45057](https://github.com/PaddlePaddle/Paddle/pull/45057), [#45160](https://github.com/PaddlePaddle/Paddle/pull/45160), [#42491](https://github.com/PaddlePaddle/Paddle/pull/42491), [#42704](https://github.com/PaddlePaddle/Paddle/pull/42704), [#42853](https://github.com/PaddlePaddle/Paddle/pull/42853), [#46287](https://github.com/PaddlePaddle/Paddle/pull/46287), [#46362](https://github.com/PaddlePaddle/Paddle/pull/46362), [#46490](https://github.com/PaddlePaddle/Paddle/pull/46490), [#46412](https://github.com/PaddlePaddle/Paddle/pull/46412), [#46623](https://github.com/PaddlePaddle/Paddle/pull/46623), [#40051](https://github.com/PaddlePaddle/Paddle/pull/40051) ） -- Performance optimization of distributed training for collective communications - - To improve pipeline parallel scheduling efficiency, support the dynamic graph Interleaving1F1B scheduling policy. In the GPT-3 model, the performance is improved by 3%-4%. [#45797](https://github.com/PaddlePaddle/Paddle/pull/45797) , [#45869](https://github.com/PaddlePaddle/Paddle/pull/45869) , [#45922](https://github.com/PaddlePaddle/Paddle/pull/45922) , [#46209](https://github.com/PaddlePaddle/Paddle/pull/46209) , [#45402](https://github.com/PaddlePaddle/Paddle/pull/45402) , [#45444](https://github.com/PaddlePaddle/Paddle/pull/45444) , [#45497](https://github.com/PaddlePaddle/Paddle/pull/45497) , [#45797](https://github.com/PaddlePaddle/Paddle/pull/45797) , [#45869](https://github.com/PaddlePaddle/Paddle/pull/45869) , [#45922](https://github.com/PaddlePaddle/Paddle/pull/45922), [#46209](https://github.com/PaddlePaddle/Paddle/pull/46209), [#46399](https://github.com/PaddlePaddle/Paddle/pull/46399) , [#46483](https://github.com/PaddlePaddle/Paddle/pull/46483) , [#46876](https://github.com/PaddlePaddle/Paddle/pull/46876) , [#47242](https://github.com/PaddlePaddle/Paddle/pull/47242) , [#47249](https://github.com/PaddlePaddle/Paddle/pull/47249) , [#47497](https://github.com/PaddlePaddle/Paddle/pull/47497) , [#47517](https://github.com/PaddlePaddle/Paddle/pull/47517) - - To improve the distributed training performance of the MLPerfBERT model, the DistributedFusedLamb distributed optimizer supports hierarchical AllReduce. It improves MLPerfBERT performance by 17% on the DCU1024 card. [#44821](https://github.com/PaddlePaddle/Paddle/pull/44821) , [#44843](https://github.com/PaddlePaddle/Paddle/pull/44843) - - To optimize the video memory footprint when using DataParallel, the Buffer Lazy initialization policy for Tensor Fusion is supported, thus reducing the video memory footprint by an amount equal to the number of model parameters. [#45631](https://github.com/PaddlePaddle/Paddle/pull/45631). - - Distributed parallel policies DataParallel and Sharding support BF16 training. [#46846](https://github.com/PaddlePaddle/Paddle/pull/46846) , [#47246](https://github.com/PaddlePaddle/Paddle/pull/47246) - - To support the Sequence Parallel policy, the Distributed Pipeline Parallel supports enable_partial_send_recv policy, and supports the tensor after slice of the transmission sequence parallel. [#46992](https://github.com/PaddlePaddle/Paddle/pull/46992) , [#47083](https://github.com/PaddlePaddle/Paddle/pull/47083) - - To improve the performance of sharding stage 2 policy, implement the overlap of sharding stage 2 optimizer broadcast parameters with next step forward and use multi-CUDA Stream for communication. In the GPT 6.7B model, the 16-card training performance is improved by 11%. [#46495](https://github.com/PaddlePaddle/Paddle/pull/46495) , [#46656](https://github.com/PaddlePaddle/Paddle/pull/46656) , [#47061](https://github.com/PaddlePaddle/Paddle/pull/47061) - -### （5）Bug fix - -- Dynamic-to-static - - Fix the bug of reporting an error in dynamic-to-static of the model in a Parameter no-gradient scenario during multi-card training. [#44485](https://github.com/PaddlePaddle/Paddle/pull/44485) - - Fix the bug of where redundant frame logs are mistakenly output by the terminal in the dynamic-to-static. [#45754](https://github.com/PaddlePaddle/Paddle/pull/45754), [#46800](https://github.com/PaddlePaddle/Paddle/pull/46800) - - Fix the bug of reporting an error in the dynamic-to-static training when the control flow in the model contains a Tensor that does not require a gradient. [#43034](https://github.com/PaddlePaddle/Paddle/pull/43034) - - Fix the bug of incorrect computation value during gradient aggregation in the dynamic-to-static training. [#44893](https://github.com/PaddlePaddle/Paddle/pull/44893) - - Fix the bug of reporting an error in the dynamic-to-static when the function is decorated with @staticmethod. [#44983](https://github.com/PaddlePaddle/Paddle/pull/44983), [#45268](https://github.com/PaddlePaddle/Paddle/pull/45268), [#45277](https://github.com/PaddlePaddle/Paddle/pull/45277) - - Fix the bug of too much video memory footprint in some scenarios where the model contains the dynamic-to-static training. [#45380](https://github.com/PaddlePaddle/Paddle/pull/45380) - - Fix the bug of reporting an error of dynamic-to-static shape derivation in the networking phase when the model contains a complex control flow. [#45916](https://github.com/PaddlePaddle/Paddle/pull/45916), [#46020](https://github.com/PaddlePaddle/Paddle/pull/46020) -- Fix the error report mechanism - - Replace self.assertTrue(np.allclose(...)) with np.testing.assert_allclose to get fuller error reporting information ( [#44947](https://github.com/PaddlePaddle/Paddle/pull/44947), [#44988](https://github.com/PaddlePaddle/Paddle/pull/44988), [#45213](https://github.com/PaddlePaddle/Paddle/pull/45213)) -- Distributed training in collective communications - - Fix several bugs in communication library initialization and communication process, and enhance the system operation stability. [#44964](https://github.com/PaddlePaddle/Paddle/pull/44964) [#45100](https://github.com/PaddlePaddle/Paddle/pull/45100) [#44758](https://github.com/PaddlePaddle/Paddle/pull/44758) - - Fix the bug of frequent occurrences of hang in pipeline parallel, and enhance the ease of use of the policy [#47201](https://github.com/PaddlePaddle/Paddle/pull/47201); enhance the pipeline function to support unbalanced input. [#47199](https://github.com/PaddlePaddle/Paddle/pull/47199) - - Fix the bug that the performance of the new dynamic graph MP/PP policy is lower than the old dynamic graph. [#47071](https://github.com/PaddlePaddle/Paddle/pull/47071) - - Fix the bug that the shardingstage2 policy incorrectly maintains the parameter trainable property. [#47240](https://github.com/PaddlePaddle/Paddle/pull/47240) - - Fix the bug that tensornumel is greater than INT32_MAX in series of OPs. [#45711](https://github.com/PaddlePaddle/Paddle/pull/45711), [#45741](https://github.com/PaddlePaddle/Paddle/pull/45741), [#45897](https://github.com/PaddlePaddle/Paddle/pull/45897), [#46158](https://github.com/PaddlePaddle/Paddle/pull/46158), [#46767](https://github.com/PaddlePaddle/Paddle/pull/46767), [#47191](https://github.com/PaddlePaddle/Paddle/pull/47191), [#46045](https://github.com/PaddlePaddle/Paddle/pull/46045), [#46160](https://github.com/PaddlePaddle/Paddle/pull/46160) - - Fix the bug of too much video memory footprint in FusedAttention and Fused FeedForward OP.[#47236](https://github.com/PaddlePaddle/Paddle/pull/47236), [#47235](https://github.com/PaddlePaddle/Paddle/pull/47235) - - Fix the bug of incorrect parameter update in multi_tensor_adam and multi_tensor_momentumOP when the parameters passed in are listofdict. [#47352](https://github.com/PaddlePaddle/Paddle/pull/47352), [#47372](https://github.com/PaddlePaddle/Paddle/pull/47372) - -## 4. Deployment direction (Paddle Inference) - -### （1）New features - -- Optimize the back-end graph engine integration scheme - - In order to reduce Paddle-TensorRT plugin code development and reduce the number of Paddle-TensorRT subgraphs and thus reducing resource usage, a generic plugin mechanism has been developed, to automatically provide a unified TensorRT plugin interface for rich Phi operators in the framework. As a result, the video memory footprint can be effectively reduced in most scenarios. [#46970](https://github.com/PaddlePaddle/Paddle/pull/46070), [#46179](https://github.com/PaddlePaddle/Paddle/pull/46179), [#46580](https://github.com/PaddlePaddle/Paddle/pull/46580) - - In order to facilitate users to customize operators in the framework and make Paddle-TensorRT perform efficient inference, the function is upgraded to support the framework custom Paddle-TensorRT plugin. [#46970](https://github.com/PaddlePaddle/Paddle/pull/46070) -- Optimize the Inference library build system. The size can be pruned on demand - - Pre-compiled installer supports TensorRT by default: The pre-compiled installer for training and the pre-compiled installer for deployment (Paddle Inference) are unified into one pre-compiled installer. The build system is optimized so that the pre-compiled installer supports TensorRT by default, reducing the switching cost for users using PaddleTensorRT. [#46008](https://github.com/PaddlePaddle/Paddle/pull/46008), [#45824](https://github.com/PaddlePaddle/Paddle/pull/45824), [#46058](https://github.com/PaddlePaddle/Paddle/pull/46058) - - The size can be pruned on demand: Pruned according to the model operator. [#47033](https://github.com/PaddlePaddle/Paddle/pull/47033) , [#47049](https://github.com/PaddlePaddle/Paddle/pull/47049) , [#47047](https://github.com/PaddlePaddle/Paddle/pull/47047) -- Inference supports native AMP - - In order to make full use of GPUTensorCore computation capability and improve the model inference performance, a model accuracy conversion tool has been developed. The InferenceGPU natively supports the inference of the mixed precision model. For the usages, refer to the documentation. [documentation](https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/release/v2.4/docs-official/guides/nv_gpu_infer/gpu_mixed_precision.md), [#43814](https://github.com/PaddlePaddle/Paddle/pull/43814), [#43881](https://github.com/PaddlePaddle/Paddle/pull/43881), [#44057](https://github.com/PaddlePaddle/Paddle/pull/44057), [#44307](https://github.com/PaddlePaddle/Paddle/pull/44307), [#44457](https://github.com/PaddlePaddle/Paddle/pull/44457), [#44866](https://github.com/PaddlePaddle/Paddle/pull/44866), [#45050](https://github.com/PaddlePaddle/Paddle/pull/45050), [#45346](https://github.com/PaddlePaddle/Paddle/pull/45346), [#45379](https://github.com/PaddlePaddle/Paddle/pull/45379), [#45406](https://github.com/PaddlePaddle/Paddle/pull/45406), [#45882](https://github.com/PaddlePaddle/Paddle/pull/45882) - - In order to improve the inference performance of the mixed precision model, the FP16kernel of high-frequency operators that do not support FP16 computation is supplemented, thus reducing the possibility of inserting the cast operator due to input precision mismatch. The inference performance is improved. [#44642](https://github.com/PaddlePaddle/Paddle/pull/44642), [#45061](https://github.com/PaddlePaddle/Paddle/pull/45061), [#44653](https://github.com/PaddlePaddle/Paddle/pull/44653), [#45504](https://github.com/PaddlePaddle/Paddle/pull/45504), [#45061](https://github.com/PaddlePaddle/Paddle/pull/45061), [#44969](https://github.com/PaddlePaddle/Paddle/pull/44969), [#44558](https://github.com/PaddlePaddle/Paddle/pull/44558), [#44710](https://github.com/PaddlePaddle/Paddle/pull/44710), [#43871](https://github.com/PaddlePaddle/Paddle/pull/43871), [#44792](https://github.com/PaddlePaddle/Paddle/pull/44792) -- Upgrade the compression and inference engine - - Upgrade the quantization model storage format. The new format supports PaddleInference, PaddleLite and Paddle2ONNX 3 deployment methods. The supported chips include X86 CPU, NVIDIA GPU, and Arm CPU. （[#46305](https://github.com/PaddlePaddle/Paddle/pull/46305), [#462832](https://github.com/PaddlePaddle/Paddle/pull/46283), [#46022](https://github.com/PaddlePaddle/Paddle/pull/46022) ） - - Add the INT8 full quantization function compatible with SoC/NPU chips. This can ensure the output INT8 quantization model has the best inference acceleration and precision on SoC/NPU chips. -- Add the INT8 full quantization function compatible with SoC/NPU chips. This can ensure the output INT8 quantization model has the best inference acceleration and precision on SoC/NPU chips. - - Upgrade the interface module between the PaddlePaddle framework and compiler, to support inference models to access the compiler for optimization via Paddle Inference. （[#44499](https://github.com/PaddlePaddle/Paddle/pull/44499) [#44708](https://github.com/PaddlePaddle/Paddle/pull/44708) ） - -### （2）Underlying optimization - -- **GPU performance optimization** - - Add the TensorRT mapping for operators such as matmul_v2, LSTM, reshape, fill_constant, swish, mulitclass_nms3, bilinear_interp_v2, split, silu, shuffle_channel operators. Optimize the support for the dynamic shape. Performance improved by 7% to 90% for multi-class focused models. ([#46177](https://github.com/PaddlePaddle/Paddle/pull/46177), [#44678](https://github.com/PaddlePaddle/Paddle/pull/44678), [#44314](https://github.com/PaddlePaddle/Paddle/pull/44314), [#44561](https://github.com/PaddlePaddle/Paddle/pull/44561), [#45166](https://github.com/PaddlePaddle/Paddle/pull/45166), [#44411](https://github.com/PaddlePaddle/Paddle/pull/44411), [#43424](https://github.com/PaddlePaddle/Paddle/pull/43424), [#44516](https://github.com/PaddlePaddle/Paddle/pull/44516)) - - Add constant folding PASS for inference performance optimization, to improve the performance of SwinTransformer, HifiGAN, FastSpeech2, and other models.（[#45494](https://github.com/PaddlePaddle/Paddle/pull/45494)) - - Add cache of conv_fusionworkspacesize, to improve the computation performance of conv_fusion. ([#45902](https://github.com/PaddlePaddle/Paddle/pull/45902)) -- **Vision ViT model optimization** - - Add the ViT model Attention structure fusion PASS, and support OSSPlugin and auto padding. The ViT inference speed increases by 30%-40%. [#45019](https://github.com/PaddlePaddle/Paddle/pull/45019) [#45506](https://github.com/PaddlePaddle/Paddle/pull/45506) -- **Inference performance optimization of large model** - - To improve the inference speed of very large generative models and save the video memory, add INT8 implementation (fused_multi_transformer_int8_op) to the multi-layer Transformer fusion operator (fused_multi_transformer_op), and support quantized inference of generative models. Use the matrix multiplication algorithm to select, quantize/de-quantize the kernel fusion for performance optimization. [#46169](https://github.com/PaddlePaddle/Paddle/pull/46169) - - Add Pass for automatic matching fusion in order to improve the ease of use of fused_multi_transformer fusion for large model inference. -- **CPU performance optimization** - - Optimize the speech U2++ model. The FP32 model inference speed is improved by 35%. The INT8 model inference speed is improved by 69%. ([#47592](https://github.com/PaddlePaddle/Paddle/pull/47592), [#47127](https://github.com/PaddlePaddle/Paddle/pull/47127), [#47391](https://github.com/PaddlePaddle/Paddle/pull/47391), [#47234](https://github.com/PaddlePaddle/Paddle/pull/47234), [#47009](https://github.com/PaddlePaddle/Paddle/pull/47009), [#47080](https://github.com/PaddlePaddle/Paddle/pull/47080)) - - -### （3）Bug fix - -- TensorRT workspace size supports int64. （[#44469](https://github.com/PaddlePaddle/Paddle/pull/44469) ） -- In Paddle-TRT, fully support Op's input as weight.（[#45545](https://github.com/PaddlePaddle/Paddle/pull/45545) ） -- In Paddle-TRT, support conv2d_transpose/conv3d_transpose to have the output_padding attribute.（[#45004](https://github.com/PaddlePaddle/Paddle/pull/45004) ） -- In Paddle-TRT, enhance the strided_slice support for dynamic shape. （[#46819](https://github.com/PaddlePaddle/Paddle/pull/46819) ） -- In Paddle-TRT, optimize the video memory footprint of context when running in multi-thread scenarios.（[#45468](https://github.com/PaddlePaddle/Paddle/pull/45468) ） -- In Paddle-TRT, fix the bug of repeatedly generating serialization files in case of change of initialization sequences when multiple models run in the same process.（[#43942](https://github.com/PaddlePaddle/Paddle/pull/43942) ） -- Fix the bug of occasional crash when Predictor is initialized to run for multiple times in the same process.（[#45203](https://github.com/PaddlePaddle/Paddle/pull/45203) ） -- Fix the bug of abnormal inference accuracy of quantization models such as MobileNetV3_large, ERNIE 3.0-Medium and bert ([#45416](https://github.com/PaddlePaddle/Paddle/pull/45416), [#46283](https://github.com/PaddlePaddle/Paddle/pull/46283), [#45920](https://github.com/PaddlePaddle/Paddle/pull/45920) [#47573](https://github.com/PaddlePaddle/Paddle/pull/47574)) - -## 5. Environment adaptation - -- The pre-compiled installer for training and the pre-compiled installer for deployment (Paddle Inference) are unified into one pre-compiled installer. The build system is optimized so that the pre-compiled installer supports TensorRT by default. -- The pre-compiled installer for CUDA version 10.1 is cancelled. -- Add the pre-compiled installer for CUDA 11.7. -- Decrease of source code compilation time: Reduce inter-module dependencies, improve the parallel, and optimize the compilation speed of some modules. The full compilation time is reduced by about 20 minutes in total. -- Support the running of PaddlePaddle on windows 11, Centos 8, Ubuntu 22.04, Jetson 5.02 system environment. Support to run PaddlePaddle linux installer in windows system by using the WSL 2 tool. -- Fix the running error bug of the PaddlePaddle in glibc2.34+ environment. -- Optimize the code style of C++, Python, CMake in the whole code repository. Introduce or upgrade the following code style checking tools. - - pre-commit is upgraded from 1.10.4 to 2.17.0： [#43103](https://github.com/PaddlePaddle/Paddle/pull/43103) - - pylint is changed from default version to specify as： [#43103](https://github.com/PaddlePaddle/Paddle/pull/43103) - - remove-crlf is upgraded from 1.0.1 to 1.1.14 ： [#43103](https://github.com/PaddlePaddle/Paddle/pull/43103) - - cpplint is changed from default version to specify as 1.6.0 ： [#43175](https://github.com/PaddlePaddle/Paddle/pull/43175), [#43978](https://github.com/PaddlePaddle/Paddle/pull/43978), [#43673](https://github.com/PaddlePaddle/Paddle/pull/43673), [#43679](https://github.com/PaddlePaddle/Paddle/pull/43679), [#43695](https://github.com/PaddlePaddle/Paddle/pull/43695), [#43733](https://github.com/PaddlePaddle/Paddle/pull/43733), [#43740](https://github.com/PaddlePaddle/Paddle/pull/43740) - - clang-format is upgrade from 3.8 to 13.0 ： [#42840](https://github.com/PaddlePaddle/Paddle/pull/42840), [#43248](https://github.com/PaddlePaddle/Paddle/pull/43248), [#43329](https://github.com/PaddlePaddle/Paddle/pull/43329), [#43333](https://github.com/PaddlePaddle/Paddle/pull/43333), [#43633](https://github.com/PaddlePaddle/Paddle/pull/43633), [#43678](https://github.com/PaddlePaddle/Paddle/pull/43678) - - Introduce the black tool for python code style checking ：[#46014](https://github.com/PaddlePaddle/Paddle/pull/46014) - - Introduce the cmakelint tool for cmake file code checking. Version is 1.4.2 ： [#43222](https://github.com/PaddlePaddle/Paddle/pull/43222), [#43406](https://github.com/PaddlePaddle/Paddle/pull/43406), [#43414](https://github.com/PaddlePaddle/Paddle/pull/43414), [#43428](https://github.com/PaddlePaddle/Paddle/pull/43428) - - Introduce cmake-format for automatic formatting of cmake files. Version is 0.6.13 ： [#43057](https://github.com/PaddlePaddle/Paddle/pull/43057) - -## 6. Hardware adaptation ### Hygon DCU -- Add the Profiler function on DCU, to collect, count and display performance data of model running process on DCU, and support DCU occupancy display at kernel level. -### Kunlunxin Chip -- Add Profiler function on Kunlunxin 2 generation chip, which can collect, count and display the performance data of model running process on Kunlunxin 2 generation chip, and support occupancy display of Kunlunxin 2 generation chip at kernel level. -- Training/reasoning support for Kunlunxin 2 generation chips (Kunlunxin AI accelerator cards R200, R300, R200-8F, R200-8FS, RG800), a total of 51 models such as PPYOLOE, PP-OCR, ERNIE3.0, PP-TSM, PP-TTS, DLRM, PPO, etc. have been verified, supporting static graph + dynamic graph training, supporting mixed precision training, support single machine single card and single machine multi-card training, covering 5 fields of intelligent vision, natural language processing, intelligent speech, intelligent recommendation, reinforcement learning. -### Cambricon -- Support the training/inference of Cambricon MLU chip (MLU370 series of boards): The ResNet50, BERT, YoloV3, OCR-DB, Deeplabv3 and many other models are verified. Support the static graph + dynamic graph training. Support mixed precision training. Support the single machine single card and single machine multi-card training. -### Graphcore -- Support the training/inference of Graphcore IPU chip (including IPU Mk2 GC200 and Bow IPU). Support ResNet50, BERT and other models. Support the static graph and dynamic-to-static graph mode training. Support the single chip, single machine, and multi-machine distributed training. -- Add the support of more operators -- Upgrade to Poplar SDK v3.0.0 [#46892](https://github.com/PaddlePaddle/Paddle/pull/46892) -* Support the training models by using the dynamic-to-static graph mode. Add a new paddle.incubate.identity_loss op to assist with composition [#43770](https://github.com/PaddlePaddle/Paddle/pull/43770) -* Support the Paddle native distributed training API: paddle.distributed.launch [#43311](https://github.com/PaddlePaddle/Paddle/pull/43311) -* Support the training models with the mixed precision [#41733](https://github.com/PaddlePaddle/Paddle/pull/41733) -* Paddle Inference supports custom operators by using PopART [#45235](https://github.com/PaddlePaddle/Paddle/pull/45235) - -### Intel -- Migrate oneDNN operators : transpose2_grad([#46139](https://github.com/PaddlePaddle/Paddle/pull/46139)), relu6_grad([#46501](https://github.com/PaddlePaddle/Paddle/pull/46501)), gaussian_random([#46747](https://github.com/PaddlePaddle/Paddle/pull/46747), [#45481](https://github.com/PaddlePaddle/Paddle/pull/45481)), sgd and stack([#46374](https://github.com/PaddlePaddle/Paddle/pull/46374)), concat+grad, expand+grad,fill_constant([#45863](https://github.com/PaddlePaddle/Paddle/pull/45863)), slice, slice_grad, split,pad and pad3d([#46101](https://github.com/PaddlePaddle/Paddle/pull/46101)), softmax_grad([#46257](https://github.com/PaddlePaddle/Paddle/pull/46257)), Shape([#46051](https://github.com/PaddlePaddle/Paddle/pull/46051)), Sum([#46239](https://github.com/PaddlePaddle/Paddle/pull/46239)), Transpose2_grad([#46139](https://github.com/PaddlePaddle/Paddle/pull/46139)), Cast, clip+grad andpool+grad([#45775](https://github.com/PaddlePaddle/Paddle/pull/45775)), Reduce sum+grad,mean+grad, min and max([#45536](https://github.com/PaddlePaddle/Paddle/pull/45536)), Relu and abs([#45397](https://github.com/PaddlePaddle/Paddle/pull/45397)), Gelu([#45596](https://github.com/PaddlePaddle/Paddle/pull/45596)), Scale([#45537](https://github.com/PaddlePaddle/Paddle/pull/45537)) -- Optimize kernels of fill_constant, fc, conv, and a number of operators -- Add several Pass fusion optimizations -- Optimize the Adam-W CPU FP32 optimizer ([#42522](https://github.com/PaddlePaddle/Paddle/pull/42522)) -- Optimize pad3d fp32 onednn operator kernel implementation ([#43990](https://github.com/PaddlePaddle/Paddle/pull/43990)) -- Optimize the concurrent execution of matmul, FC andlookup_v2 kernels ([#44023](https://github.com/PaddlePaddle/Paddle/pull/44023), [#44078](https://github.com/PaddlePaddle/Paddle/pull/444078), [#44640](https://github.com/PaddlePaddle/Paddle/pull/44640), [#44744](https://github.com/PaddlePaddle/Paddle/pull/44744), [#45249](https://github.com/PaddlePaddle/Paddle/pull/45249)) -- FC onednn operator kernel supports bf16 ( [#42758](https://github.com/PaddlePaddle/Paddle/pull/42758), [#43154](https://github.com/PaddlePaddle/Paddle/pull/43154), [#43109](https://github.com/PaddlePaddle/Paddle/pull/43109)) -- Add the fusion of matrix multiplication and activation functions ([#43519](https://github.com/PaddlePaddle/Paddle/pull/43519), [#43198](https://github.com/PaddlePaddle/Paddle/pull/43198)) -- Support convolution operator int8 parameter production IR passes ( [#44680](https://github.com/PaddlePaddle/Paddle/pull/44680), [#42625](https://github.com/PaddlePaddle/Paddle/pull/42625)) -- Add pool/avg quantization and scales correction ([#44186](https://github.com/PaddlePaddle/Paddle/pull/44186)) -- Add the matmul and elementwise onednn operator kernel fusion ([#45077](https://github.com/PaddlePaddle/Paddle/pull/45077)) -- Fix the QAT precision bug ([#43693](https://github.com/PaddlePaddle/Paddle/pull/43693), [#45936](https://github.com/PaddlePaddle/Paddle/pull/45936), [#46378](https://github.com/PaddlePaddle/Paddle/pull/46378)) -- Migrate 42 oneDNN operator kernels to PHI operator library ([#46374](https://github.com/PaddlePaddle/Paddle/pull/46374), [#46101](https://github.com/PaddlePaddle/Paddle/pull/46101), [#45989](https://github.com/PaddlePaddle/Paddle/pull/45989), [#45863](https://github.com/PaddlePaddle/Paddle/pull/45863), [#45775](https://github.com/PaddlePaddle/Paddle/pull/45775), [#45626](https://github.com/PaddlePaddle/Paddle/pull/45626), [#45536](https://github.com/PaddlePaddle/Paddle/pull/45536), [#46501](https://github.com/PaddlePaddle/Paddle/pull/46501), [#46257](https://github.com/PaddlePaddle/Paddle/pull/46257), [#45596](https://github.com/PaddlePaddle/Paddle/pull/45596), [#45537](https://github.com/PaddlePaddle/Paddle/pull/45537), [#45481](https://github.com/PaddlePaddle/Paddle/pull/45481), [#45397](https://github.com/PaddlePaddle/Paddle/pull/45397), [#46239](https://github.com/PaddlePaddle/Paddle/pull/46239), [#46139](https://github.com/PaddlePaddle/Paddle/pull/46139), [#46051](https://github.com/PaddlePaddle/Paddle/pull/46051)) -- Quantize the elementwise_sub and shape operator kernels ([#42854](https://github.com/PaddlePaddle/Paddle/pull/42854), [#44124](https://github.com/PaddlePaddle/Paddle/pull/44124)) - -## Thanks to our Contributors - -This release contains contributions from: - -0x45f, Aganlengzi, Ainavo, Allen Guo, Asthestarsfalll, Aurelius84, Baibaifan, baoachun, BiynXu, Bo Zhang, BrilliantYuKaimin, cambriconhsq, caozhou, carryyu, ccrrong, ceci3, chalsliu, Chang Xu, Charles-hit, Chen Long, Chen Weihang, chenjian, chentianyu03, Chenxiao Niu, cifar10, crystal, csy0225, danleifeng, David Nicolas, dc-cheny, denglin-github, dongfangshenzhu, duanboqiang, duanyanhui, engineer, enzodechine, Fan Zhang, feifei-111, Feiyu Chan, Feng Ni, feng_shuai, FlyingQianMM, freeliuzc, furnace, fuyou765, fwenguang, Ghost Screaming, gongweibao, Guanghua Yu, guguguzi, Guoxia Wang, Haipeng Wang, handiz, Haohongxiang, haosicheng, helen88, heliqi, hong, HongyuJia, houj04, huangxu96, Hui Zhang, Huihuang Zheng, huzhiqiang, Jacek Czaja, Jack Zhou, jack603047588, Jackwaterveg, jakpiase, james, Jiabin Yang, jiangcheng, Jiaqi Liu, JingZhuangzhuang, joanna.wozna.intel, JYChen, JZ-LIANG, Kaipeng Deng, kangguangli, kuizhiqing, Leo Chen, Leo Guo, levi131, Li Min, Li-fAngyU, lidanqing, LielinJiang, Ligoml, Lijunhui, lilong12, limingshu, Lin Manhui, Linjie Chen, liqitong-a, littletomatodonkey, liu zhengxi, Liu-xiandong, liutiexing, Liyulingyue, LiYuRio, Lux et Veritas, lyq, Matsumoto Ruko, MayYouBeProsperous, mengqingchun02, Ming-Xu Huang, ming1753, minghaoBD, moyan, mrcangye, Netpunk, niuliling123, Nyakku Shigure, OccupyMars2025, onecatcn, pangyoki, parap1uie-s, peachlcy, piotrekobi, Qi Li, QingshuChen, qipengh, Rayman, Regan Yue, RichardWooSJTU, risemeup1, Roc, ronnywang, Rui Li, Ruibiao Chen, seemingwang, Shang Zhizhou, shangliang Xu, ShenLiang, shentanyue, Shijie, ShiningZhang, shixingbo, shiyutang, Shuangchi He, Siming Dai, Sing_chan, Skr Bang, SmirnovKol, sneaxiy, sprouteer, Sylwester Fraczek, Sławomir Siwek, taixiurong, Tao CHANG, TeFeng Chen, Thomas Young, thunder95, Thunderbrook, tiancaishaonvjituizi, tianshuo78520a, Tomasz Socha, TTerror, USTCKAY, Vigi Zhang, Walter, Wang Bojun, wangguanqun, wangguanzhong, wanghuancoder, wangna11BD, WangXi, wangxinxin08, Wangzheee, WangZhen, wangzhen38, wawltor, wbn, Wei Shengyu, Weilong Wu, weishengying, Wen Sun, wenbin, whs, Wilber, WJJ1995, wuhuachaocoding, wuhuanzhou, wuyefeilin, XiaoguangHu, xiaoguoguo626807, xiaohemaikoo, xiaoting, xiaoxiaohehe001, Xiaoxu Chen, xiayanming, Xingyuan Zhang, xiongkun, yang131313, yangguohao, YangZhou, Yanxing Shi, Yao Zihang, yaoxuefeng, yaozhixin, yeliang2258, Yilingyelu, Yiqun Liu, ykkk2333, Yuang Liu, Yuanle Liu, YuanRisheng, yuguo, Yulong Ao, Yulv-git, YUNSHEN XIE, Zhang Jun, Zhang Ting, Zhang Zheng, zhangbo9674, zhangbopd, zhangchunle, Zhangjingyu06, zhangkaihuo, zhangxiaoci, zhangyikun02, zhangzhenguo, Zhanlue Yang, zhaocaibei123, zhaoying9105, zhaoyingli, Zhen Wang, Zhengyang Song, zhiboniu, Zhong Hui, Zhou Wei, zhoutianzi666, zhupengyang, ziyoujiyi, zlsh80826, zmxdream, zn, Zuza Gawrysiak, zyfncg, 傅剑寒, 六个骨头, 津, 熊峻峰, 王明冬, 石晓伟 - -# 2.3.1 Release Note - -## **1. Important Updates** - -- V2.3.1 is built on V2.3 by fixing known issues and releasing precompiled binary that supports CUDA 11.6. - -## **2. Training Framework (distributed included)** - -### **(1) Function Optimization** - -#### API - -- Modify two initialization modes of `paddle.nn.initializer.KaimingUniform` and `paddle.nn.initializer.KaimingNormal`, to support multiple types of activation functions. ([#43721](https://github.com/PaddlePaddle/Paddle/pull/43721), [#43827](https://github.com/PaddlePaddle/Paddle/pull/43827)) -- Optimize the data pre-fetching function of `paddle.io.DataLoader`, so that it can support the setting of the `prefetch_factor` to set the cache size of pre-fetched data. This can avoid IO blocking when reading large blocks of data. ([#43674](https://github.com/PaddlePaddle/Paddle/pull/43674)) - -#### **New dynamic graph execution mechanism** - -- Modify the initialization method of optional type Tensor in the new dynamic graph API logic to prevent data exceptions caused by early destruction. ([#42561](https://github.com/PaddlePaddle/Paddle/pull/42561)) - -#### **New static graph executor** - -- Defer initialization of the thread pools in the executor, to avoid creating thread pools for `programs` that execute only once (e.g.,`save, load, startup_program`, etc.). ([#43768](https://github.com/PaddlePaddle/Paddle/pull/43768)) - -#### **Mixed precision training** - -- Disabling `state_dict` hook in `set_state_dict` in `paddle.nn.Layer`. ([#43407](https://github.com/PaddlePaddle/Paddle/pull/43407)) - -#### **Distributed training** - -- Enabling tensor parallelism in `paddle.incubate.nn.functional.fused_attention` and `paddle.incubate.nn.functional.fused_feedforward`. ([#43505](https://github.com/PaddlePaddle/Paddle/pull/43505)) - -#### **Others** - -- Adjust print format of the framework operator kernels to facilitate automated splitting and parsing. ([#42931](https://github.com/PaddlePaddle/Paddle/pull/42931)) -- Update the model quantization API to support the round-off in `rounding to nearest ties to even`, and support quantization in the range [-128, 127]. ([#43829](https://github.com/PaddlePaddle/Paddle/pull/43829)) -- Support AMP mixed precision training in quantization-aware training. ([#43689](https://github.com/PaddlePaddle/Paddle/pull/43689)) -- Add the `progress bar` at the beginning of quantization-aware training, so that it is easy to check the progress of quantization initialization. Skip the scale op when counting out_threshold to speed up the initialization process. ([#43454](https://github.com/PaddlePaddle/Paddle/pull/43454)) -- Support `conv` and `bn` fusion in the dynamic graph quantization training. Support the settings of skip_tensor_list in the static graph offline quantization, to skip some layers without quantization. ([#43301](https://github.com/PaddlePaddle/Paddle/pull/43301)) - -### **(2) Performance Optimization** - -- Optimize`paddle.incubate.nn.functional.fused_attention` and `paddle.incubate.nn.functional.fused_feedforward`operators. Add `add_residual` property to control whether to perform add-`residual` operation in the last step. The performance of CAE model is improved by 7.7%. ([#43719](https://github.com/PaddlePaddle/Paddle/pull/43719)) -- Optimize `linspace` operator. Initialize three input Tensor of `start`,`stop` and `num` on CPU, to avoid GPU->CPU copy in the operator. This can speed up SOLOv2 model performance by 6%. ([#43746](https://github.com/PaddlePaddle/Paddle/pull/43746)) - -### **(3) Bug Fix** - -#### API - -- Fix the error reported by `paddle.io.DataLoader` when `return_list=True` due to multi-thread conflict. ([#43691](https://github.com/PaddlePaddle/Paddle/pull/43691)) -- Fix the error that the `to` method reports NoneType does not have the device attribute when the `paddle.nn.Layer` parameter has the `None` type parameter. ([#43597](https://github.com/PaddlePaddle/Paddle/pull/43597)) -- Fix the bug that the calculation result of cumsum op is wrong in some `shape` settings. ([#42500](https://github.com/PaddlePaddle/Paddle/pull/42500), [#43777](https://github.com/PaddlePaddle/Paddle/pull/43777)) -- Fix the bug that the output result dimension of `Tensor.__getitem__` is 0 in the networking stage when using `bool` index in the static graph. ([#43246](https://github.com/PaddlePaddle/Paddle/pull/43246)) -- Fix the bug occurred when `paddle.slice` and `paddle.strided_slice` handle negative parameters. ([#43432](https://github.com/PaddlePaddle/Paddle/pull/43432)) -- Fix the bug that the assignment result of set_value op is abnormal when the processing slice `step` is negative. ([#43694](https://github.com/PaddlePaddle/Paddle/pull/43694)) -- Fix the bug that the `copy` interface in C++ cannot copy between multiple cards. ([#43728](https://github.com/PaddlePaddle/Paddle/pull/43728)) -- Fix the bug in inference stage caused by attribute naming in `paddle.incubate.nn.functional.fused_attention`and `paddle.incubate.nn.functional.fused_feedforward`. ([#43505](https://github.com/PaddlePaddle/Paddle/pull/43505)) -- Fix an exception in ConditionalBlockGrad op when processing Tensor that does not require `grad`. ([#43034](https://github.com/PaddlePaddle/Paddle/pull/43034)) -- Fix the bug of device memory increase caused by einsum op in the speed optimization of backward computation. By default, this optimization is enabled. ([#43397](https://github.com/PaddlePaddle/Paddle/pull/43397)) -- Fix the bug that data fails to be fixed when `paddle.io.DataLoader` multi-process data reads the fixing random seeds under a single card. ([#43702](https://github.com/PaddlePaddle/Paddle/pull/43702)) -- Fix the bug that softmax op triggers CUDNN_STATUS_NOT_SUPPORT when the Tensor exceeds 2G. ([#43719](https://github.com/PaddlePaddle/Paddle/pull/43719)) -- Fix the bug that the trace op `Event` string is indistinguishable among different operators that cause the inconvenient performance analysis. ([#42789](https://github.com/PaddlePaddle/Paddle/pull/42789)) - -#### **Others** - -- Fix the bug of overflowing device memory caused by multiple deepcopy and saving in case of dynamic-to-static. ([#43141](https://github.com/PaddlePaddle/Paddle/pull/43141)) -- Fix the bug that the device id introduced by the upgrade of PlaceType used in the custom operator is wrong in the multi-card scenario. ([#43830](https://github.com/PaddlePaddle/Paddle/pull/43830)) -- Optimize the `paddle.profiler.Profiler` timeline visualization logic, move events customized in python scripts from C++ folding display to python folding display. ([#42790](https://github.com/PaddlePaddle/Paddle/pull/42790)) - -## **3.** Deployment Direction (Paddle Inference) - -### **(1) New Features** - -#### **New functions** - -- Add the support of the PaddleSlim quantization model for ONNX Runtime backends on CPUs. ([#43774](https://github.com/PaddlePaddle/Paddle/pull/43774), [#43796](https://github.com/PaddlePaddle/Paddle/pull/43796)) - -### **(2) Underlying Optimization** - -#### **CPU performance optimization** - -- Remove `gpu_cpu_reshape2_matmul_fuse_pass` from EnableMkldnn configuration to fix the bug of ResNet50 performance degradation. ([#43750](https://github.com/PaddlePaddle/Paddle/pull/43750)) - -#### **GPU performance optimization** - -- Add the support of `bilinear_interp_v2` TensorRT convert. ([#43618](https://github.com/PaddlePaddle/Paddle/pull/43618)) -- Add `matmul_scale_fuse_pass` and `multihead_matmul_fuse_pass_v3` to GPU pass. ([#43765](https://github.com/PaddlePaddle/Paddle/pull/43765)) -- Add the support of the GPU handle deferred initialization. ([#43661](https://github.com/PaddlePaddle/Paddle/pull/43661)) - -### **(3) Bug Fixing** - -#### **Framework and API fixing** - -- Fix the compile error problem when binding Paddle-Lite XPU. ([#43178](https://github.com/PaddlePaddle/Paddle/pull/43178)) -- Fix the bug of false trigger of ERNIE 3.0 pass. ([#43948](https://github.com/PaddlePaddle/Paddle/pull/43948)) -- Fix the bug that int8 quantization attribute in multihead op cannot be read. ([#43020](https://github.com/PaddlePaddle/Paddle/pull/43020)) - -#### **Backend capability fixing** - -- Fix the bug that two ops of elementwise_mul and matmul in MKLDNN are crashed during quantitative inference. ([#43725](https://github.com/PaddlePaddle/Paddle/pull/43725)) -- Fix a bug where TensorRT subgraph serialization files are repeatedly generated for the same model during inference. ([#42945](https://github.com/PaddlePaddle/Paddle/pull/43945), [#42633](https://github.com/PaddlePaddle/Paddle/pull/42633)) -- Fix a conflict between the ONNX Runtime backend and the externally use of protobuf. ([#43159](https://github.com/PaddlePaddle/Paddle/pull/43159), [#43742](https://github.com/PaddlePaddle/Paddle/pull/43742)) -- Fix an error reported by python prediction library when using ONNX Runtime backend in case of multiple inputs. ([#43621](https://github.com/PaddlePaddle/Paddle/pull/43621)) - -## **4. Environment Adaptation** - -### **Compile and install** - -- Complete verification and adaptation of CUDA 11.6, and release CUDA 11.6 precompiled binary. ([#43935](https://github.com/PaddlePaddle/Paddle/pull/43935), [#44005](https://github.com/PaddlePaddle/Paddle/pull/44005)) -- Fix a cub error when compiling with CUDA 11.6 on Windows. ([#43935](https://github.com/PaddlePaddle/Paddle/pull/43935), [#44005](https://github.com/PaddlePaddle/Paddle/pull/44005)) -- Fix the bug of long compilation time for elementwise and reduce op. ([#43202](https://github.com/PaddlePaddle/Paddle/pull/43202), [#42779](https://github.com/PaddlePaddle/Paddle/pull/42779), [#43205](https://github.com/PaddlePaddle/Paddle/pull/43205)) - -### **New hardware adaptation** - -- Cambricon MLU supports PaddlePaddle Profiler. ([#42115](https://github.com/PaddlePaddle/Paddle/pull/42115)) -- GraphCore IPU supports visualization of compilation progress. ([#42078](https://github.com/PaddlePaddle/Paddle/pull/42078)) - -# 2.3.0 Release Note - -## 1. **Important Updates** - -We are excited to release the PaddlePaddle Framework V2.3.0. This version contains the following highlights. - -### API - -- Added more than 100 new APIs, covering automatic differentiation, linear algebra, probability distribution, sparse tensor, framework performance analysis, hardware device management, vision domain, etc. - -- Added 4 new automatic differentiation APIs, 11 new linear algebra APIs, and 21 new probability distribution APIs to better support use cases in scientific computing, reinforcement learning, xand other application areas. - -- Added 11 new Sparse Tensor APIs including basic functions of sparse tensor construction and conversion. The COO and CSR formats are supported. - -- Added 9 new framework performance analysis APIs. The new performance profiling APIs, centered around Paddle.Profiler.Profiler, help users collect and analyze performance statistics during training and inference. - -- Added 7 APIs for device management, facilitating hardware information acquistion. - -- Added several visual and text domain APIs to facilitate ~~the~~ reusability of MobileNetV3, ResNeXt and other backbone networks, to achieve the fast networking. - - -### **Paddle** HIgh reusability operator l**ibrary** - -- We announce PHI as the new Paddle HIgh reusability operator library. PHI provides Primitive API, enabling kernel reuse for operator development. As a refactored functional operator library, PHI aims to solve legacy problems that harm the framework's performance and reusability, in particular on the operator development. Such problems include inefficient ways of cross using operators, unclear operator interfaces and lacking direct calls to the operator library in C++. With PHI, new operators can be easily implemented by composing functions available in the functional library. The library provides over 200 C++ operator class APIs and nearly 500 kernels. Composing new operators through these built-in functions can greatly reduce the user's development effort. PHI supports different types of hardware (e.g., GPU and XPU). In addition, PHI is extensible with plugins for accommodating third party accelerators (such as NPU) in a low cost and reusable fashion. In short, PHI supports low level operator composability, the reuse of kernels through Primitives, and accelerators through plugins. - -### **Distributed Training** - -- Fully upgrade the adaptive distributed training architecture, including multiple modules such as elastic resource management, asynchronous pipelined executor, heterogeneous communication, and automatic parallelism, and support the hard-aware distributed training and inference under a variety of heterogeneous hardware. - -- Add MoE parallel strategy, GroupSharded parallel strategy, and Pure FP16 under dynamic graph hybrid Parallelism, which further supports the efficient distributed training of large models under the dynamic graph. - -- Comprehensively upgrade and optimize the architecture of general heterogeneous parameter server, and simplify each module, such as communication and storage, to improve the secondary development experience of parameter server. The performance of GPU parameter server is improved by 2.38 times under 100 billion parameters and 10 billion data. - - -### **Compile and Install** - -- From version 2.3.0, PaddlePaddle upgrades GPU architectures supported. - - -### **Inference Deployment** - -- Add the Java API and ONNX Runtime CPU backend. - -- Support the TensorRT 8.0 / 8.2 and structured sparsity, with deep performance optimization for ERNIE-like structural models. - - -### **Hardware Backend Extention** - -- Add custom device support: provide a plug-in way to extend PaddlePaddle hardware backend. - -- Add training/inference support for multiple heterogeneous chips such as HUAWEI Ascend 910 / GraphCore IPU / Cambricon MLU / KUNLUNXIN 2. - - -### **Framework Architecture** - -- In this version, we did a lot of work on the framework executor. For details, please see [New Dynamic Graph Execution Mechanism](#new-dynamic-graph-execution-mechanism) and [New Static Graph Executor](#new-static-graph-executor). - -## **2. Incompatibility Upgrade** - -- Due to limitation of the binary size, sm35 CUDA ARCH is dropped in pre-compiled binaries. ([#41754](https://github.com/PaddlePaddle/Paddle/pull/41754)) - -- When `paddle.to_tensor` converts a python int scalar to a Tensor, the default data type on Windows changes from int32 to int64, thus alignment with Linux/Mac. ([#39662](https://github.com/PaddlePaddle/Paddle/pull/39662)) - -- To keep consistency with division behavior under python3, the division symbol `/` has been changed from “rounding divide” to “true divide”, and the data type of the computed output has been switched from int to float. ([#40890](https://github.com/PaddlePaddle/Paddle/pull/40890)) - - - - - - - - - - - - -
-2.2 - -2.3.0 -
-
-
-```python
->>> import paddle
->>> a = paddle.to_tensor([327])
->>> b = paddle.to_tensor([80])
->>> a / b
-Tensor(shape=[1], dtype=int64, place=CUDAPlace(0), stop_gradient=True,
-      [4])
-```
-
-		 -
-
-```python
->>> import paddle
->>> a = paddle.to_tensor([327])
->>> b = paddle.to_tensor([80])
->>> a / b
-Tensor(shape=[1], dtype=float32, place=Place(gpu:0), stop_gradient=True,
-      [4.08750010])
-```
-
-
- -- Revise the ELU's formula. The computing method in case of alpha <0 aligns with the original paper, thus fixing a small number of cases where the results are incorrectly calculated. Meanwhile, elu_ will report an error in case of alpha <0, because it is not mathematically possible to compute the inverse gradient from the output only at alpha <0. ([#37316](https://github.com/PaddlePaddle/Paddle/pull/37316)) - - - - - - - - - - - -
-2.2 - -2.3.0 -
-
-
-```python
-# elu(x) = max(0, x) + min(0, α ∗ (e^x − 1))
->>> import paddle
->>> x = paddle.to_tensor([-1., 6.])
->>> m = paddle.nn.ELU(-0.2)
->>> out = m(x)
->>> out
-Tensor(shape=[2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
-       [ 0.        , -74.48576355])
->>> out = paddle.nn.functional.elu_(x, alpha=-0.2, name=None)
->>> out
-Tensor(shape=[2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
-       [ 0.        , -74.48576355])
-```
-
-		 -
-
-```python
-# elu(x) = x, if x > 0
-# elu(x) = α ∗ (e^x − 1), if x <= 0
->>> import paddle
->>> x = paddle.to_tensor([-1., 6.])
->>> m = paddle.nn.ELU(-0.2)
->>> out = m(x)
->>> out
-Tensor(shape=[2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
-       [0.12642412,  6.        ])
->>> out = paddle.nn.functional.elu_(x, alpha=-0.2, name=None)
-Traceback (most recent call last):
-  File "", line 1, in 
-  File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun
-    return caller(func, *(extras + args), **kw)
-  File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
-    return wrapped_func(*args, **kwargs)
-  File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/inplace_utils.py", line 34, in __impl__
-    return func(*args, **kwargs)
-  File "/usr/local/lib/python3.7/dist-packages/paddle/nn/functional/activation.py", line 89, in elu_
-    assert alpha >= 0., "elu_ only support alpha >= 0, please use elu instead."
-AssertionError: elu_ only support alpha >= 0, please use elu instead.
-```
-
-
- -## **3. Training Framework (with the distributed function)** - -### **(1) New functions** - -#### API - -- Add 4 new automatic differentiation APIs to support scientific computing, as listed below: ([#40692](https://github.com/PaddlePaddle/Paddle/pull/40692)) - - - `paddle.incubate.autograd.vjp`, compute vector-Jacobi matrix product. - - - `paddle.incubate.autograd.jvp`, compute Jacobi matrix-vector product. - - - `paddle.incubate.autograd.Jacobian`, compute Jacobi matrix. - - - `paddle.incubate.autograd.Hessian`, compute Hessian matrix. - -- Add linear algebra class API - - - Add `paddle.linalg.triangular_solve`, to compute a system of linear equations with unique solutions through a triangular coefficient. ([#36714](https://github.com/PaddlePaddle/Paddle/pull/36714)) - - - Add `paddle.linalg.eig`, to compute the characteristic decomposition of the general square matrix. ([#35764](https://github.com/PaddlePaddle/Paddle/pull/35764)) - - - Add `paddle.linalg.sovle`, to compute solutions to systems of linear equations. ([#35715](https://github.com/PaddlePaddle/Paddle/pull/35715)) - - - Add `paddle.linalg.lstsq`, to compute least-squares solutions to systems of linear equations. ([#38585](https://github.com/PaddlePaddle/Paddle/pull/38585), [#38621](https://github.com/PaddlePaddle/Paddle/pull/38621)) - - - Add `paddle.linalg.qr`, compute QR decomposition of matrix. ([#35742](https://github.com/PaddlePaddle/Paddle/pull/35742), [#38824](https://github.com/PaddlePaddle/Paddle/pull/38824)) - - - Add `paddle.inner`, to compute inner product of a matrix. ([#37706](https://github.com/PaddlePaddle/Paddle/pull/37706)) - - - Add `paddle.outer`, to compute outer product of a matrix. ([#37706](https://github.com/PaddlePaddle/Paddle/pull/37706)) - - - Add `paddle.linalg.cov`, to compute covariance between vectors. ([#38392](https://github.com/PaddlePaddle/Paddle/pull/38392)) - - - Add `paddle.linalg.cholesky_sovle`, to compute the cholesky solution of the equation. ([#38167](https://github.com/PaddlePaddle/Paddle/pull/38167)) - - - Add `paddle.linalg.lu` and `paddle.linalg.lu_unpack`, to compute matrix lu decomposition, and decompress lu matrix. ([#38617](https://github.com/PaddlePaddle/Paddle/pull/38617), [#38559](https://github.com/PaddlePaddle/Paddle/pull/38559), [#38616](https://github.com/PaddlePaddle/Paddle/pull/38616)) - -- Add 21 new probability distribution class APIs for reinforcement learning, variation inference, scientific computing, and other scenarios. Including 6 random variable distributions, 13 random variable transformations, and 2 KL divergence computing. as listed below: ([#40536](https://github.com/PaddlePaddle/Paddle/pull/40536), [#38820](https://github.com/PaddlePaddle/Paddle/pull/38820), [#38558](https://github.com/PaddlePaddle/Paddle/pull/38558/files), [#38445](https://github.com/PaddlePaddle/Paddle/pull/38445), [#38244](https://github.com/PaddlePaddle/Paddle/pull/38244), [#38047](https://github.com/PaddlePaddle/Paddle/pull/38047)) - - - `paddle.distribution.ExponentialFamily`, exponential distribution family base class. - - - `paddle.distribution.Beta`, `Beta` distribution. - - - `paddle.distribution.Dirichlet`, `Dirichlet` distribution. - - - `paddle.distribution.Independent`, Independent distribution, used to create higher order distributions. - - - `paddle.distribution.TransformedDistribution`, Transform distribution, used to generate higher-order distributions through the base distribution and a series of transformations. - - - `paddle.distribution.Multionmial`, a multinomial distribution. - - - `paddle.distribution.Transform`, base class for transforming random variables. - - - `paddle.distribution.AbsTransform`, take absolute value transform. - - - `paddle.distribution.AffineTransform`, affine transform. - - - `paddle.distribution.ChainTransform`, chain combination of the transform. - - - `paddle.distribution.ExpTransform`, exponential transform. - - - `paddle.distribution.IndependentTransform`, independent transform, used to extend the `event_dim` of the transform definition field. - - - `paddle.distribution.PowerTransform`, power transform. - - - `paddle.distribution.ReshapeTransform`, `reshape` transform. - - - `paddle.distribution.SigmoidTransform`, `sigmoid` transform. - - - `paddle.distribution.SoftmaxTransform`, `softmax` transform. - - - `paddle.distribution.StackTransform`, `stack` transform, used to combine multiple transforms in a `stack` method. - - - `paddle.distribution.StickBreakingTransform`, `stickbreaking` transform. - - - `paddle.distribution.TanhTransform`, `tanh` transform. - - - `paddle.distribution.kl_divergence`, compute KL divergence. - - - `paddle.distribution.register_kl`, register user-defined KL divergence calculation function. - -- Add high-level API - - - Add `paddle.vision.models.AlexNet` and `paddle.vision.models.alexnet`, to use AlexNet models directly. ([#36058](https://github.com/PaddlePaddle/Paddle/pull/36058)) - - - Add `paddle.vision.models.DenseNet`, `paddle.vision.models.densenet121`, `paddle.vision.models.densenet161`, `paddle.vision.models. densenet169`, `paddle.vision.models.densenet201`, and `paddle.vision.models.densenet264`, to use DenseNet models directly. ([#36069](https://github.com/PaddlePaddle/Paddle/pull/36069)) - - - Add `paddle.vision.models.GoogLeNet` and `paddle.vision.models.googlenet`, to use GoogLeNet models directly. ([#36034](https://github.com/PaddlePaddle/Paddle/pull/36034)) - - - Add `paddle.vision.models.InceptionV3`, `paddle.vision.models.inception_v3`, to use InceptionV3 models directly. ([#36064](https://github.com/PaddlePaddle/Paddle/pull/36064)) - - - Add `paddle.vision.models.MobileNetV3Small`, `paddle.vision.models.MobileNetV3Large`, `paddle.vision.models.mobilenet_v3_small`, and `paddle.vision.models.mobilenet_v3_large`, to use MobileNetV3 models directly. ([#38653](https://github.com/PaddlePaddle/Paddle/pull/38653)) - - - Add `paddle.vision.models.resnext50_32x4d`, `paddle.vision.models.resnext50_64x4d`, `paddle.vision.models. paddle.vision.models.resnext101_32x4d`, `paddle.vision.models.resnext101_64x4d`, `paddle.vision.models.resnext152_32x4d`, and `paddle.vision.models.resnext152_64x4d`, to use ResNeXt models directly. ([#36070](https://github.com/PaddlePaddle/Paddle/pull/36070)) - - - Add `paddle.vision.models.ShuffleNetV2`, `paddle.vision.models.shufflenet_v2_x0_25`, `paddle.vision.models.shufflenet_v2_x0_33`, `paddle.vision.models.shufflenet_v2_x0_5`, `paddle.vision.models.shufflenet_v2_x1_0`, `paddle.vision.models.shufflenet_v2_x1_5`, `paddle.vision.models.shufflenet_v2_x2_0`, and `paddle.vision.models.shufflenet_v2_swish`, to use ShuffleNetV2 models directly ([#36067](https://github.com/PaddlePaddle/Paddle/pull/36067)) - - - Add `paddle.vision.models.SqueezeNet`, `paddle.vision.models.squeezenet1_0`, and `paddle.vision.models.squeezenet1_1`, to use SqueezeNet models directly. ([#36066](https://github.com/PaddlePaddle/Paddle/pull/36066)) - - - Add `paddle.vision.models.wide_resnet50_2`, and `paddle.vision.models.wide_resnet101_2`, to use WideResNet models directly. ([#36952](https://github.com/PaddlePaddle/Paddle/pull/36952)) - - - Add `paddle.vision.ops.nms` API, to support single-category and multi-category non-maximum suppression (NMS) algorithms for target detection and prediction task acceleration ([#40962](https://github.com/PaddlePaddle/Paddle/pull/40962)) - - - Add `paddle.vision.ops.roi_pool` and `paddle.vision.ops.RoIPool`, to support RoI region pooling operations in detection tasks. ([#36154](https://github.com/PaddlePaddle/Paddle/pull/36154)) - - - Add `paddle.vision.ops.roi_align` and `paddle.vision.ops.RoIAlign`, to support RoI Align operations in detection tasks. ([#35102](https://github.com/PaddlePaddle/Paddle/pull/36154)) - - - Add `paddle.text.ViterbiDecoder`, and `paddle.text.viterbi_decode` Viterbi decoding API, mainly for sequence tagging model prediction. ([#35778](https://github.com/PaddlePaddle/Paddle/pull/35778)) - -- Add 11 Sparse class APIs, to support basic functions, such as creating Sparse Tensor in COO and CSR formats, and add C++ inter-converting with Tensor. - - - `paddle.sparse.sparse_coo_tensor`，create Sparse Tensor in COO format. ([#40780](https://github.com/PaddlePaddle/Paddle/pull/40780)) - - - `paddle.sparse.sparse_csr_tensor`，create Sparse Tensor in CSR format. ([#40780](https://github.com/PaddlePaddle/Paddle/pull/40780)) - - - `paddle.sparse.ReLU`，support ReLU activation layer for SparseCooTensor. ([#40959](https://github.com/PaddlePaddle/Paddle/pull/40959)) - - - `paddle.sparse.functional.relu`，support ReLU function of SparseCooTensor. ([#40959](https://github.com/PaddlePaddle/Paddle/pull/40959)) - - - `Tensor.values()`，c++ method to get non-zero elements of a SparseCooTensor or SparseCsrTensor. ([#40608](https://github.com/PaddlePaddle/Paddle/pull/40608)) - - - `Tensor.indices()`，c++ method to get the coordinate information of a SparseCooTensor. ([#40608](https://github.com/PaddlePaddle/Paddle/pull/40608)) - - - `Tensor.crows()`，c++ method to get information about the compressed row information of the SparseCsrTensor. ([#40608](https://github.com/PaddlePaddle/Paddle/pull/40608)) - - - `Tensor.cols()`，c++ method to get the column information of the SparseCsrTensor ([#40608](https://github.com/PaddlePaddle/Paddle/pull/40608)) - - - `Tensor.to_sparse_coo()`，c++ method to convert a DenseTensor or SparseCsrTensor to a SparseCooTensor. ([#40780](https://github.com/PaddlePaddle/Paddle/pull/40780)) - - - `Tensor.to_sparse_csr()`，c++ convert a DenseTensor or SparseCooTensor to a SparseCsrTensor. ([#40780](https://github.com/PaddlePaddle/Paddle/pull/40780)) - - - `Tensor.to_dense()`，c++ convert a SparseCooTensor or SparseCsrTensor to a DenseTensor. ([#40780](https://github.com/PaddlePaddle/Paddle/pull/40780)) - -- Add hardware related APIs - - - Add four GPU memory monitoring related APIs: `paddle.device.cuda.max_memory_allocated`, `paddle.device.cuda.max_memory_reserved`, `paddle.device.cuda.memory_allocated`, and `paddle.device.cuda.memory_reserved`, to view and analyze the GPU memory usage in real-time. ([#38657](https://github.com/PaddlePaddle/Paddle/pull/38657)) - - - Add `paddle.device.cuda.get_device_properties`, to return the properties of the GPU device. ([#35661](https://github.com/PaddlePaddle/Paddle/pull/35661)) - - - Add `paddle.device.cuda.get_device_name` and `paddle.device.cuda.get_device_capability`, to return the name and compute capability of the GPU device. ([#35672](https://github.com/PaddlePaddle/Paddle/pull/35672)) - -- Add Tensor operation API - - - Add `paddle.nansum`, to sum input Tensor along `axis` with ignoring the `NaNs` values. ([#38137](https://github.com/PaddlePaddle/Paddle/pull/38137)) - - - Add `paddle.nanmean`,to average input Tensor along `axis` with ignoring the `NaNs` values. ([#40472](https://github.com/PaddlePaddle/Paddle/pull/40472)) - - - Add `paddle.clone`, to return a copy of the input Tensor and provide gradient calculation. ([#38020](https://github.com/PaddlePaddle/Paddle/pull/38020)) - - - Add `paddle.Tensor.element_size`, to return the number of bytes allocated for a single element in a Tensor. ([#38020](https://github.com/PaddlePaddle/Paddle/pull/38020)) - - - Add `paddle.Tensor.to_uva_tensor`, to convert the numpy objects to be accessed by CUDA objects with virtual addresses, which are stored in CPU memory physically. ([#39146](https://github.com/PaddlePaddle/Paddle/pull/39146), [#38950](https://github.com/PaddlePaddle/Paddle/pull/38950)) - - - Add `paddle.rot90`, to rotate the n-dimensional Tensor by 90 degrees along the plane specified by `axes`. ([#37634](https://github.com/PaddlePaddle/Paddle/pull/37634)) - - - Add `paddle.logit` and `paddle.Tensor.logit`, to compute the logit function values for input Tensor. ([#37844](https://github.com/PaddlePaddle/Paddle/pull/37844)) - - - Add `paddle.repeat_interleave`, to copy the input along the specified axis, and return a new Tensor. ([#37981](https://github.com/PaddlePaddle/Paddle/pull/37981)) - - - Add `paddle.renorm`, to split the Tensor into multiple pieces at the specified `axis` and then perform p norm operations separately. ([#38130](https://github.com/PaddlePaddle/Paddle/pull/38130), [#38459](https://github.com/PaddlePaddle/Paddle/pull/38459)) - - - Add `paddle.mode` and `paddle.Tensor.mode`, to search the values and indices of the input Tensor along the specified axis. ([#38446](https://github.com/PaddlePaddle/Paddle/pull/38446)) - - - Add `paddle.quantile` and `paddle.Tensor.quantile`, to compute the q-quantile of a Tensor along the specified axis. ([#38567](https://github.com/PaddlePaddle/Paddle/pull/38567)) - - - Add `paddle.kthvalue` and `paddle.Tensor.kthvalue`, to find the values and indices of the kth smallest at the specified axis. ([#38386](https://github.com/PaddlePaddle/Paddle/pull/38386)) - - - Add `paddle.is_floating_point` and `paddle.Tensor.is_floating_point`, to determine if the input Tensor is the floating point type. ([#37885](https://github.com/PaddlePaddle/Paddle/pull/37885)) - - - Add `paddle.erfinv` and `paddle.Tensor.erfinv`, to compute the inverse error function of the input Tensor. ([#38295](https://github.com/PaddlePaddle/Paddle/pull/38295)) - - - Add `paddle.lerp` and `paddle.Tensor.lerp`, to compute linear interpolation among the input Tensors based on the given weights. ([#37253](https://github.com/PaddlePaddle/Paddle/pull/37253)) - - - Add `paddle.angle`, to compute the phase angle of a complex Tensor. ([#37689](https://github.com/PaddlePaddle/Paddle/pull/37689)) - - - Add `paddle.rad2deg` and `paddle.Tensor.rad2deg`, to convert each of the elements of input from the angles in radians to the degrees. ([#37598](https://github.com/PaddlePaddle/Paddle/pull/37598)) - - - Add `paddle.deg2rad` and `paddle.Tensor.deg2rad`, to convert each of the elements of input from the degrees in radians to the angles. ([#37598](https://github.com/PaddlePaddle/Paddle/pull/37598)) - - - Add `paddle.gcd` and `paddle.Tensor.gcd`, to compute the greatest common divisors of the absolute values of two inputs by element. ([#37819](https://github.com/PaddlePaddle/Paddle/pull/37819)) - - - Add `paddle.lcm` and `paddle.Tensor.lcm`, to compute the least common multiple of the absolute value of two inputs by element. ([#37819](https://github.com/PaddlePaddle/Paddle/pull/37819)) - - - Add `paddle.amax` and `paddle.Tensor.amax`, to get the maximum value of Tensor elements along the specified dimension. ([#38417](https://github.com/PaddlePaddle/Paddle/pull/38417)) - - - Add `paddle.amin` and `paddle.Tensor.amin`, to get the minimum value of Tensor elements along the specified dimension. ([#38417](https://github.com/PaddlePaddle/Paddle/pull/38417)) - - - Add `paddle.isclose`, to determine if each element of two Tensors is close to each other. ([#37135](https://github.com/PaddlePaddle/Paddle/pull/37135)) - - - Add `paddle.put_along_axis` and `paddle.take_along_axis`, for extracting or placing elements with specified index subscripts. ([#38608](https://github.com/PaddlePaddle/Paddle/pull/38608)) - - - Add `paddle.bincount` and `paddle.Tensor.bincount`, for counting the number of occurrences of each element in a Tensor. ([#36317](https://github.com/PaddlePaddle/Paddle/pull/36317)) - - - Add `paddle.fmax` and `paddle.fmin`, to extend the max/min function to support the case of NaN values in the two Tensors. If there is one NaN value in the corresponding position, return that non-NaN value; if there are two NaN values in the corresponding position, return the NaN value. ([#37826](https://github.com/PaddlePaddle/Paddle/pull/37826)) - - - Add `paddle.diff`, for computing the nth forward difference along a given dimension. It currently supports n=1. ([#37441](https://github.com/PaddlePaddle/Paddle/pull/37441)) - - - Add inverse hyperbolic functions: `paddle.asinh`, `paddle.acosh`, and `paddle.atanh`. ([#37076](https://github.com/PaddlePaddle/Paddle/pull/37076)) - - - Add `paddle.as_real` and `paddle.as_complex` for conversion between real Tensor and complex Tensor. ([#37784](https://github.com/PaddlePaddle/Paddle/pull/37784)) - - - Add `paddle.complex`, for constructing a complex Tensor with the given real and imaginary parts. ([#37918](https://github.com/PaddlePaddle/Paddle/pull/37918), [#38272](https://github.com/PaddlePaddle/Paddle/pull/38272)) - - - Add `paddle.det` and `paddle.slogdet`, to compute the determinant of a matrix and the natural logarithm of the determinant. ([#34992](https://github.com/PaddlePaddle/Paddle/pull/34992)) - - - Add `paddle.nn.utils.parameters_to_vector`, to flatten parameters to a 1-D Tensor. ([#38020](https://github.com/PaddlePaddle/Paddle/pull/38020)) - - - Add `paddle.nn.utils.vector_to_parameters`, to transform a Tensor with 1-D shape to the parameters. ([#38020](https://github.com/PaddlePaddle/Paddle/pull/38020)) - -- Add networking class APIs - - - Add `paddle.nn.Fold` and `paddle.nn.functional.fold`, to extract sliding local area blocks for the Tensors of a batch. ([#38613](https://github.com/PaddlePaddle/Paddle/pull/38613)) - - - Add `paddle.nn.CELU` and `paddle.nn.functional.celu`, to support the CELU activation layer. ([#36088](https://github.com/PaddlePaddle/Paddle/pull/36088)) - - - Add `paddle.nn.HingeEmbeddingLoss`. Add a way to compute hinge embedding loss. It is usually used for nonlinear embedding or semi-supervised learning. ([#37540](https://github.com/PaddlePaddle/Paddle/pull/37540)) - - - Add `paddle.nn.ZeroPad2D` API, for zero-padding according to the padding property. ([#37151](https://github.com/PaddlePaddle/Paddle/pull/37151)) - - - Add `paddle.nn.MaxUnPool3D` and `paddle.nn.MaxUnPool1D`, for computing 3D maximum inverse pooling and 1D maximum inverse pooling. ([#38716](https://github.com/PaddlePaddle/Paddle/pull/38716)) - - - Add `paddle.incubate.graph_khop_sampler`, `paddle.incubate.graph_sample_neighbors`, and `paddle.incubate.graph_reindex` APIs, to support graph multi-order neighbor sampling and graph reindexing operations. They are mainly used for graph neural network model training. ([#39146](https://github.com/PaddlePaddle/Paddle/pull/39146), [#40809](https://github.com/PaddlePaddle/Paddle/pull/40809)) - -- Add random number class APIs - - - Add `paddle.poisson`, to generate a Tensor that obeys Poisson distributed with the lambda parameter. ([#38117](https://github.com/PaddlePaddle/Paddle/pull/38117)) - - - Add `paddle.randint_like` API, to generate a new Tensor that obeys uniform distribution in the range [low, high), with the shape of the output matching the shape of the input. ([#36169](https://github.com/PaddlePaddle/Paddle/pull/36169)) - - - Add `paddle.Tensor.exponential_`. It is an inplace style API that populates the input Tensor with exponentially distributed random numbers. ([#38256](https://github.com/PaddlePaddle/Paddle/pull/38256)) - -- Add parameter initialization class APIs - - - Add `paddle.nn.initializer.Dirac`, to initialize 3D/4D/5D parameters with Dirac delta functions. It is commonly used for initialization of Conv1D/Conv2D/Conv3D parameters in the convolution layer. ([#37389](https://github.com/PaddlePaddle/Paddle/pull/37389)) - - - Add `paddle.nn.initializer.Orthogonal` for orthogonal matrix initialization. The initialized parameter is the (semi-) orthogonal vector. ([#37163](https://github.com/PaddlePaddle/Paddle/pull/37163)) - - - Add `paddle.nn.initializer.calculate_gain`, to get the recommended gain value for the activation function. The gain value can be used to set certain initialization APIs to adjust the initialization range. ([#37163](https://github.com/PaddlePaddle/Paddle/pull/37163)) - -- Add learning rate class API - - - Add `paddle.optimizer.lr.MultiplicativeDecay`, to provide the `lambda` function to set the learning rate. ([#38250](https://github.com/PaddlePaddle/Paddle/pull/38250)) -- Add distributed-related APIs - - - Add `paddle.incubate.optimizer.DistributedFusedLamb`, to allow the Lamb optimizer to update parameters distributedly. ([#40011](https://github.com/PaddlePaddle/Paddle/pull/40011), [#39972](https://github.com/PaddlePaddle/Paddle/pull/39972), [#39900](https://github.com/PaddlePaddle/Paddle/pull/39900), [#39747](https://github.com/PaddlePaddle/Paddle/pull/39747), [#39148](https://github.com/PaddlePaddle/Paddle/pull/39148), [#39416](https://github.com/PaddlePaddle/Paddle/pull/39416)) -- Add new optimizer-related APIs([#40710](https://github.com/PaddlePaddle/Paddle/pull/40710)) - - - `paddle.incubate.optimizer.functional.minimize_bfgs`，add second-order optimizer BFGS. - - - `paddle.incubate.optimizer.functional.minimize_lbfgs`，add second-order optimizer L-BFGS. - -- Add `paddle.incubate.multiprocessing` module, to provide Tensor (CPU/GPU) data transfer between python processes. ([#37302](https://github.com/PaddlePaddle/Paddle/pull/37302), [#41339](https://github.com/PaddlePaddle/Paddle/pull/41339)) - -- Add `paddle.incubate.autotune.set_config` API, to support multi-version Kernel auto-selection, mixed precision data layout auto-conversion, and num_workers auto-selection for DataLoader to automatically improve model performance. ([#42301](https://github.com/PaddlePaddle/Paddle/pull/42301)) - -- Add `paddle.incubate.nn.FusedMultiTransformer` and `paddle.incubate.nn.functional.fused_multi_transformer` API, to fuse multiple layers of transformers into a single op to improve model inference performance. It should be noted that only forward is supported. ([#42311](https://github.com/PaddlePaddle/Paddle/pull/42311)) - -- Add einsum_v2 operators for consistent interface between dynamic graph mode and static graph mode. It is compatible with the `paddle.einsum` implementation at the original python side, while supporting dynamic to static export and more complete Infershape inference. ([#42495](https://github.com/PaddlePaddle/Paddle/pull/42495), [#42327](https://github.com/PaddlePaddle/Paddle/pull/42327), [#42397](https://github.com/PaddlePaddle/Paddle/pull/42397), [#42105](https://github.com/PaddlePaddle/Paddle/pull/42105)) - - -#### IR(Intermediate Representation) - -- Dynamic graph to static graph - - - For the variable type StaticAnalysis module, add support for type tag similar to `a, b = paddle.shape(x)`. ([#39245](https://github.com/PaddlePaddle/Paddle/pull/39245)) - - - Add a computed field, supporting `InputSpec.name` as the Program cache hash key. ([#38273](https://github.com/PaddlePaddle/Paddle/pull/38273)) - - - Add syntax for supporting `dict['key'] = x.shape`. ([#40611](https://github.com/PaddlePaddle/Paddle/pull/40611)) - - - Add the support for Pure FP16 training. ([#36944](https://github.com/PaddlePaddle/Paddle/pull/36944)) - - - Add the support `for i in [x,y,z]` syntax. ([#37259](https://github.com/PaddlePaddle/Paddle/pull/37259)) - - - Add the support for type hint syntax of python3. ([#36544](https://github.com/PaddlePaddle/Paddle/pull/36544)) - -- Pass development - - - Add forward and backward fusion for FC + [relu|gelu] based on NVIDIA cuBlasLt Epilogue. ([#39437](https://github.com/PaddlePaddle/Paddle/pull/39437)) -- Kernel Primitive API - - - Add KP operators on GPU platform, including cast, scale, clip, bce_loss, abs_grad, reduce_sum_grad, reduce_mean_grad, clip, bce_loss, full, full_like, distribution, random, masked_select_kernel, where_index, masked_select_grad, dropout, sigmoid, where, and abs_grad. ([#36203](https://github.com/PaddlePaddle/Paddle/pull/36203), [#36423](https://github.com/PaddlePaddle/Paddle/pull/36423), [#39390](https://github.com/PaddlePaddle/Paddle/pull/39390), [#39734](https://github.com/PaddlePaddle/Paddle/pull/39734), [#38500](https://github.com/PaddlePaddle/Paddle/pull/38500), [#38959](https://github.com/PaddlePaddle/Paddle/pull/38959), [#39197](https://github.com/PaddlePaddle/Paddle/pull/39197/), [#39563](https://github.com/PaddlePaddle/Paddle/pull/39563), [#39666](https://github.com/PaddlePaddle/Paddle/pull/39666), [#40517](https://github.com/PaddlePaddle/Paddle/pull/40517), [#40617](https://github.com/PaddlePaddle/Paddle/pull/40617), [#40766](https://github.com/PaddlePaddle/Paddle/pull/40766), [#39898](https://github.com/PaddlePaddle/Paddle/pull/39898), [#39609](https://github.com/PaddlePaddle/Paddle/pull/39609)) - - - Add the support for XPU2 source code compilation mode. ([#37254](https://github.com/PaddlePaddle/Paddle/pull/37254), [#40397](https://github.com/PaddlePaddle/Paddle/pull/40397), [#38455](https://github.com/PaddlePaddle/Paddle/pull/38455)) - - - Add the support for KP operator reuse on XPU2 and GPU, including reduce, broadcast, elementwise_add, `exp、log、relu、sigmoid、leaky_relu、softplus、hard_swish、reciprocal`。([#36904](https://github.com/PaddlePaddle/Paddle/pull/36904), [#37226](https://github.com/PaddlePaddle/Paddle/pull/37226), [#38918](https://github.com/PaddlePaddle/Paddle/pull/38918), [#40560](https://github.com/PaddlePaddle/Paddle/pull/40560/), [#39787](https://github.com/PaddlePaddle/Paddle/pull/39787), [#39917](https://github.com/PaddlePaddle/Paddle/pull/39917), [#40002](https://github.com/PaddlePaddle/Paddle/pull/40002), [#40364](https://github.com/PaddlePaddle/Paddle/pull/40364)) - - - Add unit tests of KP operators on the XPU2 platform, including `brelu、ceil、celu、elu、floor、hard_shrink、hard_sigmoid、log1p、logsigmoid、relu6、silu、soft_relu、softsign、sqrt、square、swish、thresholded_relu、softshrink`。([#40448](https://github.com/PaddlePaddle/Paddle/pull/40448), [#40524](https://github.com/PaddlePaddle/Paddle/pull/40524)) - - - Add the support for XPU2 KP models, including resnet50, deepfm, wide_deep, yolov3-darknet53, det_mv3_db, bert, transformer, mobilenet_v3, and GPT2. - - -#### **Mixed Precision Training** - -- Split the `paddle.amp.GradScaler.unscale_` method from the `minimize` of the mixed precision training `paddle.amp.GradScaler`, to provide a separate interface for recovering the loss. ([#35825](https://github.com/PaddlePaddle/Paddle/pull/35825)) - -- Add the FP16 support for `paddle.nn.ClipByGlobalNorm` dynamic graph mode. Add FP16 Kernel for clip op to enable clip-related operations to support FP16 compute. ([#36198](https://github.com/PaddlePaddle/Paddle/pull/36198), [#36577](https://github.com/PaddlePaddle/Paddle/pull/36577)) - -- Support the case that the `optimizer` parameter transferred from `paddle.amp.decorate` is Nan. ([#37541](https://github.com/PaddlePaddle/Paddle/pull/37541)) - -- For the merged_momentum op，add the support of input multiple learning rates, the computing for use_nesterov policy and the regularization computing. ([#37527](https://github.com/PaddlePaddle/Paddle/pull/37527)) - -- Add multi_tensor policy to `paddle.optimizer.Momentum` optimizer. Add `set_to_zero` branch to `clear_grad` of `Optimzizer` class. ([#37564](https://github.com/PaddlePaddle/Paddle/pull/37564)) - -- Add multi_tensor policy to `paddle.optimizer.Adam`. ([#38010](https://github.com/PaddlePaddle/Paddle/pull/38010)) - -- Add multi_precision policy to `paddle.optimizer.SGD` optimizer. ([#38231](https://github.com/PaddlePaddle/Paddle/pull/38231)) - -- Add the storage `master weight` parameter to the optimizer `state_dict` method. ([#39121](https://github.com/PaddlePaddle/Paddle/pull/39121)) - -- Add support for op CUDA bfloat16 mixed precision training. Support for O1 and O2 modes. Enable the above training modes via `paddle.amp.auto_cast`. ([#39029](https://github.com/PaddlePaddle/Paddle/pull/39029), [#39815](https://github.com/PaddlePaddle/Paddle/pull/39815)) - -- Add bfloat16 CUDA Kernel for the following ops: matmul, concat, split, dropout, reshape, slice, squeeze, stack, transpose, unbind, elementwize_max, elementwize_add, elementwize_mul, elementwize_sub, scale, sum, layer_norm, p_norm, reduce_sum, softmax, log_softmax, sigmoid, sqrt, softplus, square, gaussian_random, fill_constant, and fill_any_like. ([#39485](https://github.com/PaddlePaddle/Paddle/pull/39485), [#39380](https://github.com/PaddlePaddle/Paddle/pull/39380), [#39395](https://github.com/PaddlePaddle/Paddle/pull/39380), [#39402](https://github.com/PaddlePaddle/Paddle/pull/39402), [#39457](https://github.com/PaddlePaddle/Paddle/pull/39457), [#39461](https://github.com/PaddlePaddle/Paddle/pull/39461), [#39602](https://github.com/PaddlePaddle/Paddle/pull/39602), [#39716](https://github.com/PaddlePaddle/Paddle/pull/39716), [#39683](https://github.com/PaddlePaddle/Paddle/pull/39683), [#39843](https://github.com/PaddlePaddle/Paddle/pull/39843), [#39999](https://github.com/PaddlePaddle/Paddle/pull/39999), [#40004](https://github.com/PaddlePaddle/Paddle/pull/40004), [#40027](https://github.com/PaddlePaddle/Paddle/pull/40027)) - -- Add bfloat16 CPU Kernel for the following ops: dropout, reshape, slice, squeeze, unsqueeze, stack, transpose, unbind, elementwize_max, elementwise_mul, elementwise_sub, and gather. ([#39380](https://github.com/PaddlePaddle/Paddle/pull/39380), [#39395](https://github.com/PaddlePaddle/Paddle/pull/39380), [#39402](https://github.com/PaddlePaddle/Paddle/pull/39402), [#39457](https://github.com/PaddlePaddle/Paddle/pull/39457), [#39461](https://github.com/PaddlePaddle/Paddle/pull/39461), [#39602](https://github.com/PaddlePaddle/Paddle/pull/39602), [#39716](https://github.com/PaddlePaddle/Paddle/pull/39716), [#39683](https://github.com/PaddlePaddle/Paddle/pull/39683)) - -- Support printing of Tensor with data of bfloat16. ([#39375](https://github.com/PaddlePaddle/Paddle/pull/39375), [#39370](https://github.com/PaddlePaddle/Paddle/pull/39370)) - -- Add support for FP16 computation for `p_norm`, `elementwise_max`, and `fill_constant_batch_size_like ``scatter`. ([#35888](https://github.com/PaddlePaddle/Paddle/pull/35888), [#39907](https://github.com/PaddlePaddle/Paddle/pull/39907), [#38136](https://github.com/PaddlePaddle/Paddle/pull/38136), [#38499](https://github.com/PaddlePaddle/Paddle/pull/38499)) - -- Add support for int16_t for the following ops: cumsum, less_than, less_equal, greater_than, greater_equal, equal, not_equal, fill_any_like, grather_nd reduce_sum, where_index, reshape, and unsqueeze. ([#39636](https://github.com/PaddlePaddle/Paddle/pull/39636)) - -- Add support for int16_t label type for cross_entropy op. ([#39409](https://github.com/PaddlePaddle/Paddle/pull/39409)) - -- Add support for int16_t id type for embedding op. ([#39381](https://github.com/PaddlePaddle/Paddle/pull/39381)) - -- Add support for FP16 type for reduce_mean op. ([#38289](https://github.com/PaddlePaddle/Paddle/pull/38289)) - -- Add support for FP16 type for elementwise_min op. ([#38123](https://github.com/PaddlePaddle/Paddle/pull/38123)) - -- Update bfloat16 AMP oneDNN default support list. ([#39304](https://github.com/PaddlePaddle/Paddle/pull/39304)) - - -#### **Paddle HIgh reusability operator library** - -We announce PHI as the new Paddle HIgh reusability operator library. PHI provides Primitive API, enabling kernel reuse for operator development. As a refactored functional operator library, PHI aims to solve legacy problems that harm the framework's performance and reusability, in particular on the operator development. Such problems include inefficient ways of cross using operators, unclear operator interfaces and lacking direct calls to the operator library in C++. With PHI, new operators can be easily implemented by composing functions available in the functional library. The library provides over 200 C++ operator class APIs and nearly 500 kernels. Composing new operators through these built-in functions can greatly reduce the user's development effort. PHI supports different types of hardware (e.g., GPU and XPU). In addition, PHI is extensible with plugins for accommodating third party accelerators (such as NPU) in a low cost and reusable fashion. In short, PHI supports low level operator composabilty, the reuse of kernels through Primitives, and accelerators through plugins.The main contents include six parts as below: - -- **The implementation of the operator library infrastructure, core components and mechanisms**: The directory structure of the new operator library is reasonably planned, design and implement the common base data structure of the new operator library, the new functional InferMeta and Kernel development paradigm and the corresponding registration and management components. Support the automated compilation object generation and compilation dependency generation of Kernel files, allowing developers to focus only on the functional Kernel implementation, and making the development paradigm clear and concise. ([#34425](https://github.com/PaddlePaddle/Paddle/pull/34425), [#37107](https://github.com/PaddlePaddle/Paddle/pull/37107), [#36946](https://github.com/PaddlePaddle/Paddle/pull/36946), [#36948](https://github.com/PaddlePaddle/Paddle/pull/36948), [#37876](https://github.com/PaddlePaddle/Paddle/pull/37876), [#37916](https://github.com/PaddlePaddle/Paddle/pull/37916), [#37977](https://github.com/PaddlePaddle/Paddle/pull/37977), [38078](https://github.com/PaddlePaddle/Paddle/pull/38078), [#38861](https://github.com/PaddlePaddle/Paddle/pull/38861), [#39123](https://github.com/PaddlePaddle/Paddle/pull/39123), [#39131](https://github.com/PaddlePaddle/Paddle/pull/39131), [#39748](https://github.com/PaddlePaddle/Paddle/pull/39748), [#39790](https://github.com/PaddlePaddle/Paddle/pull/39790), [#39941](https://github.com/PaddlePaddle/Paddle/pull/39941), [#40239](https://github.com/PaddlePaddle/Paddle/pull/40239), [#40635](https://github.com/PaddlePaddle/Paddle/pull/40635), [#41091](https://github.com/PaddlePaddle/Paddle/pull/41091), [#37409](https://github.com/PaddlePaddle/Paddle/pull/37409), [#37942](https://github.com/PaddlePaddle/Paddle/pull/37942), [#39002](https://github.com/PaddlePaddle/Paddle/pull/39002), [#38109](https://github.com/PaddlePaddle/Paddle/pull/38109), [#37881](https://github.com/PaddlePaddle/Paddle/pull/37881), [#37517](https://github.com/PaddlePaddle/Paddle/pull/37517), [#39870](https://github.com/PaddlePaddle/Paddle/pull/39870), [#40975](https://github.com/PaddlePaddle/Paddle/pull/40975), [#39475](https://github.com/PaddlePaddle/Paddle/pull/39475), [#37304](https://github.com/PaddlePaddle/Paddle/pull/37304), #36910, #37120, #37146, #37215, #37255, #37369, #38258, #38257, #38355, #38853, #38937, #38977, #38946, #39085, #39153, #39228, #38301, #38275, #38506, #38607, #38473, #38632, #38811, #38880, #38996, #38914, #39101) - -- **Operator library C++ API system construction**: design and implement yaml configuration file-based operator definition paradigm, to automatically generate more than 200 C++ operator class APIs for internal and external developers to reuse. This reduces the cost of repeated development of basic operators. ([#37668](https://github.com/PaddlePaddle/Paddle/pull/37668), [#36938](https://github.com/PaddlePaddle/Paddle/pull/36938), [#38172](https://github.com/PaddlePaddle/Paddle/pull/38172), [#38182](https://github.com/PaddlePaddle/Paddle/pull/38182), [#38311](https://github.com/PaddlePaddle/Paddle/pull/38311), [#38438](https://github.com/PaddlePaddle/Paddle/pull/38438), [#39057](https://github.com/PaddlePaddle/Paddle/pull/39057), [#39229](https://github.com/PaddlePaddle/Paddle/pull/39229), [#39281](https://github.com/PaddlePaddle/Paddle/pull/39281), [#39263](https://github.com/PaddlePaddle/Paddle/pull/39263), [#39408](https://github.com/PaddlePaddle/Paddle/pull/39408), [#39436](https://github.com/PaddlePaddle/Paddle/pull/39436), [#39482](https://github.com/PaddlePaddle/Paddle/pull/39482), [#39497](https://github.com/PaddlePaddle/Paddle/pull/39497), [#39651](https://github.com/PaddlePaddle/Paddle/pull/39651), [#39521](https://github.com/PaddlePaddle/Paddle/pull/39521), [#39760](https://github.com/PaddlePaddle/Paddle/pull/39760), [#40060](https://github.com/PaddlePaddle/Paddle/pull/40060), [#40196](https://github.com/PaddlePaddle/Paddle/pull/40196), [#40218](https://github.com/PaddlePaddle/Paddle/pull/40218), [#40640](https://github.com/PaddlePaddle/Paddle/pull/40640), [#40732](https://github.com/PaddlePaddle/Paddle/pull/40732), [#40729](https://github.com/PaddlePaddle/Paddle/pull/40729), [#40840](https://github.com/PaddlePaddle/Paddle/pull/40840), [#40867](https://github.com/PaddlePaddle/Paddle/pull/40867), [#41025](https://github.com/PaddlePaddle/Paddle/pull/41025), [#41368](https://github.com/PaddlePaddle/Paddle/pull/41368)) - -- **Operator library compatible with various execution systems**: Implement new InferMeta and Kernel to access the original dynamic and static graph execution system. Support the safe removal of the original OpKernel registration and migration to the new Kernel form. ([#34425](https://github.com/PaddlePaddle/Paddle/pull/34425), [#38825](https://github.com/PaddlePaddle/Paddle/pull/38825), [#38837](https://github.com/PaddlePaddle/Paddle/pull/38837), [#38842](https://github.com/PaddlePaddle/Paddle/pull/38842), [#38976](https://github.com/PaddlePaddle/Paddle/pull/38976), [#39134](https://github.com/PaddlePaddle/Paddle/pull/39134), [#39140](https://github.com/PaddlePaddle/Paddle/pull/39140), [#39135](https://github.com/PaddlePaddle/Paddle/pull/39135), [#39252](https://github.com/PaddlePaddle/Paddle/pull/39252), [#39222](https://github.com/PaddlePaddle/Paddle/pull/39222), [#39351](https://github.com/PaddlePaddle/Paddle/pull/39351)) - -- **Decouple the underlying data structures and tool functions of the operator library from the framework**: Relieve PHI's dependence on the framework for core data structures, lay the foundation for subsequent independent compilation of PHI, and support infrt, custom Kernel, and a series of Phi-based construction work ([#38583](https://github.com/PaddlePaddle/Paddle/pull/38583), [#39188](https://github.com/PaddlePaddle/Paddle/pull/39188), [#39560](https://github.com/PaddlePaddle/Paddle/pull/39560), [#39931](https://github.com/PaddlePaddle/Paddle/pull/39931), [#39169](https://github.com/PaddlePaddle/Paddle/pull/39169), [#38951](https://github.com/PaddlePaddle/Paddle/pull/38951), [#38898](https://github.com/PaddlePaddle/Paddle/pull/38898), [#38873](https://github.com/PaddlePaddle/Paddle/pull/38873), [#38696](https://github.com/PaddlePaddle/Paddle/pull/38696), [#38651](https://github.com/PaddlePaddle/Paddle/pull/38651), [#39359](https://github.com/PaddlePaddle/Paddle/pull/39359), [#39305](https://github.com/PaddlePaddle/Paddle/pull/39305), [#39234](https://github.com/PaddlePaddle/Paddle/pull/39234), [#39098](https://github.com/PaddlePaddle/Paddle/pull/39098), [#39120](https://github.com/PaddlePaddle/Paddle/pull/39120), [#38979](https://github.com/PaddlePaddle/Paddle/pull/38979), [#38899](https://github.com/PaddlePaddle/Paddle/pull/38899), [#38844](https://github.com/PaddlePaddle/Paddle/pull/38844), [#39714](https://github.com/PaddlePaddle/Paddle/pull/39714), [#39729](https://github.com/PaddlePaddle/Paddle/pull/39729), [#39889](https://github.com/PaddlePaddle/Paddle/pull/39889), [#39587](https://github.com/PaddlePaddle/Paddle/pull/39587), [#39558](https://github.com/PaddlePaddle/Paddle/pull/39558), [#39514](https://github.com/PaddlePaddle/Paddle/pull/39514), [#39502](https://github.com/PaddlePaddle/Paddle/pull/39502), [#39300](https://github.com/PaddlePaddle/Paddle/pull/39300), [#39246](https://github.com/PaddlePaddle/Paddle/pull/39246), [#39124](https://github.com/PaddlePaddle/Paddle/pull/39124)) - -- **Integration between custom operator mechanism and Phi with improvement**: support for calling over 200 C++ operator class APIs automatically generated by PHI when writing custom operators. This reduces custom operator development costs. A series of bugs are fixed. ([#37122](https://github.com/PaddlePaddle/Paddle/pull/37122), [#37276](https://github.com/PaddlePaddle/Paddle/pull/37276), [#37281](https://github.com/PaddlePaddle/Paddle/pull/37281), [#37262](https://github.com/PaddlePaddle/Paddle/pull/37281), [#37415](https://github.com/PaddlePaddle/Paddle/pull/37415), [#37423](https://github.com/PaddlePaddle/Paddle/pull/37423), [#37583](https://github.com/PaddlePaddle/Paddle/pull/37683), [#38776](https://github.com/PaddlePaddle/Paddle/pull/38776), [#39353](https://github.com/PaddlePaddle/Paddle/pull/39353), [#41072](https://github.com/PaddlePaddle/Paddle/pull/41072)) - -- **Operator scale migration and refactoring**: migrate about 250 high-frequency forward and backward operator Kernel to the new operator library and refactor them as a single function. Achieve the high-performance operator by encapsulating multiple base Kernel functions on the C++ side for the fast combination. Meanwhile, add the corresponding yaml operator definition, and access to the new dynamic graph execution system to improve the python API scheduling performance. The migrated and refactored operators include: - - - sqrt ([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - square([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - sin ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - sinh ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - elementwise_fmax([#40140](https://github.com/PaddlePaddle/Paddle/pull/40140)) - - - elementwise_fmin([#40140](https://github.com/PaddlePaddle/Paddle/pull/40140)) - - - pool2d([#40208](https://github.com/PaddlePaddle/Paddle/pull/40208), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - max_pool2d_with_index([#40208](https://github.com/PaddlePaddle/Paddle/pull/40208), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - pool3d([#40208](https://github.com/PaddlePaddle/Paddle/pull/40208), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - max_pool3d_with_index([#40208](https://github.com/PaddlePaddle/Paddle/pull/40208), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - fill_constant ([#36930](https://github.com/PaddlePaddle/Paddle/pull/36930), [#39465](https://github.com/PaddlePaddle/Paddle/pull/39465)) - - - p_norm ([#40819](https://github.com/PaddlePaddle/Paddle/pull/40819)) - - - fill_constant_batch_size_like ([#40784](https://github.com/PaddlePaddle/Paddle/pull/40784)) - - - conv2d([#39354](https://github.com/PaddlePaddle/Paddle/pull/39354)) - - - conv2d_transpose([#40675](https://github.com/PaddlePaddle/Paddle/pull/40675), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - conv3d([#39354](https://github.com/PaddlePaddle/Paddle/pull/39354)) - - - conv3d_transpose([#40675](https://github.com/PaddlePaddle/Paddle/pull/40675), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - mish([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - gather_nd ([#40090](https://github.com/PaddlePaddle/Paddle/pull/40090), [#40043](https://github.com/PaddlePaddle/Paddle/pull/40043)) - - - gather ([#40500](https://github.com/PaddlePaddle/Paddle/pull/40500)) - - - scatter ([#40090](https://github.com/PaddlePaddle/Paddle/pull/40090), [#40043](https://github.com/PaddlePaddle/Paddle/pull/40043)) - - - scatter_nd_add ([#40090](https://github.com/PaddlePaddle/Paddle/pull/40090), [#40043](https://github.com/PaddlePaddle/Paddle/pull/40043)) - - - sgd([40045](https://github.com/PaddlePaddle/Paddle/pull/40045)) - - - momentum ([#41319](https://github.com/PaddlePaddle/Paddle/pull/41319)) - - - rmsprop([#40994](https://github.com/PaddlePaddle/Paddle/pull/40994)) - - - index_sample([#38130](https://github.com/PaddlePaddle/Paddle/pull/38130), [#38459](https://github.com/PaddlePaddle/Paddle/pull/38459),[#39905](https://github.com/PaddlePaddle/Paddle/pull/39905)) - - - adam ([#40351](https://github.com/PaddlePaddle/Paddle/pull/40351)) - - - layer_norm([#40193](https://github.com/PaddlePaddle/Paddle/pull/40193)) - - - adagrad([#40994](https://github.com/PaddlePaddle/Paddle/pull/40994/)) - - - adamax ([#40173](https://github.com/PaddlePaddle/Paddle/pull/40173)) - - - adadelta ([#40173](https://github.com/PaddlePaddle/Paddle/pull/40173)) - - - clip([#40602](https://github.com/PaddlePaddle/Paddle/pull/40602), [#41661](https://github.com/PaddlePaddle/Paddle/pull/41661), [#41675](https://github.com/PaddlePaddle/Paddle/pull/41675)) - - - ceil ([#40913](https://github.com/PaddlePaddle/Paddle/pull/40913)) - - - cos ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - atan ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - cosh ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - erf([#40388](https://github.com/PaddlePaddle/Paddle/pull/40388)) - - - asin ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - acos ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - scale ([#39278](https://github.com/PaddlePaddle/Paddle/pull/39278)) - - - elementwise_pow ([#40993](https://github.com/PaddlePaddle/Paddle/pull/40993)) - - - elementwise_sub ([#39225](https://github.com/PaddlePaddle/Paddle/pull/39225), [#37260](https://github.com/PaddlePaddle/Paddle/pull/37260)) - - - round ([#40913](https://github.com/PaddlePaddle/Paddle/pull/40913)) - - - floor ([#40913](https://github.com/PaddlePaddle/Paddle/pull/40913)) - - - pow ([#40913](https://github.com/PaddlePaddle/Paddle/pull/40913)) - - - elementwise_floordiv ([#40993](https://github.com/PaddlePaddle/Paddle/pull/40993)) - - - reciprocal([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - log1p ([#40785](https://github.com/PaddlePaddle/Paddle/pull/40785)) - - - allclose ([#40469](https://github.com/PaddlePaddle/Paddle/pull/40469)) - - - mul ([#40833](https://github.com/PaddlePaddle/Paddle/pull/40833)) - - - elementwise_max ([#40590](https://github.com/PaddlePaddle/Paddle/pull/40590)) - - - elementwise_min ([#40590](https://github.com/PaddlePaddle/Paddle/pull/40590)) - - - elementwise_mod ([#40590](https://github.com/PaddlePaddle/Paddle/pull/40590)) - - - elementwise_add ([#39048](https://github.com/PaddlePaddle/Paddle/pull/39048), [#37043](https://github.com/PaddlePaddle/Paddle/pull/37043)) - - - matmul_v2 ([#36844](https://github.com/PaddlePaddle/Paddle/pull/36844), [#38713](https://github.com/PaddlePaddle/Paddle/pull/38713)) - - - elementwise_mul ([#41042](https://github.com/PaddlePaddle/Paddle/pull/41042), [#40252](https://github.com/PaddlePaddle/Paddle/pull/40252), [#37471](https://github.com/PaddlePaddle/Paddle/pull/37471)) - - - elementwise_div ([#40172](https://github.com/PaddlePaddle/Paddle/pull/40172), [#40039](https://github.com/PaddlePaddle/Paddle/pull/40039), [#37418](https://github.com/PaddlePaddle/Paddle/pull/37418)) - - - SelectedRows ([#39037](https://github.com/PaddlePaddle/Paddle/pull/39037), [#39087](https://github.com/PaddlePaddle/Paddle/pull/39087), [#39128](https://github.com/PaddlePaddle/Paddle/pull/39128), [#39162](https://github.com/PaddlePaddle/Paddle/pull/39162), [#39236](https://github.com/PaddlePaddle/Paddle/pull/39236)) - - - fill_any_like ([#39807](https://github.com/PaddlePaddle/Paddle/pull/39807)) - - - dot([#38359](https://github.com/PaddlePaddle/Paddle/pull/38359)) - - - sum ([#40873](https://github.com/PaddlePaddle/Paddle/pull/40873)) - - - cumsum ([#39976](https://github.com/PaddlePaddle/Paddle/pull/39976), [#40200](https://github.com/PaddlePaddle/Paddle/pull/40200)) - - - diag_v2 ([#39914](https://github.com/PaddlePaddle/Paddle/pull/39914)) - - - auc ([#39976](https://github.com/PaddlePaddle/Paddle/pull/39976), [#40200](https://github.com/PaddlePaddle/Paddle/pull/40200)) - - - log_loss ([#39976](https://github.com/PaddlePaddle/Paddle/pull/39976), [#40200](https://github.com/PaddlePaddle/Paddle/pull/40200)) - - - one_hot_v2([39876](https://github.com/PaddlePaddle/Paddle/pull/39876)) - - - sigmoid_cross_entropy_with_logits ([#39976](https://github.com/PaddlePaddle/Paddle/pull/39976), [#40200](https://github.com/PaddlePaddle/Paddle/pull/40200)) - - - bce_loss ([#39868](https://github.com/PaddlePaddle/Paddle/pull/39868)) - - - argsort ([#40151](https://github.com/PaddlePaddle/Paddle/pull/40151)) - - - arg_max ([#40222](https://github.com/PaddlePaddle/Paddle/pull/40222)) - - - arg_min ([#40222](https://github.com/PaddlePaddle/Paddle/pull/40222)) - - - segment_pool ([#40099](https://github.com/PaddlePaddle/Paddle/pull/40099)) - - - frobenius_norm([#40707](https://github.com/PaddlePaddle/Paddle/pull/40707), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - dist ([#40178](https://github.com/PaddlePaddle/Paddle/pull/40178)) - - - isnan_v2 ([#40076](https://github.com/PaddlePaddle/Paddle/pull/40076)) - - - logical_and ([#39942](https://github.com/PaddlePaddle/Paddle/pull/39942)) - - - logical_not ([#39942](https://github.com/PaddlePaddle/Paddle/pull/39942)) - - - isfinite_v2 ([#40076](https://github.com/PaddlePaddle/Paddle/pull/40076)) - - - logical_or ([#39942](https://github.com/PaddlePaddle/Paddle/pull/39942)) - - - isinf_v2 ([#40076](https://github.com/PaddlePaddle/Paddle/pull/40076)) - - - is_empty ([#39919](https://github.com/PaddlePaddle/Paddle/pull/39919)) - - - logical_xor ([#39942](https://github.com/PaddlePaddle/Paddle/pull/39942)) - - - less_than([#39970](https://github.com/PaddlePaddle/Paddle/pull/39970)) - - - not_equal([#39970](https://github.com/PaddlePaddle/Paddle/pull/39970)) - - - equal([#39970](https://github.com/PaddlePaddle/Paddle/pull/39970)) - - - less_equal([#39970](https://github.com/PaddlePaddle/Paddle/pull/39970)) - - - equal_all([#39970](https://github.com/PaddlePaddle/Paddle/pull/39970)) - - - uniform_random ([#39937](https://github.com/PaddlePaddle/Paddle/pull/39937)) - - - randint ([#39876](https://github.com/PaddlePaddle/Paddle/pull/39876), [#41375](https://github.com/PaddlePaddle/Paddle/pull/41375)) - - - randperm ([#41265](https://github.com/PaddlePaddle/Paddle/pull/41265)) - - - unbind ([#39789](https://github.com/PaddlePaddle/Paddle/pull/39789)) - - - bernoulli ([#39590](https://github.com/PaddlePaddle/Paddle/pull/39590)) - - - increment ([#39858](https://github.com/PaddlePaddle/Paddle/pull/39858), [#39913](https://github.com/PaddlePaddle/Paddle/pull/39913)) - - - multinomial ([#39858](https://github.com/PaddlePaddle/Paddle/pull/39858), [#39913](https://github.com/PaddlePaddle/Paddle/pull/39913)) - - - addmm ([#39858](https://github.com/PaddlePaddle/Paddle/pull/39858), [#39913](https://github.com/PaddlePaddle/Paddle/pull/39913)) - - - cholesky ([#39858](https://github.com/PaddlePaddle/Paddle/pull/39858), [#39913](https://github.com/PaddlePaddle/Paddle/pull/39913)) - - - where ([#39811](https://github.com/PaddlePaddle/Paddle/pull/39811)) - - - log10 ([#40785](https://github.com/PaddlePaddle/Paddle/pull/40785)) - - - log2 ([#40785](https://github.com/PaddlePaddle/Paddle/pull/40785)) - - - expm1([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - atan2 ([#39806](https://github.com/PaddlePaddle/Paddle/pull/39806)) - - - gaussian_random ([#39932](https://github.com/PaddlePaddle/Paddle/pull/39932), [#40122](https://github.com/PaddlePaddle/Paddle/pull/40122), [#40191](https://github.com/PaddlePaddle/Paddle/pull/40191)) - - - empty ([#38334](https://github.com/PaddlePaddle/Paddle/pull/38334)) - - - truncated_gaussian_random ([#39971](https://github.com/PaddlePaddle/Paddle/pull/39971), [#40191](https://github.com/PaddlePaddle/Paddle/pull/40191)) - - - mv ([#39861](https://github.com/PaddlePaddle/Paddle/pull/39861), [#39954](https://github.com/PaddlePaddle/Paddle/pull/39954)) - - - tan ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - set_value ([#40195](https://github.com/PaddlePaddle/Paddle/pull/40195), [#40478](https://github.com/PaddlePaddle/Paddle/pull/40478), [#40636](https://github.com/PaddlePaddle/Paddle/pull/40636)) - - - bitwise_and ([#40031](https://github.com/PaddlePaddle/Paddle/pull/40031)) - - - bitwise_not([#40031](https://github.com/PaddlePaddle/Paddle/pull/40031)) - - - bitwise_or([#40031](https://github.com/PaddlePaddle/Paddle/pull/40031)) - - - poisson([#39814](https://github.com/PaddlePaddle/Paddle/pull/39814)) - - - cholesky_solve([#40387](https://github.com/PaddlePaddle/Paddle/pull/40387)) - - - bitwise_xor([#40031](https://github.com/PaddlePaddle/Paddle/pull/40031)) - - - triangular_solve([#40417](https://github.com/PaddlePaddle/Paddle/pull/40417)) - - - sigmoid ([#40626](https://github.com/PaddlePaddle/Paddle/pull/40626)) - - - atanh ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - softsign([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - thresholded_relu ([#40385](https://github.com/PaddlePaddle/Paddle/pull/40385)) - - - tanh_shrink ([#40565](https://github.com/PaddlePaddle/Paddle/pull/40565)) - - - stanh([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - reduce_mean ([#37559](https://github.com/PaddlePaddle/Paddle/pull/37559)) - - - reduce_max([#40225](https://github.com/PaddlePaddle/Paddle/pull/40225)) - - - reduce_min ([#40374](https://github.com/PaddlePaddle/Paddle/pull/40374)) - - - mean ([#40872](https://github.com/PaddlePaddle/Paddle/pull/40872), [#41319](https://github.com/PaddlePaddle/Paddle/pull/41319)) - - - reduce_all ([#40374](https://github.com/PaddlePaddle/Paddle/pull/40374)) - - - reduce_any ([#40374](https://github.com/PaddlePaddle/Paddle/pull/40374)) - - - logsumexp ([#40790](https://github.com/PaddlePaddle/Paddle/pull/40790)) - - - softshrink([#40565](https://github.com/PaddlePaddle/Paddle/pull/40565)) - - - range ([#41265](https://github.com/PaddlePaddle/Paddle/pull/41265), [#40581](https://github.com/PaddlePaddle/Paddle/pull/40851)) - - - stack([#40581](https://github.com/PaddlePaddle/Paddle/pull/40851)) - - - tile ([#40371](https://github.com/PaddlePaddle/Paddle/pull/40371)) - - - unique([#40581](https://github.com/PaddlePaddle/Paddle/pull/40851)) - - - unstack([#40581](https://github.com/PaddlePaddle/Paddle/pull/40851)) - - - slice([#40736](https://github.com/PaddlePaddle/Paddle/pull/40736)) - - - transpose2([#39327](https://github.com/PaddlePaddle/Paddle/pull/39327)) - - - unsqueeze2( [#40596](https://github.com/PaddlePaddle/Paddle/pull/40596)) - - - squeeze2( [#40596](https://github.com/PaddlePaddle/Paddle/pull/40596)) - - - strided_slice ([#40708](https://github.com/PaddlePaddle/Paddle/pull/40708)) - - - softmax ([#39547](https://github.com/PaddlePaddle/Paddle/pull/39547)) - - - leaky_relu ([#40385](https://github.com/PaddlePaddle/Paddle/pull/40385)) - - - gelu ([#40393](https://github.com/PaddlePaddle/Paddle/pull/40393)) - - - prelu ([#40393](https://github.com/PaddlePaddle/Paddle/pull/40393)) - - - log_softmax ([#40393](https://github.com/PaddlePaddle/Paddle/pull/40393)) - - - elu ([#40565](https://github.com/PaddlePaddle/Paddle/pull/40565)) - - - logsigmoid ([#40626](https://github.com/PaddlePaddle/Paddle/pull/40626)) - - - psroi_pool ([#40353](https://github.com/PaddlePaddle/Paddle/pull/40353), [#41173](https://github.com/PaddlePaddle/Paddle/pull/41173)) - - - kthvalue([#40575](https://github.com/PaddlePaddle/Paddle/pull/40575)) - - - mode ([#40571](https://github.com/PaddlePaddle/Paddle/pull/40571)) - - - yolo_box([#40112](https://github.com/PaddlePaddle/Paddle/pull/40112)) - - - yolov3_loss ([#40944](https://github.com/PaddlePaddle/Paddle/pull/40944)) - - - temporal_shift([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - depthwise_conv2d([#39354](https://github.com/PaddlePaddle/Paddle/pull/39354)) - - - pad3d ([#40701](https://github.com/PaddlePaddle/Paddle/pull/40701)) - - - pad( [#40012](https://github.com/PaddlePaddle/Paddle/pull/40012)) - - - greater_equal([#39970](https://github.com/PaddlePaddle/Paddle/pull/39970)) - - - kldiv_loss ([#39770](https://github.com/PaddlePaddle/Paddle/pull/39770)) - - - isclose ([#39770](https://github.com/PaddlePaddle/Paddle/pull/39770)) - - - silu ([#40565](https://github.com/PaddlePaddle/Paddle/pull/40565)) - - - unfold ([#39778](https://github.com/PaddlePaddle/Paddle/pull/39778)) - - - batch_norm([39347](https://github.com/PaddlePaddle/Paddle/pull/39347)) - - - norm([#39324](https://github.com/PaddlePaddle/Paddle/pull/39324)) - - - roi_pool ([#40574](https://github.com/PaddlePaddle/Paddle/pull/40574), [#40682](https://github.com/PaddlePaddle/Paddle/pull/40682), [#41173](https://github.com/PaddlePaddle/Paddle/pull/41173)) - - - roi_align ([#40382](https://github.com/PaddlePaddle/Paddle/pull/40382), [#40556](https://github.com/PaddlePaddle/Paddle/pull/40556), [#41402](https://github.com/PaddlePaddle/Paddle/pull/41402)) - - - deformable_conv ([#40700](https://github.com/PaddlePaddle/Paddle/pull/40700), [#40794](https://github.com/PaddlePaddle/Paddle/pull/40794), [#41644](https://github.com/PaddlePaddle/Paddle/pull/41644)) - - - deformable_conv_v1 ([#40794](https://github.com/PaddlePaddle/Paddle/pull/40794), [#41644](https://github.com/PaddlePaddle/Paddle/pull/41644)) - - - label_smooth ([#39796](https://github.com/PaddlePaddle/Paddle/pull/39796)) - - - grid_sampler ([#40585](https://github.com/PaddlePaddle/Paddle/pull/40585)) - - - greater_than([#39970](https://github.com/PaddlePaddle/Paddle/pull/39970)) - - - pixel_shuffle ([#39949](https://github.com/PaddlePaddle/Paddle/pull/39949), [#39712](https://github.com/PaddlePaddle/Paddle/pull/39712)) - - - nearest_interp_v2 ([#40855](https://github.com/PaddlePaddle/Paddle/pull/40855)) - - - bilinear_interp_v2 ([#40855](https://github.com/PaddlePaddle/Paddle/pull/40855)) - - - softmax_with_cross_entropy ([#40832](https://github.com/PaddlePaddle/Paddle/pull/40832)) - - - rnn ([#41007](https://github.com/PaddlePaddle/Paddle/pull/41007)) - - - reverse ([#40791](https://github.com/PaddlePaddle/Paddle/pull/40791)) - - - trace ([#39510](https://github.com/PaddlePaddle/Paddle/pull/39510)) - - - kron([#40427](https://github.com/PaddlePaddle/Paddle/pull/40427)) - - - accuracy([#39982](https://github.com/PaddlePaddle/Paddle/pull/39982)) - - - gather_tree ([#40082](https://github.com/PaddlePaddle/Paddle/pull/40082), [#39844](https://github.com/PaddlePaddle/Paddle/pull/39844)) - - - dropout([#40148](https://github.com/PaddlePaddle/Paddle/pull/40148)) - - - bincount ([#39947](https://github.com/PaddlePaddle/Paddle/pull/39947)) - - - warpctc ([#41389](https://github.com/PaddlePaddle/Paddle/pull/41389), [#40023](https://github.com/PaddlePaddle/Paddle/pull/https://github.com/PaddlePaddle/Paddle/pull/40023)) - - - multiplex([#40007](https://github.com/PaddlePaddle/Paddle/pull/40007), [#40102](https://github.com/PaddlePaddle/Paddle/pull/40102)) - - - qr([#40007](https://github.com/PaddlePaddle/Paddle/pull/40007), [#40007](https://github.com/PaddlePaddle/Paddle/pull/40007)) - - - assign_value ([#40967](https://github.com/PaddlePaddle/Paddle/pull/40967)) - - - assign ([#40022](https://github.com/PaddlePaddle/Paddle/pull/40022)) - - - cast ([#37610](https://github.com/PaddlePaddle/Paddle/pull/37610)) - - - tril_triu([#40007](https://github.com/PaddlePaddle/Paddle/pull/40007), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - where_index ([#40255](https://github.com/PaddlePaddle/Paddle/pull/40255)) - - - index_select ([#40260](https://github.com/PaddlePaddle/Paddle/pull/40260), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - roll ([#40257](https://github.com/PaddlePaddle/Paddle/pull/40257), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - cumprod (Xiong Kun [#39770](https://github.com/PaddlePaddle/Paddle/pull/39770)) - - - shard_index ([#40254](https://github.com/PaddlePaddle/Paddle/pull/40254)) - - - reshape2 ([#40914](https://github.com/PaddlePaddle/Paddle/pull/40914), [#39631](https://github.com/PaddlePaddle/Paddle/pull/39631), [#38833](https://github.com/PaddlePaddle/Paddle/pull/38833), [#37164](https://github.com/PaddlePaddle/Paddle/pull/37164)) - - - flip ([#39822](https://github.com/PaddlePaddle/Paddle/pull/39822), [#40974](https://github.com/PaddlePaddle/Paddle/pull/40974)) - - - eye ([#39712](https://github.com/PaddlePaddle/Paddle/pull/39712), [#40105](https://github.com/PaddlePaddle/Paddle/pull/40105), [#41476](https://github.com/PaddlePaddle/Paddle/pull/41476)) - - - lookup_table_v2([#39901](https://github.com/PaddlePaddle/Paddle/pull/39901)) - - - searchsorted([#40520](https://github.com/PaddlePaddle/Paddle/pull/40520), [#41053](https://github.com/PaddlePaddle/Paddle/pull/41053)) - - - adamw ([#40351](https://github.com/PaddlePaddle/Paddle/pull/40351)) - - - tanh ([#40385](https://github.com/PaddlePaddle/Paddle/pull/40385)) - - - cross ([#39829](https://github.com/PaddlePaddle/Paddle/pull/39829)) - - - concat ([#38955](https://github.com/PaddlePaddle/Paddle/pull/38955), [#41112](https://github.com/PaddlePaddle/Paddle/pull/41112)) - - - split ([#39060](https://github.com/PaddlePaddle/Paddle/pull/39060)) - - - linspace ([#40124](https://github.com/PaddlePaddle/Paddle/pull/40124)) - - - huber_loss ([#39761](https://github.com/PaddlePaddle/Paddle/pull/39761)) - - - hierarchical_sigmoid([#40553](https://github.com/PaddlePaddle/Paddle/pull/40553)) - - - nll_loss ([#39936](https://github.com/PaddlePaddle/Paddle/pull/https://github.com/PaddlePaddle/Paddle/pull/39936)) - - - graph_send_recv ([#40092](https://github.com/PaddlePaddle/Paddle/pull/40092), [#40320](https://github.com/PaddlePaddle/Paddle/pull/40320)) - - - abs([#39492](https://github.com/PaddlePaddle/Paddle/pull/39492), [#39762](https://github.com/PaddlePaddle/Paddle/pull/39762)) - - - exp([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - rsqrt([#40727](https://github.com/PaddlePaddle/Paddle/pull/40727)) - - - viterbi_decode ([#40186](https://github.com/PaddlePaddle/Paddle/pull/40186)) - - - conj ([#38247](https://github.com/PaddlePaddle/Paddle/pull/38247)) - - - real ([#39777](https://github.com/PaddlePaddle/Paddle/pull/39777), [#41173](https://github.com/PaddlePaddle/Paddle/pull/41173)) - - - imag ([#39777](https://github.com/PaddlePaddle/Paddle/pull/39777), [#41173](https://github.com/PaddlePaddle/Paddle/pull/41173)) - - - take_along_axis ([#39959](https://github.com/PaddlePaddle/Paddle/pull/39959), [#40270](https://github.com/PaddlePaddle/Paddle/pull/40270), [#40974](https://github.com/PaddlePaddle/Paddle/pull/40974)) - - - put_along_axis ([#39959](https://github.com/PaddlePaddle/Paddle/pull/39959), [#40974](https://github.com/PaddlePaddle/Paddle/pull/40974)) - - - lgamma ([#39770](https://github.com/PaddlePaddle/Paddle/pull/39770)) - - - relu ([#40175](https://github.com/PaddlePaddle/Paddle/pull/40175)) - - - maxout ([#39959](https://github.com/PaddlePaddle/Paddle/pull/39959), [#40974](https://github.com/PaddlePaddle/Paddle/pull/40974)) - - - log ([#40785](https://github.com/PaddlePaddle/Paddle/pull/40785)) - - - bilinear_tensor_product([#39903](https://github.com/PaddlePaddle/Paddle/pull/39903)) - - - flatten_contiguous_range ([#38712](https://github.com/PaddlePaddle/Paddle/pull/38712), [#36957](https://github.com/PaddlePaddle/Paddle/pull/36957), [#41345](https://github.com/PaddlePaddle/Paddle/pull/41345)) - - - matrix_rank ([#40074](https://github.com/PaddlePaddle/Paddle/pull/40074), [#40519](https://github.com/PaddlePaddle/Paddle/pull/40519), [#41466](https://github.com/PaddlePaddle/Paddle/pull/41466)) - - - logit ([#37844](https://github.com/PaddlePaddle/Paddle/pull/37844)) - - - lerp ([#40105](https://github.com/PaddlePaddle/Paddle/pull/40105), [#39524](https://github.com/PaddlePaddle/Paddle/pull/39524)) - - - erfinv ([#39949](https://github.com/PaddlePaddle/Paddle/pull/39949), [#39712](https://github.com/PaddlePaddle/Paddle/pull/39712)) - - - broadcast_tensors([#40047](https://github.com/PaddlePaddle/Paddle/pull/40047)) - - - gumbel_softmax([#39873](https://github.com/PaddlePaddle/Paddle/pull/39873)) - - - diagonal ([#39575](https://github.com/PaddlePaddle/Paddle/pull/39575)) - - - trunc ([#39543](https://github.com/PaddlePaddle/Paddle/pull/39543), [#39772](https://github.com/PaddlePaddle/Paddle/pull/39772)) - - - multi_dot ([#40038](https://github.com/PaddlePaddle/Paddle/pull/40038)) - - - matrix_power ([#40231](https://github.com/PaddlePaddle/Paddle/pull/40231)) - - - digamma([#39240](https://github.com/PaddlePaddle/Paddle/pull/39240)) - - - masked_select([#39193](https://github.com/PaddlePaddle/Paddle/pull/39193)) - - - determinant ([#40539](https://github.com/PaddlePaddle/Paddle/pull/40539)) - - - eigh ([#40213](https://github.com/PaddlePaddle/Paddle/pull/40213)) - - - size ([#39949](https://github.com/PaddlePaddle/Paddle/pull/39949), [#39712](https://github.com/PaddlePaddle/Paddle/pull/39712)) - - - shape ([#40248](https://github.com/PaddlePaddle/Paddle/pull/40248)) - - - reduce_sum([#37559](https://github.com/PaddlePaddle/Paddle/pull/37559), [#41295](https://github.com/PaddlePaddle/Paddle/pull/41295)) - - - reduce_prod ([#39844](https://github.com/PaddlePaddle/Paddle/pull/39844)) - - - histogram([#39496](https://github.com/PaddlePaddle/Paddle/pull/39496)) - - - meshgrid ([#41411](https://github.com/PaddlePaddle/Paddle/pull/41411)) - - - brelu ([#40385](https://github.com/PaddlePaddle/Paddle/pull/40385)) - - - hard_swish ([#40913](https://github.com/PaddlePaddle/Paddle/pull/40913)) - - - hard_shrink ([#40565](https://github.com/PaddlePaddle/Paddle/pull/40565)) - - - selu ([#39819](https://github.com/PaddlePaddle/Paddle/pull/39819)) - - - expand_v2 ([#39471](https://github.com/PaddlePaddle/Paddle/pull/39471)) - - - top_k_v2([#40064](https://github.com/PaddlePaddle/Paddle/pull/40064)) - - - expand_as_v2([#40373](https://github.com/PaddlePaddle/Paddle/pull/40373)) - - - swish ([#40913](https://github.com/PaddlePaddle/Paddle/pull/40913)) - - - hard_sigmoid ([#40626](https://github.com/PaddlePaddle/Paddle/pull/40626)) - - - exp, det, assign, gaussian_random, matrix_rank, eye, and deformable_conv. ([#41755](https://github.com/PaddlePaddle/Paddle/pull/41755), [#41737](https://github.com/PaddlePaddle/Paddle/pull/41737)) - -#### **New Dynamic Graph Execution Mechanism** - -To improve scheduling performance and custom development capability of the dynamic graph execution mechanism of the PaddlePaddle, we have reconstructed the underlying execution mechanism of the dynamic graph. With the new execution method, the PHI operator library can be used for efficient runtime execution. For the operators supported by the PHI operator library, switching to the new dynamic graph mode will get a significant improvement in scheduling performance. However, due to the huge workload required in the upgrade of the overall framework execution mechanism and this part of the work is coupled with a lot on the PHI operator library, we still do not use this execution method by default in this version. If you want to try it, you can switch to it by setting the environment variable `FLAGS_enable_eager_mode=1`.The details are as follows: - -- **Implementation of dynamic graph execution infrastructure, core components and mechanism**: By staticizing dynamic graph-related execution codes, the original homogeneous operators constructing converted to specific calling for different PHI APIs, thus greatly optimizing the scheduling overhead. ([#36059](https://github.com/PaddlePaddle/Paddle/pull/36059), [#37323](https://github.com/PaddlePaddle/Paddle/pull/37323), [#37556](https://github.com/PaddlePaddle/Paddle/pull/37556), [#37555](https://github.com/PaddlePaddle/Paddle/pull/37555), [#37478](https://github.com/PaddlePaddle/Paddle/pull/37478), [#37458](https://github.com/PaddlePaddle/Paddle/pull/37458), [#37479](https://github.com/PaddlePaddle/Paddle/pull/37479), [#37599](https://github.com/PaddlePaddle/Paddle/pull/37599), [#37659](https://github.com/PaddlePaddle/Paddle/pull/37659), [#37654](https://github.com/PaddlePaddle/Paddle/pull/37654), [#39200](https://github.com/PaddlePaddle/Paddle/pull/39200), [#39309](https://github.com/PaddlePaddle/Paddle/pull/39309), [#39319](https://github.com/PaddlePaddle/Paddle/pull/39319), [#39414](https://github.com/PaddlePaddle/Paddle/pull/39414), [#39504](https://github.com/PaddlePaddle/Paddle/pull/39504), [#39526](https://github.com/PaddlePaddle/Paddle/pull/39526), [#39878](https://github.com/PaddlePaddle/Paddle/pull/39878), [#39963](https://github.com/PaddlePaddle/Paddle/pull/39963)) - -- **New dynamic graph execution mechanism sub-function development and adaptation**: support more flexible and complete dynamic graph sub-functions such as hook, pylayer, double_grad, inplace, amp, etc. ([#41396](https://github.com/PaddlePaddle/Paddle/pull/41396), [#40400](https://github.com/PaddlePaddle/Paddle/pull/40400), [#40695](https://github.com/PaddlePaddle/Paddle/pull/40695), [#41043](https://github.com/PaddlePaddle/Paddle/pull/41043), [#40915](https://github.com/PaddlePaddle/Paddle/pull/40915), [#41104](https://github.com/PaddlePaddle/Paddle/pull/41104), [#41350](https://github.com/PaddlePaddle/Paddle/pull/41350), [#41209](https://github.com/PaddlePaddle/Paddle/pull/41209), [#40830](https://github.com/PaddlePaddle/Paddle/pull/40830), [#40891](https://github.com/PaddlePaddle/Paddle/pull/40891), [#36814](https://github.com/PaddlePaddle/Paddle/pull/36814), [#37377](https://github.com/PaddlePaddle/Paddle/pull/37377), [#37193](https://github.com/PaddlePaddle/Paddle/pull/37193), [#36965](https://github.com/PaddlePaddle/Paddle/pull/36965), [#37810](https://github.com/PaddlePaddle/Paddle/pull/37810), [#36837](https://github.com/PaddlePaddle/Paddle/pull/36837), [#38488](https://github.com/PaddlePaddle/Paddle/pull/38488), [#39282](https://github.com/PaddlePaddle/Paddle/pull/39282), [#39449](https://github.com/PaddlePaddle/Paddle/pull/39449), [#39531](https://github.com/PaddlePaddle/Paddle/pull/39531), [#39638](https://github.com/PaddlePaddle/Paddle/pull/39638), [#39674](https://github.com/PaddlePaddle/Paddle/pull/39674), [#39893](https://github.com/PaddlePaddle/Paddle/pull/39893), [#40170](https://github.com/PaddlePaddle/Paddle/pull/40170), [#40693](https://github.com/PaddlePaddle/Paddle/pull/40693), [#40937](https://github.com/PaddlePaddle/Paddle/pull/40937), [#41016](https://github.com/PaddlePaddle/Paddle/pull/41016), [#41051](https://github.com/PaddlePaddle/Paddle/pull/41051), [#41121](https://github.com/PaddlePaddle/Paddle/pull/41121), [#41198](https://github.com/PaddlePaddle/Paddle/pull/41198), [#41287](https://github.com/PaddlePaddle/Paddle/pull/41287), [#41380](https://github.com/PaddlePaddle/Paddle/pull/41380), [#41306](https://github.com/PaddlePaddle/Paddle/pull/41306), [#41387](https://github.com/PaddlePaddle/Paddle/pull/41387), [#40623](https://github.com/PaddlePaddle/Paddle/pull/40623), [#40945](https://github.com/PaddlePaddle/Paddle/pull/40945), [#39282](https://github.com/PaddlePaddle/Paddle/pull/39282), [#39449](https://github.com/PaddlePaddle/Paddle/pull/39449), [#38488](https://github.com/PaddlePaddle/Paddle/pull/38488)) - -- **Automatic code generation mechanism for new dynamic graph execution**: When we are trying to split the computation and scheduling logic of a large number of homogeneous operators into different specific scheduling logics, we find that it is a huge workload. So we introduce a new automatic code generation logic to generate code and thus simplify the runtime logic of dynamic graphs. Meanwhile, in order to adapt to the various types of runtime logic in the previous framework, we also use some complicated compilation techniques to obtain information at runtime to generate more accurate scheduling code. ([#37574](https://github.com/PaddlePaddle/Paddle/pull/37574), [#37575](https://github.com/PaddlePaddle/Paddle/pull/37575), [#37639](https://github.com/PaddlePaddle/Paddle/pull/37639), [#37723](https://github.com/PaddlePaddle/Paddle/pull/37723), [#37753](https://github.com/PaddlePaddle/Paddle/pull/37753), [#37812](https://github.com/PaddlePaddle/Paddle/pull/37812), [#37837](https://github.com/PaddlePaddle/Paddle/pull/37837), [#37910](https://github.com/PaddlePaddle/Paddle/pull/37910), [#37943](https://github.com/PaddlePaddle/Paddle/pull/37943), [#37992](https://github.com/PaddlePaddle/Paddle/pull/37992), [#37959](https://github.com/PaddlePaddle/Paddle/pull/37959), [#38017](https://github.com/PaddlePaddle/Paddle/pull/38017), [#37969](https://github.com/PaddlePaddle/Paddle/pull/37969), [#38160](https://github.com/PaddlePaddle/Paddle/pull/38160), [#38085](https://github.com/PaddlePaddle/Paddle/pull/38085), [#38562](https://github.com/PaddlePaddle/Paddle/pull/38562), [#38573](https://github.com/PaddlePaddle/Paddle/pull/38573), [#39192](https://github.com/PaddlePaddle/Paddle/pull/39192), [#39215](https://github.com/PaddlePaddle/Paddle/pull/39215), [#39355](https://github.com/PaddlePaddle/Paddle/pull/39355), [#39358](https://github.com/PaddlePaddle/Paddle/pull/39358), [#39328](https://github.com/PaddlePaddle/Paddle/pull/39328), [#39233](https://github.com/PaddlePaddle/Paddle/pull/39233), [#39628](https://github.com/PaddlePaddle/Paddle/pull/39628), [#39767](https://github.com/PaddlePaddle/Paddle/pull/39767), [#39743](https://github.com/PaddlePaddle/Paddle/pull/39743), [#39897](https://github.com/PaddlePaddle/Paddle/pull/39897), [#39797](https://github.com/PaddlePaddle/Paddle/pull/39797), [#39997](https://github.com/PaddlePaddle/Paddle/pull/39997), [#40058](https://github.com/PaddlePaddle/Paddle/pull/40058), [#40080](https://github.com/PaddlePaddle/Paddle/pull/40080), [#40107](https://github.com/PaddlePaddle/Paddle/pull/40107), [#39962](https://github.com/PaddlePaddle/Paddle/pull/39962), [#40132](https://github.com/PaddlePaddle/Paddle/pull/40132), [#40276](https://github.com/PaddlePaddle/Paddle/pull/40276), [#40266](https://github.com/PaddlePaddle/Paddle/pull/40266), [#40480](https://github.com/PaddlePaddle/Paddle/pull/40480), [#40482](https://github.com/PaddlePaddle/Paddle/pull/40482), [#40368](https://github.com/PaddlePaddle/Paddle/pull/40368), [#40650](https://github.com/PaddlePaddle/Paddle/pull/40650), [#40815](https://github.com/PaddlePaddle/Paddle/pull/40815), [#40907](https://github.com/PaddlePaddle/Paddle/pull/40907), [#40935](https://github.com/PaddlePaddle/Paddle/pull/40935), [#41089](https://github.com/PaddlePaddle/Paddle/pull/41089)) - -- **New dynamic graph execution mechanism accessed into the main framework and Integration test**: we currently use some environment variables to distinguish between static graph mode and dynamic graph mode (including new dynamic graph and old dynamic graph mode). We have adapted most logics of dynamic graphs in these modes. However, there are still a lot of problems being fixed. ([#37638](https://github.com/PaddlePaddle/Paddle/pull/37638), [#37643](https://github.com/PaddlePaddle/Paddle/pull/37643), [#37653](https://github.com/PaddlePaddle/Paddle/pull/37653), [#38314](https://github.com/PaddlePaddle/Paddle/pull/38314), [#38337](https://github.com/PaddlePaddle/Paddle/pull/38337), [#38338](https://github.com/PaddlePaddle/Paddle/pull/38338), [#39164](https://github.com/PaddlePaddle/Paddle/pull/39164), [#39326](https://github.com/PaddlePaddle/Paddle/pull/39326), [#40391](https://github.com/PaddlePaddle/Paddle/pull/40391), [#40201](https://github.com/PaddlePaddle/Paddle/pull/40201), [#40854](https://github.com/PaddlePaddle/Paddle/pull/40854), [#40887](https://github.com/PaddlePaddle/Paddle/pull/40887)) - -- **Update some judgment logics under dynamic graphs, to support fast execution paths for dynamic graphs in compatible forms**：([#40786](https://github.com/PaddlePaddle/Paddle/pull/40786)) - - - Non-static graph mode (current transition scheme): `_non_static_mode()`。 - - - Determined as new dynamic graph in dynamic graph mode (recommended judgment logic): `_in_dygrah_mode()`。 - - - Determined as old dynamic graph in dynamic graph mode (Not recommended. It will be deprecated in future versions): `_in_legacy_dygraph()`。 - - - Enable old dynamic graph and disable new dynamic graph in dynamic graph mode: `_enable_legacy_dygraph()` or exit `_test_eager_guard()`。 - - - Enable new dynamic graph and disable old dynamic graph in dynamic graph mode: `_disable_legacy_dygraph()` or with `with _test_eager_guard()`。 - - - Determine in new dynamic graph in static or dynamic graph mode: `_in_eager_without_dygraph_check()`。 - -- **Support inplace after dynamic graph reconstruction**: input and output are the same Tensor. - - - Adapt the inplace strategy for dynamic graph reconstruction intermediate states. ([#40400](https://github.com/PaddlePaddle/Paddle/pull/40400)) - - - Adapt the inplace strategy to the final state of the dynamic graph reconstruction. ([#40695](https://github.com/PaddlePaddle/Paddle/pull/40695)) - - - Add inplace strategy to PyLayer function after dynamical graph reconstruction. ([#41043](https://github.com/PaddlePaddle/Paddle/pull/41043)) - - - Add inplace strategy for Tensor's setitem function after dynamical graph reconstruction. ([#40915](https://github.com/PaddlePaddle/Paddle/pull/40915)) - - - Add `_reset_grad_inplace_version` interface after dynamic graph reconstruction, to set the inplace version of the Tensor's gradient to 0. ([#41101](https://github.com/PaddlePaddle/Paddle/pull/41101)) - - - If the value of the forward Tensor is not needed during the inverse computation (no need buffer property), the inplace version detection operation is not needed for that Tensor. For Tensor with no_need_buffer, skip the inplace version check. ([#41350](https://github.com/PaddlePaddle/Paddle/pull/41350)) - - - Unify error messages for inplace version checks after and before reconstruction of dynamic graphs. ([#41209](https://github.com/PaddlePaddle/Paddle/pull/41209)) - -- **Support view strategy after dynamical graph reconstruction**: input and output Tensor share underlying data. - - - Adapt the view strategy for dynamic graph reconstruction intermediate states. Include `reshape`, `squeeze`, `unsqueeze`, and `flatten` APIs. ([#40830](https://github.com/PaddlePaddle/Paddle/pull/40830)) - - - Adapt the view strategy for dynamic graph reconstruction final state. Include `reshape` API. ([#40891](https://github.com/PaddlePaddle/Paddle/pull/40891)) - -- **Add support for weakref on the python side of the new dynamic graph eager Tensor.** ([#41797](https://github.com/PaddlePaddle/Paddle/pull/41797)) - -- **Enhance the new dynamic graph DoubleGrad function** to support the basic DoubleGrad feature. ([#41893](https://github.com/PaddlePaddle/Paddle/pull/41893), [#41894](https://github.com/PaddlePaddle/Paddle/pull/41894), [#41895](https://github.com/PaddlePaddle/Paddle/pull/41895)) - -- **Add `core.eager.StringTensor` interface**, to support the construction of StringTensor on python side and the use of the StringTensor related APIs. ([#41039](https://github.com/PaddlePaddle/Paddle/pull/41039)) - -- **Add `_grad_name` and `_grad_value`*to `core.eager.Tensor` to return the name and value of a gradient. ([#41990](https://github.com/PaddlePaddle/Paddle/pull/41990)) - -- **Add the processing of the no_need_buffer attribute for dynamic graph intermediate state.** The Tensor with the no_need_buffer attribute is skipped in the inplace backward check operation. ([#41720](https://github.com/PaddlePaddle/Paddle/pull/41720)) - - -#### **New Static Graph Executor** - -In order to solve the problem that the original static graph executor of the PaddlePaddle is not good enough for scheduling in some scenarios and it is not easy to use multiple streams, we have implemented a new static graph executor with superior performance. It is easy to take advantage of the asynchronous scheduling capabilities of multi-streams and multi-threads. The new executor is a compatible upgrade of the original executor. At present, it is used by default in single-card scenarios. Users do not need to make any changes in the training codes. It can be used automatically. Of course, we also provide an interface to switch back to the original executor. Users can switch back to the original executor by setting the environment variable: `FLAGS_USE_STANDALONE_EXECUTOR=false`. ([#41179](https://github.com/PaddlePaddle/Paddle/pull/41179)) The main contents are as follows. - -- Basic components: High-performance thread pool for multi-threaded scheduling in the executor ([#35470](https://github.com/PaddlePaddle/Paddle/pull/35470), [#35930](https://github.com/PaddlePaddle/Paddle/pull/35930), [#36030](https://github.com/PaddlePaddle/Paddle/pull/36030), [#36480](https://github.com/PaddlePaddle/Paddle/pull/36480), [#36688](https://github.com/PaddlePaddle/Paddle/pull/36688), [#36740](https://github.com/PaddlePaddle/Paddle/pull/36740), [#38335](https://github.com/PaddlePaddle/Paddle/pull/38335), [#40770](https://github.com/PaddlePaddle/Paddle/pull/40770)) and thread co-op component ([#38779](https://github.com/PaddlePaddle/Paddle/pull/38779), [#40876](https://github.com/PaddlePaddle/Paddle/pull/40876), [#40912](https://github.com/PaddlePaddle/Paddle/pull/40912)). There is the timely memory recovery after operator execution ([#37642](https://github.com/PaddlePaddle/Paddle/pull/37642), [#39617](https://github.com/PaddlePaddle/Paddle/pull/39617), [#40859](https://github.com/PaddlePaddle/Paddle/pull/40859)). There is the new dependency analysis algorithm for parallel executor ([#37231](https://github.com/PaddlePaddle/Paddle/pull/37231)) etc. - -- Scheduling logic: Optimize the scheduling method of operator in the executor. Support multi-stream multi-threaded asynchronous scheduling mechanism. Change transforms such as data type, device, and layout to the operator scheduling to improve performance. Support caching the selection of operator Kernel. Support the selection of new PHI operator. ([#35024](https://github.com/PaddlePaddle/Paddle/pull/35024), [#34922](https://github.com/PaddlePaddle/Paddle/pull/34922), [#35711](https://github.com/PaddlePaddle/Paddle/pull/35711), [#35928](https://github.com/PaddlePaddle/Paddle/pull/35928), [#39458](https://github.com/PaddlePaddle/Paddle/pull/39458)，[#36899](https://github.com/PaddlePaddle/Paddle/pull/36899))。 - -- Interface compatibility: Compatible with the user interface and functionality of the original executor, such as alignment with python interface Executor.run(), support for managing Tensor in Scope, etc. This ensures that users can switch to the new executor without perception. ([#37278](https://github.com/PaddlePaddle/Paddle/pull/37278), [#37379](https://github.com/PaddlePaddle/Paddle/pull/37379), [#37445](https://github.com/PaddlePaddle/Paddle/pull/37445), [#37510](https://github.com/PaddlePaddle/Paddle/pull/37510), [#40955](https://github.com/PaddlePaddle/Paddle/pull/40955), [#41778](https://github.com/PaddlePaddle/Paddle/pull/41178), [#41058](https://github.com/PaddlePaddle/Paddle/pull/41058), [#38584](https://github.com/PaddlePaddle/Paddle/pull/38584), [#37957](https://github.com/PaddlePaddle/Paddle/pull/37957), [#37672](https://github.com/PaddlePaddle/Paddle/pull/37672), [#37474](https://github.com/PaddlePaddle/Paddle/pull/37474), [#37085](https://github.com/PaddlePaddle/Paddle/pull/37085), [#37061](https://github.com/PaddlePaddle/Paddle/pull/37061), [#36945](https://github.com/PaddlePaddle/Paddle/pull/36945)) - -- Enhance debugging and error reporting in multi-threaded scenarios by capturing error reports from sub-threads and throwing them uniformly in the main thread. This can improve user experience. ([#36692](https://github.com/PaddlePaddle/Paddle/pull/36692)，[#36802](https://github.com/PaddlePaddle/Paddle/pull/36802)) - -- Fix the bug with the new executor communication flow resetting stream cache information in the allocator, to reduce RecordStream overhead in cross-stream scenarios. This improves performance of DeepFM models by about 8% after optimization. ([#42046](https://github.com/PaddlePaddle/Paddle/pull/42046)) - -- Optimize the dependency analysis method between new executor operators to improve runtime performance. Establish correct dependencies for send/recv communication operators to support pipeline parallel. ([#42009](https://github.com/PaddlePaddle/Paddle/pull/42009)) - - - -#### **Distributed Training** - -- Basic functions of multi-machine multi-card parallel training based on collective communication - - - Add support for elastic training, enables scaling up and down the number of workers, enables training process resuming when node failure，to improve the fault tolerance of distributed training. ([#36684](https://github.com/PaddlePaddle/Paddle/pull/36684), [#37177](https://github.com/PaddlePaddle/Paddle/pull/37177), [#37781](https://github.com/PaddlePaddle/Paddle/pull/37781)) - - - Refactor launch startup module, add `master` collaboration and node number `nnodes` definition, to improve the ease of using the distributed startup. ([#40086](https://github.com/PaddlePaddle/Paddle/pull/40086), [#40568](https://github.com/PaddlePaddle/Paddle/pull/40568), [#40782](https://github.com/PaddlePaddle/Paddle/pull/40782), [#40844](https://github.com/PaddlePaddle/Paddle/pull/40844), [#40936](https://github.com/PaddlePaddle/Paddle/pull/40936), [#41190](https://github.com/PaddlePaddle/Paddle/pull/41190), [#41314](https://github.com/PaddlePaddle/Paddle/pull/41314)) - - - Add support for GPU/NPU/XPU multi-hardware heterogeneous training. ([#37613](https://github.com/PaddlePaddle/Paddle/pull/37613), [#37998](https://github.com/PaddlePaddle/Paddle/pull/37998)) - - - Add fleet_executor asynchronous pipeline executor. ([#36966](https://github.com/PaddlePaddle/Paddle/pull/36966), [#37049](https://github.com/PaddlePaddle/Paddle/pull/37049), [#37087](https://github.com/PaddlePaddle/Paddle/pull/37087), [#37126](https://github.com/PaddlePaddle/Paddle/pull/37126), [#37150](https://github.com/PaddlePaddle/Paddle/pull/37150), [#37203](https://github.com/PaddlePaddle/Paddle/pull/37203), [#37167](https://github.com/PaddlePaddle/Paddle/pull/37167), [#37282](https://github.com/PaddlePaddle/Paddle/pull/37282), [#37319](https://github.com/PaddlePaddle/Paddle/pull/37319), [#37462](https://github.com/PaddlePaddle/Paddle/pull/37462), [#37507](https://github.com/PaddlePaddle/Paddle/pull/37507), [#37533](https://github.com/PaddlePaddle/Paddle/pull/37533), [#37576](https://github.com/PaddlePaddle/Paddle/pull/37576), [#37605](https://github.com/PaddlePaddle/Paddle/pull/37605), [#37691](https://github.com/PaddlePaddle/Paddle/pull/37691), [#37742](https://github.com/PaddlePaddle/Paddle/pull/37742), [#37783](https://github.com/PaddlePaddle/Paddle/pull/37783), [#37809](https://github.com/PaddlePaddle/Paddle/pull/37809), [#37862](https://github.com/PaddlePaddle/Paddle/pull/37862), [#37882](https://github.com/PaddlePaddle/Paddle/pull/37882), [#37934](https://github.com/PaddlePaddle/Paddle/pull/37934), [#38024](https://github.com/PaddlePaddle/Paddle/pull/38024), [#38083](https://github.com/PaddlePaddle/Paddle/pull/38083), [#38164](https://github.com/PaddlePaddle/Paddle/pull/38164), [#38261](https://github.com/PaddlePaddle/Paddle/pull/38261), [#38290](https://github.com/PaddlePaddle/Paddle/pull/38290), [#40607](https://github.com/PaddlePaddle/Paddle/pull/40607), [#37093](https://github.com/PaddlePaddle/Paddle/pull/37093), [#37106](https://github.com/PaddlePaddle/Paddle/pull/37106), [#37143](https://github.com/PaddlePaddle/Paddle/pull/37143), [#37338](https://github.com/PaddlePaddle/Paddle/pull/37338), [#37376](https://github.com/PaddlePaddle/Paddle/pull/37376), [#37485](https://github.com/PaddlePaddle/Paddle/pull/37485), [#37531](https://github.com/PaddlePaddle/Paddle/pull/37531), [#37623](https://github.com/PaddlePaddle/Paddle/pull/37623), [#37693](https://github.com/PaddlePaddle/Paddle/pull/37693), [#37755](https://github.com/PaddlePaddle/Paddle/pull/37755), [#37807](https://github.com/PaddlePaddle/Paddle/pull/37807), [#37889](https://github.com/PaddlePaddle/Paddle/pull/37889), [#38420](https://github.com/PaddlePaddle/Paddle/pull/38420), [#38539](https://github.com/PaddlePaddle/Paddle/pull/38539), [#36892](https://github.com/PaddlePaddle/Paddle/pull/36892), [#37084](https://github.com/PaddlePaddle/Paddle/pull/37084), [#37158](https://github.com/PaddlePaddle/Paddle/pull/37158), [#37361](https://github.com/PaddlePaddle/Paddle/pull/37361), [#37509](https://github.com/PaddlePaddle/Paddle/pull/37509), [#37603](https://github.com/PaddlePaddle/Paddle/pull/37603), [#37703](https://github.com/PaddlePaddle/Paddle/pull/37703), [#37824](https://github.com/PaddlePaddle/Paddle/pull/37824), [#38114](https://github.com/PaddlePaddle/Paddle/pull/38114), [#38322](https://github.com/PaddlePaddle/Paddle/pull/38322), [#38535](https://github.com/PaddlePaddle/Paddle/pull/38535), [#38650](https://github.com/PaddlePaddle/Paddle/pull/38650), [#38709](https://github.com/PaddlePaddle/Paddle/pull/38709), [#38799](https://github.com/PaddlePaddle/Paddle/pull/38799), [#38839](https://github.com/PaddlePaddle/Paddle/pull/38839), [#38904](https://github.com/PaddlePaddle/Paddle/pull/38904)) - - - Add distributed inference function for large-scale model. ([#38795](https://github.com/PaddlePaddle/Paddle/pull/38795), [#39012](https://github.com/PaddlePaddle/Paddle/pull/39012), [#39032](https://github.com/PaddlePaddle/Paddle/pull/39032), [#39076](https://github.com/PaddlePaddle/Paddle/pull/39076), [#39194](https://github.com/PaddlePaddle/Paddle/pull/39194), [#39207](https://github.com/PaddlePaddle/Paddle/pull/39207), [#39241](https://github.com/PaddlePaddle/Paddle/pull/39241), [#39603](https://github.com/PaddlePaddle/Paddle/pull/39603), [#39758](https://github.com/PaddlePaddle/Paddle/pull/39758), [#39992](https://github.com/PaddlePaddle/Paddle/pull/39992)). - -- Dynamic graph hybrid parallelism - - - Reconstruct `paddle.distributed.fleet.utils.recompute`, to support new dynamic computational graph. ([#41396](https://github.com/PaddlePaddle/Paddle/pull/41396)) - - - Add pure FP16 training to support data parallelism. ([#36420](https://github.com/PaddlePaddle/Paddle/pull/36420)) - - - Add MoE (Mixture of Experts) parallel strategy, to support large-scale MoE model training. ([#41092](https://github.com/PaddlePaddle/Paddle/pull/41092), [#40895](https://github.com/PaddlePaddle/Paddle/pull/40895), [#40850](https://github.com/PaddlePaddle/Paddle/pull/40580), [#39224](https://github.com/PaddlePaddle/Paddle/pull/39224)) - - - Add GroupSharded parallel strategy. Support stage1, stage2, stage3, and it supports synchronous and asynchronous communication. It can be used together with the basic function combinations such as Recompute, AMP O1\O2, Offload, GroupShardedClipGrad, and GroupShardedScaler. ([#37489](https://github.com/PaddlePaddle/Paddle/pull/37489), [#37568](https://github.com/PaddlePaddle/Paddle/pull/37568), [#37707](https://github.com/PaddlePaddle/Paddle/pull/37707), [#37836](https://github.com/PaddlePaddle/Paddle/pull/37836), [#37947](https://github.com/PaddlePaddle/Paddle/pull/37947), [#38151](https://github.com/PaddlePaddle/Paddle/pull/38151), [#38407](https://github.com/PaddlePaddle/Paddle/pull/38407), [#38052](https://github.com/PaddlePaddle/Paddle/pull/38052), [#39112](https://github.com/PaddlePaddle/Paddle/pull/39112), [#38989](https://github.com/PaddlePaddle/Paddle/pull/38989), [#39171](https://github.com/PaddlePaddle/Paddle/pull/39171), [#39285](https://github.com/PaddlePaddle/Paddle/pull/39285), [#39334](https://github.com/PaddlePaddle/Paddle/pull/39334), [#39397](https://github.com/PaddlePaddle/Paddle/pull/39397), [#39581](https://github.com/PaddlePaddle/Paddle/pull/39581), [#39668](https://github.com/PaddlePaddle/Paddle/pull/39668), [#40129](https://github.com/PaddlePaddle/Paddle/pull/40129), [#40396](https://github.com/PaddlePaddle/Paddle/pull/40396), [#40488](https://github.com/PaddlePaddle/Paddle/pull/40488), [#40601](https://github.com/PaddlePaddle/Paddle/pull/40601)，[#37725](https://github.com/PaddlePaddle/Paddle/pull/37725)，[#37904](https://github.com/PaddlePaddle/Paddle/pull/37904), [#38064](https://github.com/PaddlePaddle/Paddle/pull/38064)) - -- Static graph hybrid parallelism - - - Add `scale_gradient` flag bit to `gradient_scale_configs` to control the position where the gradient aggregation operation averages the gradients under pipeline parallelism. ([#36384](https://github.com/PaddlePaddle/Paddle/pull/36384)) - - - Under tensor parallelism, the dropout op supports the settings of deterministic random seed generators, to ensure random consistency for non-distributed variables and randomness of distributed variables. ([#36228](https://github.com/PaddlePaddle/Paddle/pull/36228)) - - - NPU hybrid parallelism supports Offload, with saving 40% of NPU memory. ([#37224](https://github.com/PaddlePaddle/Paddle/pull/37224)) - - - Add `force_cpu` optional parameter to the seed op, to allow dropout to read seed values directly from CPU. ([#35820](https://github.com/PaddlePaddle/Paddle/pull/35820)) - - - Improve the Automatic Sparsity (ASP) sharding strategy and support the selection of sharding strategy according to the program. ([#40028](https://github.com/PaddlePaddle/Paddle/pull/40028)) - -- Automatic parallel - - - Add the process restart (relaunch) after automatic mapping between logical processes and physical devices. ([#37523](https://github.com/PaddlePaddle/Paddle/pull/37523), [#37326](https://github.com/PaddlePaddle/Paddle/pull/37326)) - - - Improve the underlying mechanism and interface for automatic parallel to facilitate the unification of modules and add the optimized pass. ([#36617](https://github.com/PaddlePaddle/Paddle/pull/36617), [#38132](https://github.com/PaddlePaddle/Paddle/pull/38132)) - - - Add unified resource representation, to support for automatic mapping between logical processes and physical devices. ([#37091](https://github.com/PaddlePaddle/Paddle/pull/37091), [#37482](https://github.com/PaddlePaddle/Paddle/pull/37482), [#37094](https://github.com/PaddlePaddle/Paddle/pull/37094)) - - - Improve the distributed attribute complementation for the backward and update parts of the computation graph. ([#36744](https://github.com/PaddlePaddle/Paddle/pull/36744)) - - - Add data slicing function. ([#36055](https://github.com/PaddlePaddle/Paddle/pull/36055)) - - - Add tensor resharding function to reshard the tensor according to the distributed properties of the tensor and operator. ([#40865](https://github.com/PaddlePaddle/Paddle/pull/40865), [#41106](https://github.com/PaddlePaddle/Paddle/pull/41106)) - - - Add the automatic conversion pass of distributed parameters when the number of resources or parallel policy changes. ([#40434](https://github.com/PaddlePaddle/Paddle/pull/40434)) - - - Add GradientMerge pass to reduce the number of communications and improve training efficiency. ([#38259](https://github.com/PaddlePaddle/Paddle/pull/38259), [#40737](https://github.com/PaddlePaddle/Paddle/pull/40737)) - - - Add Recompute pass to reduce the activation memory storage. ([#38920](https://github.com/PaddlePaddle/Paddle/pull/38920)) - - - Add Sharding optimization pass, to support p-g-os 3 stage optimization. ([#38502](https://github.com/PaddlePaddle/Paddle/pull/38502)) - - - Add AMP + FP16 optimization pass. ([#38764](https://github.com/PaddlePaddle/Paddle/pull/38764), [#40615](https://github.com/PaddlePaddle/Paddle/pull/40615)) - - - Add fused QKV parallelization for Transformer class model. ([#39080](https://github.com/PaddlePaddle/Paddle/pull/39080)) - - - Improve the sharding propagation for while op to ensure convergence of the fix-point algorithm. ([#39939](https://github.com/PaddlePaddle/Paddle/pull/39939), [#39086](https://github.com/PaddlePaddle/Paddle/pull/39086), [#39014](https://github.com/PaddlePaddle/Paddle/pull/39014)) - - - Support training and inference for sub-block and while op control flow. ([#39612](https://github.com/PaddlePaddle/Paddle/pull/39612), [#39895](https://github.com/PaddlePaddle/Paddle/pull/39895), [#40077](https://github.com/PaddlePaddle/Paddle/pull/40077)) - -- Parameter Server - - - Add NaN/Inf value checking tool under GPUPS. ([#38131](https://github.com/PaddlePaddle/Paddle/pull/38131)) - - - Under GPUPS, add set_date interface to adapt incremental training. ([#36194](https://github.com/PaddlePaddle/Paddle/pull/36194)) - - - Under GPUPS, add asynchronous release dataset function. ([#37790](https://github.com/PaddlePaddle/Paddle/pull/37790)) - - - Under GPUPS, support the Dump parameters and intermediate layers([#36157](https://github.com/PaddlePaddle/Paddle/pull/36157))； - - - Under GPUPS, support the optimizer parameter configuration. ([#39783](https://github.com/PaddlePaddle/Paddle/pull/39783), [#39849](https://github.com/PaddlePaddle/Paddle/pull/39849)) - - - Under the Unified Parameter Server, refactor the base classes of each module such as communication and storage, to improve the ease of secondary development of each module. ([#41207](https://github.com/PaddlePaddle/Paddle/pull/41207), [#41022](https://github.com/PaddlePaddle/Paddle/pull/41022), [#40702](https://github.com/PaddlePaddle/Paddle/pull/40702), [#39341](https://github.com/PaddlePaddle/Paddle/pull/39341) [#39377](https://github.com/PaddlePaddle/Paddle/pull/39377), [#39191](https://github.com/PaddlePaddle/Paddle/pull/39191), [#39064](https://github.com/PaddlePaddle/Paddle/pull/39064)) - - - Add evaluation metrics module under the Unified Parameter Server, to support AUC/WuAUC/MaskAUC and other evaluation metrics calculation and customizable extensions. ([#38789](https://github.com/PaddlePaddle/Paddle/pull/38789)) - - - Supports XPU parameter server training on KUNLUNXIN 2. ([#41917](https://github.com/PaddlePaddle/Paddle/pull/41917), [#42266](https://github.com/PaddlePaddle/Paddle/pull/42266), [#41916](https://github.com/PaddlePaddle/Paddle/pull/41916)) - -#### Profiler - -- Add the performance analysis module `paddle.profiler` in the Python layer: Provide the ability to collect, export, and count performance data during the training push. ([#40065](https://github.com/PaddlePaddle/Paddle/pull/40065), [#40357](https://github.com/PaddlePaddle/Paddle/pull/40357), [#40888](https://github.com/PaddlePaddle/Paddle/pull/40888)) - - - `paddle.profiler.Profiler`: performance analyzer, interface for user interaction. ([#41029](https://github.com/PaddlePaddle/Paddle/pull/41029), [#41524](https://github.com/PaddlePaddle/Paddle/pull/41524), [#41157](https://github.com/PaddlePaddle/Paddle/pull/41157), [#40249](https://github.com/PaddlePaddle/Paddle/pull/40249), [#40111](https://github.com/PaddlePaddle/Paddle/pull/40111), [#39964](https://github.com/PaddlePaddle/Paddle/pull/39964), [#40133](https://github.com/PaddlePaddle/Paddle/pull/40133)) - - - `paddle.profiler.RecordEvent`: provide custom punches to record time. ([#39693](https://github.com/PaddlePaddle/Paddle/pull/39693), [#39694](https://github.com/PaddlePaddle/Paddle/pull/39694), [#39695](https://github.com/PaddlePaddle/Paddle/pull/39695), [#39675](https://github.com/PaddlePaddle/Paddle/pull/39675),[#41445](https://github.com/PaddlePaddle/Paddle/pull/41445), [#41132](https://github.com/PaddlePaddle/Paddle/pull/41132)) - - - `paddle.profiler.ProfilerTarget`: specify the target device for performance analysis. - - - `paddle.profiler.ProfilerState`: indicate the state of the performance analyzer. - - - `paddle.profiler.SortedKeys`: specify the sorting method of the data within the statistics form. - - - `paddle.profiler.make_scheduler`: the scheduler generating the performance analyzer state and implement the periodic control of the collection scope. - - - `paddle.profiler.export_chrome_tracing`: save performance data to a google chrome tracing file viewable by the chrome://tracing plugin. ([#39316](https://github.com/PaddlePaddle/Paddle/pull/39316), [#39984](https://github.com/PaddlePaddle/Paddle/pull/39984), [#41029](https://github.com/PaddlePaddle/Paddle/pull/41029)) - - - `paddle.profiler.export_protobuf`: save performance data to a protobuf file represented by internal structure. ([#39519](https://github.com/PaddlePaddle/Paddle/pull/39519), [#39109](https://github.com/PaddlePaddle/Paddle/pull/39109), [#39474](https://github.com/PaddlePaddle/Paddle/pull/39474)) - - - `paddle.profiler.load_profiler_result`: load the performance data saved to a protobuf file. - - - `paddle.profiler.Profiler` generate statistics for data reading, step overhead and throughput for the model training by specifying the `timer_only` parameter. ([#40386](https://github.com/PaddlePaddle/Paddle/pull/40386)) - -- Refactor Profiler underlying infrastructure in C++ layer - - - Refactor the Profiler's controller architecture. ([#38826](https://github.com/PaddlePaddle/Paddle/pull/38826), [#39230](https://github.com/PaddlePaddle/Paddle/pull/39230), [#39779](https://github.com/PaddlePaddle/Paddle/pull/39779) ) - - - Add Host Tracer to collect host-side performance metrics. ([#37629](https://github.com/PaddlePaddle/Paddle/pull/39629), [#37766](https://github.com/PaddlePaddle/Paddle/pull/37766), [#37944](https://github.com/PaddlePaddle/Paddle/pull/37944), [#38280](https://github.com/PaddlePaddle/Paddle/pull/38280), [#39975](https://github.com/PaddlePaddle/Paddle/pull/39975), [#40460](https://github.com/PaddlePaddle/Paddle/pull/40460)) - - - Add CUDA Tracer to collect device-side performance metrics. ([#39488](https://github.com/PaddlePaddle/Paddle/pull/39488)) - - - Profiler support for grading. ([#39926](https://github.com/PaddlePaddle/Paddle/pull/39926)) - -- Modify the name and type of logging for op under new dynamic graph. ([#41771](https://github.com/PaddlePaddle/Paddle/pull/41771/) - -- Add Kernel running statistics into profilers' summarization and optimize the summarization. ([#41989](https://github.com/PaddlePaddle/Paddle/pull/41989) - -- Remove side-effect to performance in forward computing forward when Profiler is off. ([#42142](https://github.com/PaddlePaddle/Paddle/pull/42142)) - -#### **CINN compiler adoption** - -With the recent development of PaddlePaddle's compiler, a.k.a, CINN([GitHub - PaddlePaddle/CINN: Compiler Infrastructure for Neural Networks](https://github.com/PaddlePaddle/CINN)), paddle framework has also been changed to adapt the compiler CINN features. These include the subgraph management related functions for the Paddle-CINN runtime, optimization of memory and speed performance, and bug fixing during development. - -- Functions developed: - - - Subgraph op related functions: - - - Add the function to find and generate CINN subgraphs from computational graphs. ([#36345](https://github.com/PaddlePaddle/Paddle/pull/36345)) - - - Add cinn_launch op as a runtime entry point to CINN. It is responsible for scheduling CINN to compile the subgraph, to initialize the data, and to execute the generated kernels. ([#36600](https://github.com/PaddlePaddle/Paddle/pull/36600)) - - - Add a helper class `CinnLaunchContext` to the kernel implementation of cinn_launch op to manage the intermediate data for compiling and running subgraphs, to improve scalability and code readability. ([#37938](https://github.com/PaddlePaddle/Paddle/pull/37938)) - - - Add additional fetch nodes to CINN subgraphs, thus ensuring that CINN external nodes can fetch the values of variables. ([#37172](https://github.com/PaddlePaddle/Paddle/pull/37172), [#37190](https://github.com/PaddlePaddle/Paddle/pull/37190)) - - - Add the function to symbolize a CINN subgraph, which is used to topologically sort the subgraphs and return the CINN execution sequence. ([#36417](https://github.com/PaddlePaddle/Paddle/pull/36417) - - - Add `CinnCompiler` class for involking subgraphs in the CINN compiled graph that can be replaced by using CINN operators. ([#36562](https://github.com/PaddlePaddle/Paddle/pull/36562), [#36975](https://github.com/PaddlePaddle/Paddle/pull/36975)) - - - Add the interface to CINN symbolization class to get the names of subgraph fetched variables to prevent fetched variables from being eliminated in compilation optimizations. ([#37218](https://github.com/PaddlePaddle/Paddle/pull/37218)) - - - Checking, debugging, and PI changes related: - - - Synchronize the update of NetBuilder API name changes in CINN. ([#40392](https://github.com/PaddlePaddle/Paddle/pull/40392)) - - - Add necessary log information to Paddle-CINN for better debugging. ([#36867](https://github.com/PaddlePaddle/Paddle/pull/36867)) - - - Add the bidirectional conversion function between Paddle desc and CINN desc. ([#36100](https://github.com/PaddlePaddle/Paddle/pull/36100)) - - - The operator implemented in CINN may not use some input variables compared to Paddle. Therefore, remove the check that the input variables must be used in the cinn_launch op. ([#37119](https://github.com/PaddlePaddle/Paddle/pull/37119)) - - - Added cinn_instruction_run op for invoking CINN to execute a single generation instruction, facilitating the construction of scheduling run subgraphs on the Paddle side. ([#39435](https://github.com/PaddlePaddle/Paddle/pull/39435), [#39576](https://github.com/PaddlePaddle/Paddle/pull/39576)) - - - Add control macros to Paddle for CUDA/CUBLAS/MKL/CINN pass application required to compile CINN. ([#37066](https://github.com/PaddlePaddle/Paddle/pull/37066), [#36660](https://github.com/PaddlePaddle/Paddle/pull/36660)) - - - Add two control flags FLAGS_allow_cinn_ops and FLAGS_deny_cinn_ops to control the categories of CINN operators used to replace native operators during Paddle training. ([#36842](https://github.com/PaddlePaddle/Paddle/pull/36842)) - -- Performance optimization: - - - Speed optimization - - - Optimize the computational time consumed by CinnCacheKey. ([#37786](https://github.com/PaddlePaddle/Paddle/pull/37786), [#37317](https://github.com/PaddlePaddle/Paddle/pull/37317)) - - - Cache variable scope for CINN compiled subgraphs to reduce runtime parameter construction overhead. ([#37983](https://github.com/PaddlePaddle/Paddle/pull/37983)) - - - Utilize CINN's auto-tuning in case of subgraph compilation, could be enabled by flag, for further tuning of training performance. ([#41795](https://github.com/PaddlePaddle/Paddle/pull/41795)) - - - Refactor the correctness check of compilation results in case of subgraph compilation to avoid repeated checks at runtime and reduce the scheduling overhead. ([#41777](https://github.com/PaddlePaddle/Paddle/pull/41777)) - - - Enable TransposeFolding and GemmRewriter optimization passes by default in Paddle-CINN training. ([#41084](https://github.com/PaddlePaddle/Paddle/pull/41084)) - - - Pass the cuda stream created in Paddle into CINN so that Paddle and CINN can use the same CUDA stream in cuda computing. ([#37337](https://github.com/PaddlePaddle/Paddle/pull/37337)) - - - Move CINN optimization pass application logic from Paddle to CINN. ([#42047](https://github.com/PaddlePaddle/Paddle/pull/42047), [#42070](https://github.com/PaddlePaddle/Paddle/pull/42070)) - - - Device memory optimization - - - Add NoNeedBufferVars to cinn_launch op to declare a list of input variables that do not require a buffer, so that the memory can be freed in advance. ([#38367](https://github.com/PaddlePaddle/Paddle/pull/38367)) - - - Pass in reference count information for external variables to the subgraph, so that subgraphs within cinn_launch can reuse memory optimization passes and reduce the memory overhead in using CINN. ([#39209](https://github.com/PaddlePaddle/Paddle/pull/39209), [#39622](https://github.com/PaddlePaddle/Paddle/pull/39622)) - - - Add the function to convert a collection of executable instructions generated by CINN compilation to a Paddle Graph, supporting reuse of the Paddle scheduler and memory optimization pass, further reducing the memory overhead in using CINN. ([#39724](https://github.com/PaddlePaddle/Paddle/pull/39724), [#39911](https://github.com/PaddlePaddle/Paddle/pull/39911)) - - - Add Kernel of cinn_instruction_run op, to support dynamic device memory requests based on data types inferred from compilation results. ([#40920](https://github.com/PaddlePaddle/Paddle/pull/40920)) - -- Bug fixing: - - - Fix and optimize the generation logic of CINN subgraphs. ([#36503](https://github.com/PaddlePaddle/Paddle/pull/36503)) - - - Fix the bug that Paddle-CINN does not support no-input subgraphs. ([#40814](https://github.com/PaddlePaddle/Paddle/pull/40814)) - - - Fix an error reported due to CINN not being able to handle useless outputs in operators such as batch_norm. ([#36996](https://github.com/PaddlePaddle/Paddle/pull/36996)) - - - Fix several bugs in CINN subgraph partitioning and symbolization, and solve problems with Paddle training accessing the CINN. ([#36739](https://github.com/PaddlePaddle/Paddle/pull/36739), [#36698](https://github.com/PaddlePaddle/Paddle/pull/36698) ) - - - CINN does not yet support the control flow yet. Add logic to skip control flow when encountered. ([#40812](https://github.com/PaddlePaddle/Paddle/pull/40812)) - -#### **Other** - -- Model quantization - - - Upgrade quantization storage format to unify quantization formats for dynamic and static graphs. ([#41041](https://github.com/PaddlePaddle/Paddle/pull/41041)) - - - Add new post training quantization (PTQ): EMD and Adaround. ([#40421](https://github.com/PaddlePaddle/Paddle/pull/40421), [#38460](https://github.com/PaddlePaddle/Paddle/pull/38460)) - - - Support to quantize more operations in PTQ and QAT, such as crop, split, ab, unsqueeze etc. ([#40083](https://github.com/PaddlePaddle/Paddle/pull/40083)) - - - Support to quantize operators in control flow. ([#37498](https://github.com/PaddlePaddle/Paddle/pull/37498)) - - - Support quantization of matmul_v2 operator. ([#36469](https://github.com/PaddlePaddle/Paddle/pull/36469)) - - - Add support for quantized matmul_v2 inference on TensorRT. ([#36594](https://github.com/PaddlePaddle/Paddle/pull/36594)) - -- CUDA memory optimization - - - Implement multi-stream safe Allocator to support safe and efficient use of CUDA memory in asynchronous computing scenarios. ([#37290](https://github.com/PaddlePaddle/Paddle/pull/37290)) - - - Add new APIs (paddle.device.cuda.max_memory_allocated, paddle.device.cuda.max_memory_reserved, paddle.device.cuda.memory_allocated and paddle.device.cuda.memory_reserved) for GPU memory monitoring in runtime. ([#38657](https://github.com/PaddlePaddle/Paddle/pull/38657)) - - - Support allocate CUDA Managed Memory to train super large models in memory-constrained scenarios. ([#39075](https://github.com/PaddlePaddle/Paddle/pull/39075)) - - - Add GetBasePtr interface in C++ to get device address created with *cudaMalloc*. ([#37978](https://github.com/PaddlePaddle/Paddle/pull/37978)) - - - Reduce the number of free blocks in AutoGrowth Allocator to improve memory allocation performance. ([#35732](https://github.com/PaddlePaddle/Paddle/pull/35732)) - - - Remove redundant Float32 temporary tensor and cast operation for tensor with data type FP16 in`initializer.Normal` and `initializer.Constant`to save 2x memory. ([#38818](https://github.com/PaddlePaddle/Paddle/pull/38818)) - -- High-order derivative testing for models in dynamic graphs. - - - Add third-order derivative testing for network in dynamic graphs. ([#36814](https://github.com/PaddlePaddle/Paddle/pull/36814), [#37377](https://github.com/PaddlePaddle/Paddle/pull/37377)) -- Custom op: Support to custom op in ROCm(HIP) platform. ([#36771](https://github.com/PaddlePaddle/Paddle/pull/36771)) - -- Cost Model: Add basic Cost Model based on profiling infomation. ([#35774](https://github.com/PaddlePaddle/Paddle/pull/35774)) - -- Added a function to allow user to add their own layer and correspond pruning way to ASP support. ([#40253](https://github.com/PaddlePaddle/Paddle/pull/40253)) - -- Add string tensor data structure, allowing the framework to have the ability to represent and process string. ([#39830](https://github.com/PaddlePaddle/Paddle/pull/39830), [#40992](https://github.com/PaddlePaddle/Paddle/pull/40992)) - -- Add or upgrade oneDNN FP32/int8/bfloat16 Kernel, including: - - - ELU ([#37149](https://github.com/PaddlePaddle/Paddle/pull/37149)) - - - exp ([#38624](https://github.com/PaddlePaddle/Paddle/pull/38624)) - - - stack ([#37002](https://github.com/PaddlePaddle/Paddle/pull/37002)) - - - softplus ([#36382](https://github.com/PaddlePaddle/Paddle/pull/36382)) - - - round ([#39653](https://github.com/PaddlePaddle/Paddle/pull/39653)) - - - shape ([#36033](https://github.com/PaddlePaddle/Paddle/pull/36033)) - - - flatten and flatten2 ([#35892](https://github.com/PaddlePaddle/Paddle/pull/35892)) - - - slice ([#37630](https://github.com/PaddlePaddle/Paddle/pull/37630)) - - - elementwise_mul ([#40546](https://github.com/PaddlePaddle/Paddle/pull/40546)) - - - elementwise_add ([#38176](https://github.com/PaddlePaddle/Paddle/pull/38176)) - - - ementwise_div ([#36158](https://github.com/PaddlePaddle/Paddle/pull/36158)) - - - elementwise_sub ([#35662](https://github.com/PaddlePaddle/Paddle/pull/35662)) - - - roi_align ([#37848](https://github.com/PaddlePaddle/Paddle/pull/37848)) - - - nearest_interp and nearest_interp_v2 ([#37985](https://github.com/PaddlePaddle/Paddle/pull/37985)，[#38622](https://github.com/PaddlePaddle/Paddle/pull/38622)，[#39490](https://github.com/PaddlePaddle/Paddle/pull/39490)) - - - assembly optimized Adam ([#39158](https://github.com/PaddlePaddle/Paddle/pull/39158)) - - - logsoftmax ([#39793](https://github.com/PaddlePaddle/Paddle/pull/39793)) - - - activation ([#40721](https://github.com/PaddlePaddle/Paddle/pull/40721)) - - - mul ([#38552](https://github.com/PaddlePaddle/Paddle/pull/38552)) - - - mean ([#37104](https://github.com/PaddlePaddle/Paddle/pull/37104)) - - - relu ([#36265](https://github.com/PaddlePaddle/Paddle/pull/36265)) - - - pool2d ([#37081](https://github.com/PaddlePaddle/Paddle/pull/37081)) - - - concat ([#35889](https://github.com/PaddlePaddle/Paddle/pull/35889)) - - - conv2d ([#38507](https://github.com/PaddlePaddle/Paddle/pull/38507)，[#38938](https://github.com/PaddlePaddle/Paddle/pull/38938)，[#36284](https://github.com/PaddlePaddle/Paddle/pull/36284)) - - - LayerNorm ([#40418](https://github.com/PaddlePaddle/Paddle/pull/40418)) - -- Add the 3-stage storage graph retrieval engine based on SSD - host memory - GPU device memory, to support large-scale graph neural network training. ([#42472](https://github.com/PaddlePaddle/Paddle/pull/42472), [#42321](https://github.com/PaddlePaddle/Paddle/pull/42321), [#42027](https://github.com/PaddlePaddle/Paddle/pull/42027)) - -- Add heterogeneous multi-cloud training communication module switch, implement the Send/Recv interface function, and support multiple heterogeneous cloud communication. ([#40965](https://github.com/PaddlePaddle/Paddle/pull/40965) [40911](https://github.com/PaddlePaddle/Paddle/pull/40911)) - -### **(2) Function optimization** - -#### API - -- Add backward implementation of `paddle.linalg.det `. ([#36013](https://github.com/PaddlePaddle/Paddle/pull/36013)) - -- Add support for mixed precision training O2 mode for `paddle.Model`, i.e., support for Pure FP16 training mode of the original dynamic/static graphs. ([#36441](https://github.com/PaddlePaddle/Paddle/pull/40962441)) - -- Support for self chain calls for `paddle.nn.Layer`. ([#36609](https://github.com/PaddlePaddle/Paddle/pull/36609)) - -- Add settings of `is_distributed` property for the `to` method of `paddle.nn.Layer` to ensure that the distributed properties remain consistent before and after network parameter transform. ([#36221](https://github.com/PaddlePaddle/Paddle/pull/36221)) - -- Improve the parameter conversion logic of the `to` method of `paddle.nn.Layer`, to reduce the peak memory consumption of the conversion process and improve the conversion success rate. ([#36862](https://github.com/PaddlePaddle/Paddle/pull/36862)) - -- Support settings of the shape of the output Tensor for `paddle.incubate.graph_send_recv` to reduce the memory usage during the actual computation. ([#40509](https://github.com/PaddlePaddle/Paddle/pull/40509)) - -- Add the support of int32 and int64 data types for `paddle.incubate.segment_sum`, `segment_mean`, `segment_max`, and `segment_min`. ([#40577](https://github.com/PaddlePaddle/Paddle/pull/40577)) - -- Add the support of the bool type for transpose op. ([#35886](https://github.com/PaddlePaddle/Paddle/pull/35886)) - -- Switch the `paddle.mm` underlying operator from matmul to matmul_v2. ([#35770](https://github.com/PaddlePaddle/Paddle/pull/35770)) - -- Support static graph mode and support the unknown shape for `paddle.einsum`. ([#40360](https://github.com/PaddlePaddle/Paddle/pull/40360)) - -- Support data`parallelism for paddle.nn.functional.margin_cross_entropy` and `paddle.nn.functional.class_center_sample`. ([#39852](https://github.com/PaddlePaddle/Paddle/pull/39852)) - -- Support input of shape [1] for `paddle.nn.functional.grid_sample`. ([#36183](https://github.com/PaddlePaddle/Paddle/pull/36183)) - -- Support NHWC data format for `paddle.nn.PRelu`. ([#37019](https://github.com/PaddlePaddle/Paddle/pull/37019)) - -- Support the fixed random state using `paddle.seed` for `paddle.nn.functional.class_center_sample`. ([#38248](https://github.com/PaddlePaddle/Paddle/pull/38248)) - -- Add ROCM backend support for all APIs under `paddle.fft`, and optimize CUFFT backend error messages. ([#36415](https://github.com/PaddlePaddle/Paddle/pull/36415), [#36114](https://github.com/PaddlePaddle/Paddle/pull/36114/files)) - -- Support the function that the slicing dimension i 0, that is, allow slicing index results to be empty. ([#37313](https://github.com/PaddlePaddle/Paddle/pull/37313)) - -- Support int and bool type Tensor with using bool index for `Tensor.setitem`. ([#37761](https://github.com/PaddlePaddle/Paddle/pull/37761)) - -- Support nearest mode for `paddle.nn.functional.interpolate` when the input shape is 5D. ([#38868](https://github.com/PaddlePaddle/Paddle/pull/38868)) - -- Add the support of int16 for `paddle.nn.Embedding`and`paddle.gather`. ([#40964](https://github.com/PaddlePaddle/Paddle/pull/40964), [#40052](https://github.com/PaddlePaddle/Paddle/pull/40052)) - -- Support data`parallelism on single machine on``CPU platform``in paddle.distributed.spawn`. ([#35745](https://github.com/PaddlePaddle/Paddle/pull/35745), [#36758](https://github.com/PaddlePaddle/Paddle/pull/36758), [#36637](https://github.com/PaddlePaddle/Paddle/pull/36637)) - -- Add `depthwise_conv2d` MKLDNN operator. ([#38484](https://github.com/PaddlePaddle/Paddle/pull/38484)) - -- Add complex types check in the static graph model for API`paddle.abs`, `paddle.transpose`, `paddle.squeeze`, `paddle.unsqueeze`, `paddle.matmul`, and `paddle.full`. ([#40113](https://github.com/PaddlePaddle/Paddle/pull/40113)) - -- Support tuple and list type arguments for `paddle.autograd.PyLayer`. ([#38146](https://github.com/PaddlePaddle/Paddle/pull/38146)) - -- Add check whether tensor is inplace and leaf when calculate gradient. ([#37931](https://github.com/PaddlePaddle/Paddle/pull/37931)) - -- Support HIP library for `paddle.autograd.PyLayer`. ([#38184](https://github.com/PaddlePaddle/Paddle/pull/38184)) - -- Support more size inputs for `paddle.take_along_axis` and `paddle.put_along_axis`, and allow index matrix shape size to be larger than array matrix shape size. ([#39072](https://github.com/PaddlePaddle/Paddle/pull/39072)) - -- Optimize the error report message of API `paddle.nn.Pad2D` when replicate is 0. ([#36510](https://github.com/PaddlePaddle/Paddle/pull/36510/files)) - -- Support pad input in tuple format for API `paddle.nn.Pad2D`. ([#35985](https://github.com/PaddlePaddle/Paddle/pull/35985/files)) - -- Add tdm_sample API in `paddle.distributed.InMemoryDataset` to support sampling operations in TDM algorithms. ([#37044](https://github.com/PaddlePaddle/Paddle/pull/37044)) - -- Add Pre-saving Hooks mechanism for `paddle.jit.save`. ([#38186](https://github.com/PaddlePaddle/Paddle/pull/38186)) - -- Add new higher-order differentiation-related APIs. - - - `elementwise_add`: add third-order Kernel, to support computation of third-order differentiation. ([#36508](https://github.com/PaddlePaddle/Paddle/pull/36508), [#36618](https://github.com/PaddlePaddle/Paddle/pull/36618)) - - - `matmul_v2`: add third-order Kernel, to support computation of third-order differentiation. ([#36459](https://github.com/PaddlePaddle/Paddle/pull/36459)) - - - `elementwise_mul`: Add third-order Kernel, to support computation of third-order differentiation. ([#37152](https://github.com/PaddlePaddle/Paddle/pull/37547)) - -- Improve the logic of the `paddle.amp.GradScaler` to call check_finite_and_unscale op, to eliminate the cudaMemcpy introduced by the creation of the bool variable. ([#37770](https://github.com/PaddlePaddle/Paddle/pull/37770)) - -- Add check for unstack and unique op in case of input Tensor with 0 elements. ([#36021](https://github.com/PaddlePaddle/Paddle/pull/36021)) - -- Add new multi-layer, bi-directional LSTM function that supports KUNLUNXIN 2, to improve RNN forward/backward ops, and support the use of temporal model training. ([#](https://github.com/PaddlePaddle/Paddle/pull/41781)[42076](https://github.com/PaddlePaddle/Paddle/pull/42076)) - -- Add bce_loss forward/backward ops for KUNLUNXIN 2. ([#41610](https://github.com/PaddlePaddle/Paddle/pull/41610)) - -- Add backward implementation of `paddle.linalg.det `. ([#36013](https://github.com/PaddlePaddle/Paddle/pull/36013)) - -#### IR(Intermediate Representation) - -- Dynamic Graphs to Static Graphs - - - Optimize the behavior of the `ProgramCache.last` interface for dynamic graph to static graph so that it returns the most recently used Program instead of the final generated Program. ([#39541](https://github.com/PaddlePaddle/Paddle/pull/39541)) - - - Optimize the error report message for the `paddle.reshape` API for dynamic graph to static graph, and add a new recommended usage hint. ([#40599](https://github.com/PaddlePaddle/Paddle/pull/40599)) - - - Optimize the type of exception catch in the `is_api_in_module` function when transcribing dynamic code to static code. ([#40243](https://github.com/PaddlePaddle/Paddle/pull/40243)) - - - Optimize the hint of error message for dynamic graph to static graph，hide warning information by default. ([#39730](https://github.com/PaddlePaddle/Paddle/pull/https://github.com/PaddlePaddle/Paddle/pull/39730)) - - - Add the support of type hint syntax for dynamic graph to static graph to improve the accuracy of variable type analysis. ([#39572](https://github.com/PaddlePaddle/Paddle/pull/39572)) - - - Optimize the `paddle.cond` function to allow values are equal for basic types such as bool and int. ([#37888](https://github.com/PaddlePaddle/Paddle/pull/37888)) - - - Optimize the decorate function `@to_static` to allow the switch of the train/eval mode. ([#37383](https://github.com/PaddlePaddle/Paddle/pull/37383)) - - - Optimize the stack of error report for dynamic graph to static graph, to highlight user-related codes and reduce the framework redundant error stack. ([#36741](https://github.com/PaddlePaddle/Paddle/pull/36741)) - - - Remove `no_value` placeholder from the return value of `paddle.cond`. ([#36513](https://github.com/PaddlePaddle/Paddle/pull/36513)、[#36826](https://github.com/PaddlePaddle/Paddle/pull/36826)) - - - Adapt the run_program op to the new dynamic graph mode. ([#40198](https://github.com/PaddlePaddle/Paddle/pull/40198), [#40355](https://github.com/PaddlePaddle/Paddle/pull/40355)) - - - Add check for zip syntax. ([#37846](https://github.com/PaddlePaddle/Paddle/pull/https://github.com/PaddlePaddle/Paddle/pull/37846)) - - - Fix the dynamic graph to static graph failure due to the error of dimension and type judgment in the `paddle.signal.frame`, `paddle.signal.stft` and `paddle.signal.istft`. ([#40113](https://github.com/PaddlePaddle/Paddle/pull/40113)) - - - Add registration of plural type Kernel for mean, pad3d ops. ([#40113](https://github.com/PaddlePaddle/Paddle/pull/40113)) - - -#### **Mixed Precision Training** - -- Add GPU Compute Capability environment check for amp. Add the usage warning for GPU environments that the fail acceleration for training. ([#38086](https://github.com/PaddlePaddle/Paddle/pull/38086)) - -- Add check of calling order when using `paddle.amp.decorate` and `paddle.DataParallel` at the same time. ([#38785](https://github.com/PaddlePaddle/Paddle/pull/38785)) - - -#### **Distributed Training** - -- Basic functions of the distributed training - - - Optimize Fleet API and DistributedStrategy configuration to use dynamic graph parallel function conveniently. ([#40408](https://github.com/PaddlePaddle/Paddle/pull/40408)) - - - Optimize Dynamic Graph mixed parallel HybridParallelClipGrad strategy, support 4D hybrid parallel and Pure FP16 training. ([#36237](https://github.com/PaddlePaddle/Paddle/pull/36237), [#36555](https://github.com/PaddlePaddle/Paddle/pull/36555)) - - - Restructure dynamic graph data parallel strategy, to support new dynamic graph and communication. ([#40389](https://github.com/PaddlePaddle/Paddle/pull/40389), [#40593](https://github.com/PaddlePaddle/Paddle/pull/40593), [#40836](https://github.com/PaddlePaddle/Paddle/pull/40836), [#41119](https://github.com/PaddlePaddle/Paddle/pull/41119), [#41413](https://github.com/PaddlePaddle/Paddle/pull/41413), [#39987](https://github.com/PaddlePaddle/Paddle/pull/39987)) - - - Support distributed tensor model parallel for fused_attention op. ([#40101](https://github.com/PaddlePaddle/Paddle/pull/40101)) - - - Support the distributed tensor model parallel for fused_feedforward op. ([#40160](https://github.com/PaddlePaddle/Paddle/pull/40160)) - -- Graph retrieval engine - - - Optimize the data format returned by the graph sampling interface of the graph engine, with a 3x improvement of the sampling speed. ([#37315](https://github.com/PaddlePaddle/Paddle/pull/37315)) - - - Reduce the amount of graph engine threads to improve performance. ([#37098](https://github.com/PaddlePaddle/Paddle/pull/37098)) - - - Optimize graph engine data transfer to improve performance. ([#37341](https://github.com/PaddlePaddle/Paddle/pull/37341)) - - - Optimize the merge logic of embedding op to improve performance by exploiting the topological relationship of embedding op in the model. [(#35942)](https://github.com/PaddlePaddle/Paddle/pull/35942) - -- Communication library: restructure the communication library to improve the scalability and development of the communication library, and support heterogeneous communication. ([#41398](https://github.com/PaddlePaddle/Paddle/pull/41398), [#39720](https://github.com/PaddlePaddle/Paddle/pull/39720), [#40911](https://github.com/PaddlePaddle/Paddle/pull/40911), [#40579](https://github.com/PaddlePaddle/Paddle/pull/40579), [#40629](https://github.com/PaddlePaddle/Paddle/pull/40629), [#40437](https://github.com/PaddlePaddle/Paddle/pull/40437), [#40430](https://github.com/PaddlePaddle/Paddle/pull/40430), [#40228](https://github.com/PaddlePaddle/Paddle/pull/40228), [#40181](https://github.com/PaddlePaddle/Paddle/pull/40181), [#40100](https://github.com/PaddlePaddle/Paddle/pull/40100), [#40097](https://github.com/PaddlePaddle/Paddle/pull/40097), [#39892](https://github.com/PaddlePaddle/Paddle/pull/39892), [#39384](https://github.com/PaddlePaddle/Paddle/pull/39384), [#39737](https://github.com/PaddlePaddle/Paddle/pull/39737), [#40040](https://github.com/PaddlePaddle/Paddle/pull/40040)) - -- Support the publication of MoE-related interfaces in `paddle.incubate.distributed.models.moe ` (`moe.GShardGate `, `moe.BaseGate `, `moe.SwitchGate `, `moe.MoELayer `, and `moe. ClipGradForMOEByGlobalNorm `). ([#42300](https://github.com/PaddlePaddle/Paddle/pull/42300)) - -- Fix the error report in the use of recomputing in `paddle.incubate.distributed.models.moe.MoELayer `. ([#42128](https://github.com/PaddlePaddle/Paddle/pull/42128)) - -- Fix the error report in the new dynamic graph pipeline parallel caused by different data types ([#41937](https://github.com/PaddlePaddle/Paddle/pull/41937) [#42053](https://github.com/PaddlePaddle/Paddle/pull/42053)) - -- Fix the error report in the new dynamic graph tensor model parallel due to different data types([#41960](https://github.com/PaddlePaddle/Paddle/pull/41960)) - -#### **Custom operator** - -- Enhance the C++ custom operator mechanism for writing second-order gradient operators, to support adding suffixes to the gradient input variables of second-order gradient operators for use as outputs. ([#41781](https://github.com/PaddlePaddle/Paddle/pull/41781)) - -- Remove the use of the deprecated enumeration type `PlaceType` from the Tensor API member methods, make it compatible, and add a deprecation warning. ([#41882](https://github.com/PaddlePaddle/Paddle/pull/41882)) - -- Add deprecated warning for a number of deprecated interfaces of the original Tensor API, including the incomplete constructor, reshape, mutable_data, and copy_to methods. ([#41882](https://github.com/PaddlePaddle/Paddle/pull/41882)) - -#### **Other** - -- Error report and debugging optimization - - - Optimize `the error message of the label` boundary check for the cross_entropy op. ([#40001](https://github.com/PaddlePaddle/Paddle/pull/40001)) - - - Add profile record for `infer_shape` and `compute` methods of op execution of dynamic graphs, show their cost in timeline. ([#39023](https://github.com/PaddlePaddle/Paddle/pull/39023)) - - - Replace `pybind::index_error` error hint on Windows for unknown exceptions. ([#40538](https://github.com/PaddlePaddle/Paddle/pull/40538)) - - - Add the error message in the out-of-bounds checks for user scatter op. ([#37429](https://github.com/PaddlePaddle/Paddle/pull/37429)) - -- Download tool: For the problem of slow decompression of directories with multiple files in `paddle.utils.download.get_path_from_url`, replace the original way (traverse directory in loop) of decompressing files in directories one by one by calling extractall on the directory, which greatly improves the decompression speed. ([#37311](https://github.com/PaddlePaddle/Paddle/pull/37311)) - -- Speed up the quantization training for`fake_quantize_range_abs_max`、`fake_quantize_abs_max`、`fake_quantize_dequantize_abs_max`、 `fake_quantize_moving_average_abs_max`, etc. ([#40491](https://github.com/PaddlePaddle/Paddle/pull/40491)) - - -### **(3) Performance optimization** - -#### **Distributed Training** - -- Hybrid parallel optimizer `sharding_optimizer` supports `optimize_cast` optimization, which move the parameter cast during forward and backwark stage to the optimizer stage. This improves performance by 7%. ([#35878](https://github.com/PaddlePaddle/Paddle/pull/35878)) - -- GPUPS optimization: support for gradient fuse allreduce training. This improves training performance by 20%. ([#35131](https://github.com/PaddlePaddle/Paddle/pull/35131)) - -- GPUPS optimization: dump CPU optimization speed improves by 3.21x. ([#40068](https://github.com/PaddlePaddle/Paddle/pull/40068)) - -- CPU parameter server streaming training optimization: support for automatic statistics of sparse parameter statistics, incremental saving of sparse parameters, etc. The training performance improves by 20%. ([#36465](https://github.com/PaddlePaddle/Paddle/pull/36465), [#36601](https://github.com/PaddlePaddle/Paddle/pull/36601), [#36734](https://github.com/PaddlePaddle/Paddle/pull/36734), [#36909](https://github.com/PaddlePaddle/Paddle/pull/36909), [#36943](https://github.com/PaddlePaddle/Paddle/pull/36943), [#37181](https://github.com/PaddlePaddle/Paddle/pull/37181), [#37194](https://github.com/PaddlePaddle/Paddle/pull/37194), [#37515](https://github.com/PaddlePaddle/Paddle/pull/37515), [#37626](https://github.com/PaddlePaddle/Paddle/pull/37626), [#37995](https://github.com/PaddlePaddle/Paddle/pull/37995), [#38582](https://github.com/PaddlePaddle/Paddle/pull/38582), [#39250](https://github.com/PaddlePaddle/Paddle/pull/39250), [#40762](https://github.com/PaddlePaddle/Paddle/pull/40762), [#41234](https://github.com/PaddlePaddle/Paddle/pull/41234), [#41320](https://github.com/PaddlePaddle/Paddle/pull/41320), [#41400](https://github.com/PaddlePaddle/Paddle/pull/41400)) - -#### **Auto-tuning** - -Add hardware-aware automatic performance tuning for the full training process, with performance improvements of about 3% to 50% or more on image classification, segmentation, detection, and image generation tasks compared to the model's default configuration. The auto-tuning status is set via the `paddle.incubate.autotune.set_config ` API. By default, it is currently disabled. Auto-tuning has three specific levels: - -- Add the auto-tuning function to `paddle.io.DataLoader `, to select the best num_workers based on training data and device resources. ([#42004](https://github.com/PaddlePaddle/Paddle/pull/42004)) - -- Add mixed-precision training data layout auto-tuning feature, to select the best data layout based on device type and data type, and automatically convert it at runtime. ([#41964](https://github.com/PaddlePaddle/Paddle/pull/41964)) - -- Add the automatic tuning of the required workspace size threshold for Conv, which is automatically set based on the GPU's currently available requested device memory resources. Add the automatic selection of Conv cuDNN algorithms based on the generic AlgorithmCache design and Kernel timing component, which supports data variation length models. ([#41833](https://github.com/PaddlePaddle/Paddle/pull/41833)) - -#### **Operator Optimization** - -- Optimize `FasterTokenizer` performance, with a 10% performance improvement compared to pre-optimization. ([#36701](https://github.com/PaddlePaddle/Paddle/pull/36701)) - -- Optimize `index_select` inverse computation, with 3.7~25.2x performance improvement over pre-optimization. ([#37055](https://github.com/PaddlePaddle/Paddle/pull/37055)) - -- Optimize the performance of `paddle.nn.ClipByGlobalNorm`. Take 10*10 `paddle.nn.Linear` as an example. In contrast to pre-optimization, the performance improves by about 30%. ([#38209](https://github.com/PaddlePaddle/Paddle/pull/38209)) - -- Optimize the performance of `pnorm` with very large or very small `axis` dimensions, with 31-96x improvement in forward speed and 1.1-19x improvement in backward speed. ([#37685](https://github.com/PaddlePaddle/Paddle/pull/37685), [#38215](https://github.com/PaddlePaddle/Paddle/pull/38215), [#39011](https://github.com/PaddlePaddle/Paddle/pull/39011)) - -- Optimize `softmax` forward and backward performance, with a speedup ratio of about 2x for the `axis!=-1` configuration. ([#38602](https://github.com/PaddlePaddle/Paddle/pull/38602), [#38609](https://github.com/PaddlePaddle/Paddle/pull/38609), [#32387](https://github.com/PaddlePaddle/Paddle/pull/32387), [#37927](https://github.com/PaddlePaddle/Paddle/pull/37927/files)) - -- Optimize `log_softmax` forward and backward performance, with a speedup ratio of about 6x to 20x for `axis!=-1` configurations. ([#38992](https://github.com/PaddlePaddle/Paddle/pull/38992), [#40612](https://github.com/PaddlePaddle/Paddle/pull/40612)) - -- Optimize `softmax_with_cross_entropy` forward and backward performance, with a speedup ratio of about 1.3x for the `hard_label` configuration. ([#39553](https://github.com/PaddlePaddle/Paddle/pull/39553), [#40424](https://github.com/PaddlePaddle/Paddle/pull/40424), [#40643](https://github.com/PaddlePaddle/Paddle/pull/40643)) - -- Optimize `top_k` performance, with a speedup ratio of more than 22x for one-dimension and larger `k` (k=5000) configuration. ([#40941](https://github.com/PaddlePaddle/Paddle/pull/40941)) - -- Optimize `elementwise_mul` backward computation, with 1.85~12.16x performance improvement over pre-optimization. ([#37728](https://github.com/PaddlePaddle/Paddle/pull/37728)) - -- Optimize `elementwise_min` and `elementwise_max` backward computation, to equalize or improve performance by 1.05x to 18.75x over pre-optimization. ([#38236](https://github.com/PaddlePaddle/Paddle/pull/38236), [#37906](https://github.com/PaddlePaddle/Paddle/pull/37906)) - -- Optimize `nearest_interp` forward and backward computation, with forward performance improvement by 1.5x to 2.3x over pre-optimization, and backward performance improvement by 60% to 1.8x over pre-optimization. ([#38528](https://github.com/PaddlePaddle/Paddle/pull/38528), [#39067](https://github.com/PaddlePaddle/Paddle/pull/39067)) - -- Optimize `bilinear_interp` forward and backward computation, with forward performance improvement by 0.4x to 2.3x over pre-optimization, and backward performance improvement by 10%-30% over pre-optimization. ([#39243](https://github.com/PaddlePaddle/Paddle/pull/39243), [#39423](https://github.com/PaddlePaddle/Paddle/pull/39423)) - -- Optimize `dropout` forward and backward computation, with performance improvement by about 20%. ([#39795](https://github.com/PaddlePaddle/Paddle/pull/39795), [#38859](https://github.com/PaddlePaddle/Paddle/pull/38859), [#38279](https://github.com/PaddlePaddle/Paddle/pull/38279), [#40053](https://github.com/PaddlePaddle/Paddle/pull/40053)) - -- Optimize `grid_sampler` forward and backward computation, with forward performance improvement by 10% to 30% over pre-optimization, and backward performance improvement by 10% to 60% over pre-optimization. ([#39751](https://github.com/PaddlePaddle/Paddle/pull/39751)) - -- Optimize `group_norm` forward and backward computation, with the forward performance improvement by 1.04x to 2.35x, and backward performance improvement by 1.12x to 1.18x. ([#39944](https://github.com/PaddlePaddle/Paddle/pull/39944), [#40657](https://github.com/PaddlePaddle/Paddle/pull/40657), [#39596](https://github.com/PaddlePaddle/Paddle/pull/39596)) - -- Optimize `conv1d` forward and backward computation, with the forward performance improvement by 1.00x to 2.01x, and backward performance improvement by 1.01x to 474.56x. ([#38425](https://github.com/PaddlePaddle/Paddle/pull/38425)) - -- Optimize `elementwise_div` backward computation, with the backward performance improvement by 1.02x to 29.25x. ([#38044](https://github.com/PaddlePaddle/Paddle/pull/38044)) - -- Optimize `gelu` forward and backward computation, with the backward performance improvement by 1.13x to 1.43x, and reverse performance improvement by 1.10x to 1.55x. ([#38188](https://github.com/PaddlePaddle/Paddle/pull/38188), [#38263](https://github.com/PaddlePaddle/Paddle/pull/38263)) - -- Optimize `elementwise_sub` backward computation, with the backward performance improvement by 1.04x to 15.64x. ([#37754](https://github.com/PaddlePaddle/Paddle/pull/37754)) - -- Optimize `flip's` forward performance on one-dimensional data input, with the performance improvement by 100%. ([#37825](https://github.com/PaddlePaddle/Paddle/pull/37825)) - -- Optimize `layer_norm` forward and backward computation, with the forward performance improvement by 2x to 5x over pre-optimization, and backward performance improvement by 20% to 50% over pre-optimization. ([#39167](https://github.com/PaddlePaddle/Paddle/pull/39167), [#39247](https://github.com/PaddlePaddle/Paddle/pull/39247)) - -- Optimize `embedding` forward and backward computation, with a maximum improvement of 1.51x in forward performance and 1.03x to 7.79x in backward performance. ([#39856](https://github.com/PaddlePaddle/Paddle/pull/39856), [#39886](https://github.com/PaddlePaddle/Paddle/pull/398866)) - -- Optimize `gelu` FP16 forward and backward calculations, with forward performance improvement by 9% to 12% over pre-optimization, and backward performance improvement by 2% to 9% over pre-optimization. ([#38980](https://github.com/PaddlePaddle/Paddle/pull/38980)) - -- Remove CPU -> GPU explicit data transfer operation in `gather_nd` forward and backward operators, and remove the explicit synchronous operation in `index_select` forward and backward operators. Change GPU -> GPU data transfer in `scatter_nd` from synchronous operation to asynchronous operation. ([#40933](https://github.com/PaddlePaddle/Paddle/pull/40933)) - -- Optimize `Lars optimzier` computation, with the training performance improvement of Resnet50 PF16 model by 5.1% over pre-optimization. ([#35652](https://github.com/PaddlePaddle/Paddle/pull/35652), [#35476](https://github.com/PaddlePaddle/Paddle/pull/35476)) - -- Optimize `AvgPool2dGrad` computation, with the performance improvement by 2.6x over pre-optimization. ([#35389](https://github.com/PaddlePaddle/Paddle/pull/35389)) - -- Optimize `Elementwise` computation for multivariate output, improving performance by up to 15% over pre-optimization. ([#38329](https://github.com/PaddlePaddle/Paddle/pull/38329), [#38410](https://github.com/PaddlePaddle/Paddle/pull/38410)) - -- Optimize `Categorical`the probs computation, simplify the computation logic, and improve the performance by 4x to 5x. ([#42178](https://github.com/PaddlePaddle/Paddle/pull/42178)) - -- Optimize the `paddle.sum ` performance, with performance improvement by about 20%. ([#42309](https://github.com/PaddlePaddle/Paddle/pull/42309)) - -- Remove CudaStreamSync operation from `paddle.nn.ClipGradByGlobalNorm ` to reduce scheduling overhead during execution, with 5% performance improvement on ptb models. ([#42170](https://github.com/PaddlePaddle/Paddle/pull/42170)) - -- Optimize a series of underlying data structures and detailed implementations in the original dynamic graph execution system to improve the scheduling performance of the original dynamic graph. ([#42010](https://github.com/PaddlePaddle/Paddle/pull/42010), [#42171](https://github.com/PaddlePaddle/Paddle/pull/42171), [#42224](https://github.com/PaddlePaddle/Paddle/pull/42224), [#42256](https://github.com/PaddlePaddle/Paddle/pull/42256), [#42306](https://github.com/PaddlePaddle/Paddle/pull/42306), [#42329](https://github.com/PaddlePaddle/Paddle/pull/42329)[, #42340](https://github.com/PaddlePaddle/Paddle/pull/42340), [#42368](https://github.com/PaddlePaddle/Paddle/pull/42368), [#42425](https://github.com/PaddlePaddle/Paddle/pull/42425)) - -- Simplify the probs calculation logics of `paddle.distribution.Categorical `, to improve performance by 4x to 5x. ([#42178](https://github.com/PaddlePaddle/Paddle/pull/42178)) - -### **(4) Bug fixing** - -#### API - -- Fix the output type error with `paddle.sum` when the input parameter type and output parameter type do not match and the number of reduce elements on the `axis` is 1. ([#36123](https://github.com/PaddlePaddle/Paddle/pull/36123)) - -- Fix an `AttributeError` in `paddle.flops` when the layer output type is tuple. ([#38850](https://github.com/PaddlePaddle/Paddle/pull/38850)) - -- Fix the `paddle.diag` failing to propagate gradients because there is no backward kernel. ([#40447](https://github.com/PaddlePaddle/Paddle/pull/40447)) - -- Fix an error in sorting `paddle.sort` input with NaN values. ([#41070](https://github.com/PaddlePaddle/Paddle/pull/41070)) - -- Fix the error when`paddle.full_like`'s input contains INF value. ([#40232](https://github.com/PaddlePaddle/Paddle/pull/40232)) - -- Fix the bug in `paddle.strided_slice`: strided_slice result does not consistent with slice when the data in the input of starts is less than -rank. ([#39066](https://github.com/PaddlePaddle/Paddle/pull/39066)) - -- Fix the bug in the `max_pool` family of operators where infer_shape is calculated incorrectly when index is returned. This affects the APIs: `paddle.nn.functional.max_pool1d/2d/3d`, `paddle.nn.functional.adaptive_max_pool1d/2d/3d`, `paddle.nn.MaxPool1D/2D/3D`, `paddle.nn.AdaptiveMaxPool1D/2D/3D`. ([#40139](https://github.com/PaddlePaddle/Paddle/pull/40139)) - -- Fix an issue where the dtype of pooling_mask returned by the `max_pool` family of operators is incorrect. Now the dtype of pooling_mask is int32. The affected APIs are `paddle.nn.functional.max_pool1d/2d/3d`, `paddle.nn.functional.adaptive_max_pool1d/2d/3d`, `paddle.nn.MaxPool1D/2D/3D`, `paddle.nn.AdaptiveMaxPool1D/2D/3D`. ([#39314](https://github.com/PaddlePaddle/Paddle/pull/39314) ) - -- Fix the bug with `paddle.shape` where the backward gradient by default causes a computation error. ([#37340](https://github.com/PaddlePaddle/Paddle/pull/37340)) - -- Fix the bug in `paddle.nn.Layer's` `to` method when converting both dtype and place at the same time. ([#37007](https://github.com/PaddlePaddle/Paddle/pull/38007)) - -- Fix the bug that `paddle.amp.decorate` fails to rewrite the parameters of non-leaf network layers to FP16. ([#38402](https://github.com/PaddlePaddle/Paddle/pull/38402)) - -- Fix the bug that the `paddle.amp.decorate` rewrites the non-input parameter in `paddle.nn.BatchNorm1D`, `paddle.nn.BatchNorm2D`, and `paddle.nn.BatchNorm3D` to FP16. ([#38541](https://github.com/PaddlePaddle/Paddle/pull/38541)) - -- Fix the bug that the `paddle.amp.decorate` rewrites the non-input parameter in `paddle.nn.SyncBatchNorm` to FP16. ([#40943](https://github.com/PaddlePaddle/Paddle/pull/40943)) - -- Fix redundant warnings in `paddle.nn.Layer.to`. ([#36700](https://github.com/PaddlePaddle/Paddle/pull/36700)) - -- Fix the bug in `paddle.nn.RNN` when being used inside control flow. ([#41162](https://github.com/PaddlePaddle/Paddle/pull/41162)) - -- Fix the bug that the `paddle.to_tensor` fails to specify the CUDAPlace of the Tensor. ([#39662](https://github.com/PaddlePaddle/Paddle/pull/39662)) - -- Fix the issue that`paddle.nn.Identity` is not exposed. ([#39615](https://github.com/PaddlePaddle/Paddle/pull/39615)) - -- Fix the bug where the output values of the `fill_` and `zero_` inplace APIs are incorrect when the input is on a CUDAPinned Place after dynamic graph reconstruction. ([#41229](https://github.com/PaddlePaddle/Paddle/pull/41229)) - -- After refactoring the dynamic graph, fix the bug of incorrect inplace version value of the output Tensor when calling assign op using the append op. Change it to call assign op using the `_C_ops`. ([#41118](https://github.com/PaddlePaddle/Paddle/pull/41118)) - -- Remove unreasonable codes in the `elementwise_add` 's third-order kernel, and fix an uninitialized issue in the network creation process. ([#36618](https://github.com/PaddlePaddle/Paddle/pull/36618)) - -- Fix the missing attribute bug in `conv2d` execution of cuDNN Kernel. ([#38827](https://github.com/PaddlePaddle/Paddle/pull/38827)) - -- Fix an issue where `multiclass_nms3` output shape is incorrect. ([#40059](https://github.com/PaddlePaddle/Paddle/pull/40059)) - -- Fix an issue with `yolo_box` outputting incorrect shape. ([#40056](https://github.com/PaddlePaddle/Paddle/pull/40056)) - -- Fix an issue where the higher-order differentiation `gradients` interface does not take effect as expected when target_grad is specified. ([#40940](https://github.com/PaddlePaddle/Paddle/pull/40940/)) - -- Fix an issue that the network parameter type is incorrect when the default_dtype is modified in the op`_BatchNormBase` base class in the dynamic graph mode. The affected APIs are `paddle.nn.BatchNorm1D`，`paddle.nn.BatchNorm2D`，`paddle.nn.BatchNorm3D`， and `paddle.nn.SyncBatchNorm`. Specific reason: when `get_default_dtype() == 'float16'`, the default parameter data type is modified by `set_default_dtype('float32')`. The parameter type in dynamic graph mode is created by default_dtype; therefore, the change of the default parameter type causes the subsequent networking Parameter type error. ([#36376](https://github.com/PaddlePaddle/Paddle/pull/36376)) - -- Fix the bug of the undefined intermediate variable in the backward op in batchnorm op in case that the data type is FP32 and the data dimension is `dims = 2 and data_layout = NHWC`. ([#37020](https://github.com/PaddlePaddle/Paddle/pull/37020)) - -- Fix the bug that shape of weights is incorrect, when using`paddle.static.nn.prelu` in static graph mode, and input format is`NHWC`, `mode==channel`. ([#38310](https://github.com/PaddlePaddle/Paddle/pull/38310)) - -- Fix the bug of `paddle.nn.functional.class_center_sample`: CUDA seed setting issue in multi-machine case. ([#38815](https://github.com/PaddlePaddle/Paddle/pull/38815)) - -- Fix the bug of failing to report error when the input of`paddle.nn.functional.one_hot`is incorrect. ([#41335](https://github.com/PaddlePaddle/Paddle/pull/41335)) - -- Fix an issue where a callback to reclaim device memory on a DCU device is not triggered in time, resulting in an OOM of the device memory. ([#40445](https://github.com/PaddlePaddle/Paddle/pull/40445)) - -- Fix the bugs of `setitem` backward gradient abnormal and inplace logic handling abnormal in some dynamic graph scenarios. ([#37023](https://github.com/PaddlePaddle/Paddle/pull/37023), [#38298](https://github.com/PaddlePaddle/Paddle/pull/38298)) - -- Fix the bug of index abnormal when Tensor array uses the Slice to index in the dynamic to static scenarios. ([#39251](https://github.com/PaddlePaddle/Paddle/pull/39251)) - -- Fix the bug of memory or device memory leaks caused by some temporary variables not being correctly destructed when `paddle.Tensor.register_hook` interface is used. ([#40716](https://github.com/PaddlePaddle/Paddle/pull/40716)) - -- Fix the bug that `Tensor.getitem` cannot get the value when the index is a bool Tensor with all False. ([#41297](https://github.com/PaddlePaddle/Paddle/pull/41297)) - -- Fix the bug that `Tensor.getitem` cannot get the value when the index is a bool scalar Tensor. ([#40829](https://github.com/PaddlePaddle/Paddle/pull/40829)) - -- Fix the bug in `paddle.index_select` when index is a 0-shape Tensor. ([#41383](https://github.com/PaddlePaddle/Paddle/pull/41383)) - -- Fix the bug when the number of GPU threads requested by `paddle.index_select` and `paddle.index_sample` exceeds the limited machine resources. ([#41127](https://github.com/PaddlePaddle/Paddle/pull/41127), [#37816](https://github.com/PaddlePaddle/Paddle/pull/37816), [#39736](https://github.com/PaddlePaddle/Paddle/pull/39736), [#41563](https://github.com/PaddlePaddle/Paddle/pull/41563)) - -- Fix the bug when ReduceConfig, elemwise_grad, gather, gather_nd, and scatter ops request more GPU threads than the limited machine resources. ([#40813](https://github.com/PaddlePaddle/Paddle/pull/40813), [#41127](https://github.com/PaddlePaddle/Paddle/pull/41127)) - -- Fix the bug that the memory access is out of boundary when NX ! = 1 in ReadData, ReadDataBc, and ReadDataReduce in Kernel Primitive API. ([#36373](https://github.com/PaddlePaddle/Paddle/pull/36373)) - -- Fix the bug of the computation result abnormal due to data overflow caused by the IndexRandom data type error. ([#39867](https://github.com/PaddlePaddle/Paddle/pull/39867), [#39891](https://github.com/PaddlePaddle/Paddle/pull/39891)) - -- Fix the bug of the returned computing result error of reduce op when reduce_num = 1. ([#38771](https://github.com/PaddlePaddle/Paddle/pull/38771)) - -- Fix the bug of the memory access out-of-bound of reduce op in the middle dimension of reduce in HIP environments. ([#41273](https://github.com/PaddlePaddle/Paddle/pull/41273)) - -- Fix the bug of Kernel failed to properly release in the computation of two FP16 one-dimensional vectors of matmul op. - -- Fix the bug caused by CUDA integer computation overflow for some operators, including: bernoulli, gaussian_random, gumbel_softmax, multinomial, truncated_gaussian_random, uniform_ random_inplace, and uniform_random ops. ([#37670](https://github.com/PaddlePaddle/Paddle/pull/37670)) - -- Fix the bug where `paddle.nn.Sequential` reports a KeyError error when traversing sublayers in a for loop. ([#39372](https://github.com/PaddlePaddle/Paddle/pull/39372)) - -- Fix the bug of the check shape error in `paddle.nn.functional.unfold` when compiling in static graphs. ([#38907](https://github.com/PaddlePaddle/Paddle/pull/38907), [#38819](https://github.com/PaddlePaddle/Paddle/pull/38819)) - -- Fix the bug of reporting an error if `axis` is specified when using dropout for static graphs. ([#37223](https://github.com/PaddlePaddle/Paddle/pull/37223)) - -- Migrate the matmul operator in the `paddle.nn.MultiHeadAttention` to the matmul_v2 operator. ([#36222](https://github.com/PaddlePaddle/Paddle/pull/36222)) - -- Fix the bug occurred in throwing FPE when the empty Tensor is used in `paddle.nn.functional.label_smooth`. ([#35861](https://github.com/PaddlePaddle/Paddle/pull/35861)) - -- Fix the deformation bug of reshape op when input is an empty Tensor. Support the empty Tensor rehape to [-1]. ([#36087](https://github.com/PaddlePaddle/Paddle/pull/36087)) - -- Fix the bug of the modified values will incorrectly override other rows when the `fill_diagonal` 's input parameter offset is non-zero. ([#36212](https://github.com/PaddlePaddle/Paddle/pull/36212)) - -- Modify stop_gradient returned by the range op bing set to True in dynamic graph mode. ([#37486](https://github.com/PaddlePaddle/Paddle/pull/37486)) - -- Fix the bug where Lamb optimizer is updated incorrectly when Beta1Pow and Beta2Pow are on the GPU. ([#38518](https://github.com/PaddlePaddle/Paddle/pull/38518)) - -- Fix the bug where the conv2d operator doesn't respect to FLAGS_cudnn_deterministic. ([#37173](https://github.com/PaddlePaddle/Paddle/pull/37173)) - -- Fix the bug caused by an earlier version of cufft that does not define CUFFT_VERSION. ([#37312](https://github.com/PaddlePaddle/Paddle/pull/37312)) - -- Fix the computing error of `paddle.ifftshit` and `paddle.fftshift`. ([#36834](https://github.com/PaddlePaddle/Paddle/pull/36834), [#36748](https://github.com/PaddlePaddle/Paddle/pull/36748)) - -- Fix the `axis` computation error in `paddle.fft` series of APIs. ([#36321](https://github.com/PaddlePaddle/Paddle/pull/36321)) - -- Fix an output data type registration bug of batch_norm_grad op in case of FP16 data type. This bug causes the compilation failure in some scenarios. There is also the impact on FP16 computational precision. ([#42461](https://github.com/PaddlePaddle/Paddle/pull/42461)) - -- Fix the incorrect Infershape information bug in the `paddle.nn.functional.pad ` API when the padding is Tensor in dynamic to static conversion. ([#42414](https://github.com/PaddlePaddle/Paddle/pull/42414)) - -- Fix an exception in `paddle.distribution.StickBreakingTransform ` when the input dimension exceeds 2. ([#41762](https://github.com/PaddlePaddle/Paddle/pull/41672)) - -- Fix a nan/inf bug calculated with QK^T in fused_attention op. ([#42032](https://github.com/PaddlePaddle/Paddle/pull/42032)) - -- Fix a nan/inf bug calculated in fused_attention op with FusedResidualDropoutBias on V100. ([#42398](https://github.com/PaddlePaddle/Paddle/pull/42398)) - -- Fix a redundant data transform bug introduced by the full_like op during execution. ([#41973](https://github.com/PaddlePaddle/Paddle/pull/41973)) - -- Fix a problem with p_norm op calculating nan on GPU environments. ([#41804](https://github.com/PaddlePaddle/Paddle/pull/41804)) - -- Fix a section error of split op when the sections parameter has a size of 0. ([#41755](https://github.com/PaddlePaddle/Paddle/pull/41755)) - -- Fix the bug of reporting not supporting Place (gpu:0) in multi-card training when broadcast is required in 6 elementwise ops (pow, complex, divide_double, multiply_double, fmax, and fmin). ([#42332](https://github.com/PaddlePaddle/Paddle/pull/42332)) - -- Fix the bug that the deprecated interface reports a warning in case of `import paddle` due to a PIL version update. ([#42307](https://github.com/PaddlePaddle/Paddle/pull/42307)) - -- Fix the bug that `paddle.linalg.matrix_rank ` does not support tol as FP64 Tensor under static graph. ([#42085](https://github.com/PaddlePaddle/Paddle/pull/42085)) - -#### IR(Intermediate Representation) - -- Dynamic to static graphs - - - Fix a type derivation error in reverse gradient accumulation when the `tensor_array` is used with the control flow. ([#39585](https://github.com/PaddlePaddle/Paddle/pull/39585), [#39689](https://github.com/PaddlePaddle/Paddle/pull/39689)) - - - Fix an issue where the parameter gradient type is not set correctly during dynamic to static AMP training. ([#40938](https://github.com/PaddlePaddle/Paddle/pull/40938)) - - - Fix an issue of reporting an error in the dynamic to static transcription when there are misplaced annotations in the codes. ([#39035](https://github.com/PaddlePaddle/Paddle/pull/39035), [#38003](https://github.com/PaddlePaddle/Paddle/pull/38003)) - - - Fix an issue where Tensor is not properly converted to Variable when calling a non-forward function in dynamic to static codes. ([#37296](https://github.com/PaddlePaddle/Paddle/pull/37296), [#38540](https://github.com/PaddlePaddle/Paddle/pull/38540)) - - - Fix an issue where `paddle` is incorrectly passed as a variable when dynamic to static transcription. ([#37999](https://github.com/PaddlePaddle/Paddle/pull/37999)) - - - Fix an issue where model parameters are incorrectly counted when calling `paddle.flops` after model dynamic to static conversion. ([#36852](https://github.com/PaddlePaddle/Paddle/pull/36852)) - - - Fix an issue where GPU memory will keep growing in train mode and no_grad contexts after loading models using the `paddle.jit.save/load` interface. ([#36434](https://github.com/PaddlePaddle/Paddle/pull/36434)) - - - Add warning in function of convert_call when converting the generator function. ([#35369](https://github.com/PaddlePaddle/Paddle/pull/35369)) - - - Fix the run_program op dependency analysis bug. ([#38470](https://github.com/PaddlePaddle/Paddle/pull/38470)) - - - Fix the code conversion bug when returning a single value in control flow For. ([#40683](https://github.com/PaddlePaddle/Paddle/pull/40683)) - - - Fix the bug when generating a reverse op when the input to conditional_block op contains LoDTensorArray. ([#39585](https://github.com/PaddlePaddle/Paddle/pull/39585)) - - - Fix the bug that `padddle.jit.save ` loses the forward_pre_hook and forward_post_hook of the top Layer in case of the export of a dynamic-to-static graph mode. ([#42273](https://github.com/PaddlePaddle/Paddle/pull/42273)) - - - Fix the dynamic to static conversion error report where the shape parameter in `paddle.expand ` contains a Tensor. ([#41973](https://github.com/PaddlePaddle/Paddle/pull/41973)) - - -#### **Distributed Training** - -- Distributed training basic functions - - - Fix the bug of a port reporting error in the distributed multi-machine training. ([#37274](https://github.com/PaddlePaddle/Paddle/pull/37274)) - - - Fix the brpc compilation dependency bug. ([#37064](https://github.com/PaddlePaddle/Paddle/pull/37064)) - - - Fix an occupied port issue due to tcp self-connections when Fleet starts. ([#38174](https://github.com/PaddlePaddle/Paddle/pull/38174)) - - - Fix the precision degradation bug under data parallel due to inconsistent initialization of FP16 parameters under multiple cards. ([#38838](https://github.com/PaddlePaddle/Paddle/pull/38838), [#38563](https://github.com/PaddlePaddle/Paddle/pull/38563), [#38405](https://github.com/PaddlePaddle/Paddle/pull/38405)) - - - Fix the precision degradation under data parallel due to FP16 gradient synchronization without dividing by the number of cards. ([#38378](https://github.com/PaddlePaddle/Paddle/pull/38378)) - -- Dynamic graph mixing parallel - - - Fix the bug where parameters are not updated in FP16 mode under mixed parallel by using the new update interface. ([#36017](https://github.com/PaddlePaddle/Paddle/pull/36017)) -- Static graph mixing parallel - - - Fix an issue where grad merge is not compatible with ClipGradientByGlobalNorm in distributed dp mode. ([#36334](https://github.com/PaddlePaddle/Paddle/pull/36334)) - - - Fix an issue under hybrid parallelism where the non-distributed parameters of tensor model parallelism are not broadcast during the initialization phase, resulting in inconsistent non-distributed parameters across cards. ([#36186](https://github.com/PaddlePaddle/Paddle/pull/36186)) - - - Fix the issue that sharding's save_persistables interface does not save FP16 parameters and offload persistent variables when sharding is enabled with offload. ([#40477](https://github.com/PaddlePaddle/Paddle/pull/40477)) - - - Fix the bug where ema parameters are not saved on non-0 cards when sharding is enabled for training. ([#39860](https://github.com/PaddlePaddle/Paddle/pull/39860)) - - - Fix an issue where FC incorrectly calculates gradients according to column cuts. ([#38724](https://github.com/PaddlePaddle/Paddle/pull/38724)) - - - Fix the bug reported when DistributedStrategy is set to without_graph_optimizer when used with rnn. ([#36176](https://github.com/PaddlePaddle/Paddle/pull/36176)) - -- GPUPS Parameter Server Training - - - Fix the CPU branch compilation bug triggered by the GPUPS macro definition. ([#37248](https://github.com/PaddlePaddle/Paddle/pull/37248)) - - - Fix an occasional error raised when saving delta and pullsparse concurrency during GPUPS streamline training. ([#37233](https://github.com/PaddlePaddle/Paddle/pull/37233)) - - - Fix a download error issue caused by HDFSClient querying a directory without returning the full path. ([#36590](https://github.com/PaddlePaddle/Paddle/pull/36590)) - - - Fix the bug with pulling old parameters in GPUPS streamline training. ([#36512](https://github.com/PaddlePaddle/Paddle/pull/36512)) - - - Fix a GPUPS multi-stream allocation issue. ([#37476](https://github.com/PaddlePaddle/Paddle/pull/37476)) - - - Fix the bug of the GPUPS pybind out of core. ([#37287](https://github.com/PaddlePaddle/Paddle/pull/37287)) - - -#### **Other** - -- Fix the clip_extra issue when saving models for dynamic graph quantization training. ([#38323](https://github.com/PaddlePaddle/Paddle/pull/38323)) - -- Fix an issue with abs_max scale initialization for dynamic graph quantization training. ([#39307](https://github.com/PaddlePaddle/Paddle/pull/39307)) - -- Fix an issue of exceptions in saving model in dynamic graph quantization training. ([#38102](https://github.com/PaddlePaddle/Paddle/pull/38102), [#38012](https://github.com/PaddlePaddle/Paddle/pull/38012)) - -- Fix the offline quantization flatten op output error. ([#37722](https://github.com/PaddlePaddle/Paddle/pull/37722)) - -- Fix the non-matching dimension bug in case of inverse quantization matmul op. ([#36982](https://github.com/PaddlePaddle/Paddle/pull/36982)) - -- Fix the bug of adding quantization op when quantizing matmul_v2 without weights. ([#36593](https://github.com/PaddlePaddle/Paddle/pull/36593)) - -- Fix the error of saving the quant_axis attribute in the conv op channel-wise quantization when saving the models. ([#39054](https://github.com/PaddlePaddle/Paddle/pull/39054)) - -- Fix the slow training of channel-wise quantization. ([#40772](https://github.com/PaddlePaddle/Paddle/pull/40772)) - -- Fix the bug of quantization training when dividing by tensor(initialized as 0) leads to nan. ([#36762](https://github.com/PaddlePaddle/Paddle/pull/36762)) - -- Fix incorrect settings of amp_level for mixed precision in multi-threaded scenarios. ([#39198](https://github.com/PaddlePaddle/Paddle/pull/39198)) - -- Fix an issue where PyLayer and Recompute is not set mixed precision correctly when mixed precision training is used with PyLayer and Recompute. ([#39950](https://github.com/PaddlePaddle/Paddle/pull/39950), [#40042](https://github.com/PaddlePaddle/Paddle/pull/40042)) - -- Fix an issue where `D_GLIBCXX_USE_CXX11_ABI` does not take effect when compiling custom operators under Mac. ([#37878](https://github.com/PaddlePaddle/Paddle/pull/37878)) - -- Fix the bug of inconsistent dynamic and static behaviors in case of block=None the initializer-related API. ([#37827](https://github.com/PaddlePaddle/Paddle/pull/37827)) - -- Fix the bug in python 3.6 where there is no fluid module. ([#35862](https://github.com/PaddlePaddle/Paddle/pull/35862)) - -- Fix the bug where optimizer `paddle.optimizer.Adamw` incorrectly calls adam op. ([#36028](https://github.com/PaddlePaddle/Paddle/pull/36028)) - -- Fix a logic error when the `paddle.optimizer.Momentum` optimizer parameter `regularizer` property is None under the multi tensor policy. ([#38344](https://github.com/PaddlePaddle/Paddle/pull/38344)) - -- Fix the bug that the `paddle.optimizer.Momentum` and `paddle.optimizer.Adam` optimizers modify the `multi_precision` property under the multi tensor policy. ([#38991](https://github.com/PaddlePaddle/Paddle/pull/38991)) - -- Fix the code compilation error when using final-state API amp in combination with optional Tensor. ([#40980](https://github.com/PaddlePaddle/Paddle/pull/40980)) - -- Fix the bug where paddle+lite+xpu prediction library would report an error when calling lite CPU prediction, and fix the bug where paddle+lite(without NNAdapter) would report an error when compiling. ([#37449](https://github.com/PaddlePaddle/Paddle/pull/37449)) - -- Fix the bug in Debug compile mode where LoDTensorArray crashes due to inconsistent Pybind11 bindings. ([#37954](https://github.com/PaddlePaddle/Paddle/pull/37954)) - -- Fix the bug that prevents correct construction of Tensor in the extreme case where the shape parameter is a list of Tensor mix with int. ([#38284](https://github.com/PaddlePaddle/Paddle/pull/38284)) - -- Fix a compatibility issue with the `paddle.optimizer.AdamW` API. ([#37905](https://github.com/PaddlePaddle/Paddle/pull/37905)) - -- Fix the bug in _InstanceNormBase where the returne value of extra_repr is incorrect. ([#38537](https://github.com/PaddlePaddle/Paddle/pull/38537)) - -- Fix the bug that the Paddle Inference lacks of the symbol `paddle::distributed::TensorTable` when the -DWITH_DISTRIBUTED is uesd. ([#41128](https://github.com/PaddlePaddle/Paddle/pull/41128)) - -- matmul_v2 op reports error when there is a 0 value in the shape. ([#35791](https://github.com/PaddlePaddle/Paddle/pull/35791)) - -- Fix the problem of the repeated printing for no gradient input hint message of the recomputed in dynamic graphs. Change it to the printing only once with using warning. ([#38293](https://github.com/PaddlePaddle/Paddle/pull/38293)) - -- Fix the low accuracy bug on the validation set in later epoch training in visual models in the gelu op. ([#38450](https://github.com/PaddlePaddle/Paddle/pull/38450)) - -- Fix adamw op error in numerical computation. ([#37746](https://github.com/PaddlePaddle/Paddle/pull/37746)) - -- Add the parameters in the sparse_momentum `_C_ops` interface. ([#39969](https://github.com/PaddlePaddle/Paddle/pull/39969)) - -- Fix the bug where there is no `distributed` module in python 3.6. ([#35848](https://github.com/PaddlePaddle/Paddle/pull/35848)) - -- Fix the eigh unit test data initialization problem. ([#39568](https://github.com/PaddlePaddle/Paddle/pull/39568)) - -- Fix the eigvalsh unit test data initialization problem. ([#39841](https://github.com/PaddlePaddle/Paddle/pull/39841)) - -- Fix the bug of not working properly due to excessive register usage on V100 by segment op. ([#38113](https://github.com/PaddlePaddle/Paddle/pull/38113)) - -- Fix the bug with conv-related op sparsification incorrectly set dimension. ([#36054](https://github.com/PaddlePaddle/Paddle/pull/36054)) - -- Provide Automatic SParsity training for static graph-related function Alias to `Paddle.static.sparsity`. ([#36525](https://github.com/PaddlePaddle/Paddle/pull/36525)) - -- Fix the bug where divide op’s integer division is still an integer. ([#40890](https://github.com/PaddlePaddle/Paddle/pull/40890)) - -- Fix the crash bug of`paddle.multiplex` when input Tensor value is 0. ([#34972](https://github.com/PaddlePaddle/Paddle/pull/34972)) - -- Fix a speed exception for set `reduction` parameter in `paddlpaddle.nn.functional.kl_div`. ([#37283](https://github.com/PaddlePaddle/Paddle/pull/37283)) - -- Fix the data source unsorted bug in loading the Cifar dataset. ([#37272](https://github.com/PaddlePaddle/Paddle/pull/37272)) - -- Fix the conversion of loss from uint16 to float in the ProgressBar class. ([#39231](https://github.com/PaddlePaddle/Paddle/pull/39231)) - -- Fix the ShareBufferWith shared data type problem. ([#37464](https://github.com/PaddlePaddle/Paddle/pull/37464), [#37247](https://github.com/PaddlePaddle/Paddle/pull/37247)) - -- Fix the performance issue when `paddle.io.DataLoader` uses IterableDataset and num_workers>0. ([#40541](https://github.com/PaddlePaddle/Paddle/pull/40541)) - -- Fix the bug with `paddle.vision.ops.yolo_loss` returns incomplete values in dynamic graph. ([#40185](https://github.com/PaddlePaddle/Paddle/pull/40185)) - -- Remove the restriction that the input parameter dataset of `paddle.io.BatchSampler` needs to be the `paddle.io.Dataset` type, to expand the support for user-defined datasets. ([#40184](https://github.com/PaddlePaddle/Paddle/pull/40184)) - -- Fix the bug of `paddle.summary` reporting that op_flops does not exist. ([#36489](https://github.com/PaddlePaddle/Paddle/pull/36489)) - -- Fix the formula error of lars_momentum op when lars_weight_decay=0. ([#40892](https://github.com/PaddlePaddle/Paddle/pull/40892)) - -- Fix the bug that the optimize-offload cannot save presistable var. ([#36433](https://github.com/PaddlePaddle/Paddle/pull/36433)) - -- Fix an issue where optimizer-offload does not support adamw op type. ([#36432](https://github.com/PaddlePaddle/Paddle/pull/36432)) - -- Fix an issue where enable_program_desc_tracing_data in Tracer is not safe in multi-threaded scenarios. ([#39776](https://github.com/PaddlePaddle/Paddle/pull/39776)) - -- Fix an issue where the model file size is not initialized when the model is read. ([#40518](https://github.com/PaddlePaddle/Paddle/pull/40518)) - -- Fix the logic bug of the Expand op. When the dimension of the input Tensor X is smaller than the shape to be expanded, it may result in the incorrect Out.Shape. ([#38677](https://github.com/PaddlePaddle/Paddle/pull/38677)) - -- Fix the dynamic to static transcription error when the Expand_As op takes only y.shape without Y variable entered. ([#38677](https://github.com/PaddlePaddle/Paddle/pull/38677)) - -- Fix the logic error when Expand_As op computes the output shape. ([#38677](https://github.com/PaddlePaddle/Paddle/pull/38677)) - -- Fix the bug that the variables of the `core.VarDesc.VarType.STRINGS` type report error when getting the `lod_level` property and setting its `lod_level` to None. ([#39077](https://github.com/PaddlePaddle/Paddle/pull/39077)) - -- Fix an issue where the framework function `Pylayer` does not support different dtypes. ([#37974](https://github.com/PaddlePaddle/Paddle/pull/37974)) - -- Fix the bug of division by zero of the learning rate decay API `paddle.optimizer.lr.PolynomialDecay`. ([#38782](https://github.com/PaddlePaddle/Paddle/pull/38782)) - -- Fix the issue where some logs remained after calling the DisableGlogInfo() interface. ([#36356](https://github.com/PaddlePaddle/Paddle/pull/36356)) - -- Fix an error in backward of multi-layer RNN (when dropout is set to 0) in the training of SimpleRNN, GRU and LSTM API CPU. ([#37080](https://github.com/PaddlePaddle/Paddle/pull/37080)) - -- Add cache for fft on the backend of cufft and hipfft. ([#36646](https://github.com/PaddlePaddle/Paddle/pull/36646)) - -- Enable the shifts parameter of `paddle.roll` to support transfer in Tensor. ([#36727](https://github.com/PaddlePaddle/Paddle/pull/36727)) - -- Add onemkl to fft as an optional computation backend. ([#36414](https://github.com/PaddlePaddle/Paddle/pull/36414)) - -- Fix the precision bug in the bfloat16 type under two mamtul_v2 and elementwise_div ops. ([#42479](https://github.com/PaddlePaddle/Paddle/pull/42479)) - -- Fix a possible error in the next step caused by LoDTensorArray clearing only the internal Tensor and not clearing the Array during device memory recycling. ([#42398](https://github.com/PaddlePaddle/Paddle/pull/42398)) - - -## **4. Deployment Direction (Paddle Inference)** - -### **(1) New features** - -#### **New APIs** - -- Add the Java API so that Java developers can implement high performance inference on the server and in the cloud through a simple and flexible interface. ([#37162](https://github.com/PaddlePaddle/Paddle/pull/37162)) - -- Add `GetTrtCompileVersion` and `GetTrtRuntimeVersion` interfaces for getting TensorRT version information. ([#36429](https://github.com/PaddlePaddle/Paddle/pull/36429)) - -- Add the `ShareExternalData` interface to avoid memory copy of input data during inference. ([#39809](https://github.com/PaddlePaddle/Paddle/pull/39809)) - - -#### **New functions** - -- Add ONNX Runtime backend support. Currently it supports only CPU in the integrated version. ([#39988](https://github.com/PaddlePaddle/Paddle/pull/39988), [#40561](https://github.com/PaddlePaddle/Paddle/pull/40561)) - -- Add support for Ascend 310 inference based on the Paddle Lite subgraph approach. ([#35226](https://github.com/PaddlePaddle/Paddle/pull/35226)) - -- Add the native GPU FP16 inference. ([#40531](https://github.com/PaddlePaddle/Paddle/pull/40531)) - -- For the switch_ir_debug interface, add the dump model function. ([#36581](https://github.com/PaddlePaddle/Paddle/pull/36581)) - -- Add the configuration interface for TensorRT config: `void UpdateConfigInterleaved(paddle_infer::Config* c, bool with_interleaved)` for special data layout in int8 quantization inference. ([#38884](https://github.com/PaddlePaddle/Paddle/pull/38884)) - -- Add TensorRT inspector output information to the log. It is valid only for TensorRT 8.2 or later. ([#38362](https://github.com/PaddlePaddle/Paddle/pull/38362)，[#38200](https://github.com/PaddlePaddle/Paddle/pull/38200))) - -- Add the support of the TensorRT ASP sparse inference. ([#36413](https://github.com/PaddlePaddle/Paddle/pull/36413)) - - -### **(2) Underlying optimization** - -#### **CPU performance optimization** - -- Optimize the caching mechanism of MKLDNN. ([#38336](https://github.com/PaddlePaddle/Paddle/pull/38336), [#36980](https://github.com/PaddlePaddle/Paddle/pull/36980), [#36695](https://github.com/PaddlePaddle/Paddle/pull/36695)) - -- Add matmul_scale_fuse pass. ([#37962](https://github.com/PaddlePaddle/Paddle/pull/37962)) - -- Add MKLDNN reshape_transpose_matmul_v2_mkldnn_fuse_pass. ([#37847](https://github.com/PaddlePaddle/Paddle/pull/37847), [#40948](https://github.com/PaddlePaddle/Paddle/pull/40948)) - -- Add MKLDNN conv_hard_sigmoid_mkldnn_fuse_pass. ([#36869](https://github.com/PaddlePaddle/Paddle/pull/36869)) - -- Add MKLDNN matmul_v2_transpose_reshape_fuse_pass. ([#36481](https://github.com/PaddlePaddle/Paddle/pull/36481)) - -- Add MKLDNN softplus_activation_mkldnn_fuse_pass. ([#36657](https://github.com/PaddlePaddle/Paddle/pull/36657)) - -- Add MKLDNN elt_act_mkldnn_fuse_pass. ([#36541](https://github.com/PaddlePaddle/Paddle/pull/36541)) - -- Add MKLDNN mish operator and conv_mish_mkldnn_fuse_pass. ([#38623](https://github.com/PaddlePaddle/Paddle/pull/38623)) - - -#### **GPU performance optimization** - -- Change the inference default video memory allocation policy from `naive_best_fit` to `auto_growth`, to solve the problem of some models filled up with the GPU video memory. ([#41491](https://github.com/PaddlePaddle/Paddle/pull/41491)) - -- Support gelu and FC+gelu ops using TensorRT inference. ([#38399](https://github.com/PaddlePaddle/Paddle/pull/38399)) - -- Support `deformable_conv` inference using TensorRT under static shape. ([#36612](https://github.com/PaddlePaddle/Paddle/pull/36612) [#36850](https://github.com/PaddlePaddle/Paddle/pull/36850) [#37345](https://github.com/PaddlePaddle/Paddle/pull/37345)) - -- Support nearest_interp_v2 op using TensorRT inference. ([#34126](https://github.com/PaddlePaddle/Paddle/pull/34126)) - -- Add `yolo_box` TensorRT plugin to support input parameters `iou_aware` and `iou_aware_factor` so that the IoU computed by inference is used as a factor for confidence. ([#34128](https://github.com/PaddlePaddle/Paddle/pull/34128)) - -- Support `elementwise_sub` and `elementwise_div` calling for TensorRT inference. ([#40806](https://github.com/PaddlePaddle/Paddle/pull/40806) [#41253](https://github.com/PaddlePaddle/Paddle/pull/41253)) - -- Support `multiclass_nms3` using TensorRT inference. ([#41181](https://github.com/PaddlePaddle/Paddle/pull/41181) [#41344](https://github.com/PaddlePaddle/Paddle/pull/41344)) - -- Support flatten_contiguous_rang op using TensorRT inference. ([#38922](https://github.com/PaddlePaddle/Paddle/pull/38922)) - -- Support for `pool2d` attribute `padding` using TensorRT inference when dimension is 4, and `global_pooling` and `ceil_mode` are True. ([#39545](https://github.com/PaddlePaddle/Paddle/pull/39545)) - -- Support batch_norm and elementwise_add using TensorRT inference when dimension is 5. ([#36446](https://github.com/PaddlePaddle/Paddle/pull/36446)) - -- Add pool3d to use TensorRT inference. ([#36545](https://github.com/PaddlePaddle/Paddle/pull/36545), [#36783](https://github.com/PaddlePaddle/Paddle/pull/36783)) - -- Add the `reduce` int32 and float types to use TensorRT inference. Add `reduce_mean` GPU operator int32 and int64 registration. ([#39088](https://github.com/PaddlePaddle/Paddle/pull/39088)) - -- Modify MatmulV2ToMul pass. Modify the qualifier (not support of broadcast) and op_teller mapping condition. ([#36652](https://github.com/PaddlePaddle/Paddle/pull/36652)) - -- Add the support for TenorRT plugin interface AddPluginV2IOExt. ([#36493](https://github.com/PaddlePaddle/Paddle/pull/36493)) - -- Add the aligned attribute in roi_align op and support for TensorRT inference. ([#38905](https://github.com/PaddlePaddle/Paddle/pull/38905)) - -- Add the support for TensorRT inference with concat attribute `axis = -1`. ([#39096](https://github.com/PaddlePaddle/Paddle/pull/39096)) - -- Add TensorRT plugin: preln_emb_eltwise_layernorm, preln_skip_la, and rnorm ops, for ERNIE-like model performance optimization. ([#39570](https://github.com/PaddlePaddle/Paddle/pull/39570)) - -- Add TensorRT fuse pass: preln_embedding_eltwise_layernorm_fuse_pass, preln_skip_layernorm_fuse_pass, for ERNIE-like model performance optimization. ([#39508](https://github.com/PaddlePaddle/Paddle/pull/39508)) - -- Split matmul fusion-related passes based on different backends (GPU, CPU, TensorRT), to support transpose function for FC weights. ([#39369](https://github.com/PaddlePaddle/Paddle/pull/39369)) - -- Add the support to TensorRT by roll, strided_slice, and slice op in case of dynamic shapes. ([#41913](https://github.com/PaddlePaddle/Paddle/pull/41913), [#41573](https://github.com/PaddlePaddle/Paddle/pull/41573), [#41467](https://github.com/PaddlePaddle/Paddle/pull/41467)) - -- Add div op support for TensorRT. ([#41243](https://github.com/PaddlePaddle/Paddle/pull/41243)) - -- Quantization support - - - For the `PostTrainingQuantization` API, add the support for `paddle.io.DataLoader` object or `Python Generator` input. ([#38686](https://github.com/PaddlePaddle/Paddle/pull/38686)) - - - ERNIE full quantization model inference supports for interleaved data layout. ([#39424](https://github.com/PaddlePaddle/Paddle/pull/39424)) - - - Support for PaddleSlim new quantile model format inference. ([#41049](https://github.com/PaddlePaddle/Paddle/pull/41049)) - - - Add matmul int8 quantization inference op converter and plugin. ([#37285](https://github.com/PaddlePaddle/Paddle/pull/37285)) - - - Add pass to determine if all ops in the model can support int8 quantization. ([#36042](https://github.com/PaddlePaddle/Paddle/pull/36042)) - - - Support quantization inference for the FC part of the multihead attention of the non-variable-length branch. ([#39660](https://github.com/PaddlePaddle/Paddle/pull/39660)) - - -#### **Ascend NPU Related Features** - -- - Refactor shape operator forward computation logic to support execution on NPU. ([#39613](https://github.com/PaddlePaddle/Paddle/pull/39613)) - - - Refactor reshape operator forward computation logic to support ShapeTensor input. ([#38748](https://github.com/PaddlePaddle/Paddle/pull/38748)) - - - Uniform accuracy type when loading model weights. ([#39160](https://github.com/PaddlePaddle/Paddle/pull/39160)) - - -### **(3) Bug fixing** - -#### **Framework and API fixing** - -- Fix the bug of model clipping when saving static graphs. ([#37579](https://github.com/PaddlePaddle/Paddle/pull/37579)) - -- For the C API, add wrapper PD_Cstr for strings, and provide construction and destructing methods to avoid users to use C runtime library to destruct strings directly. ([#38667](https://github.com/PaddlePaddle/Paddle/pull/38667)) - -- Fix the logic bug with memory reuse at prediction time. ([#37324](https://github.com/PaddlePaddle/Paddle/pull/37324)) - -- Fix memory reuse error reporting in multi-threading. ([#37894](https://github.com/PaddlePaddle/Paddle/pull/37894)) - -- Allow passing empty strings for inference when no weight file is available. ([#38579](https://github.com/PaddlePaddle/Paddle/pull/38579)) - -- Fix an issue of clone not being supported when TensorRT dynamic shape is enabled. ([#38520](https://github.com/PaddlePaddle/Paddle/pull/38520)) - -- Fix multi-threaded clone error after TensorRT dynamic shape is enabled. ([#40067](https://github.com/PaddlePaddle/Paddle/pull/40067)) - -- Fix a TensorRT engine destructing issue. ([#35842](https://github.com/PaddlePaddle/Paddle/pull/35842), [#35938](https://github.com/PaddlePaddle/Paddle/pull/35938)) - -- For the lite xpu interface, fix an issue where the xpu card cannot be selected. ([#36610](https://github.com/PaddlePaddle/Paddle/pull/36610)) - -- The TensorRT dynamic shape parameter automatically generate the interface, to add the file existence check. ([#36628](https://github.com/PaddlePaddle/Paddle/pull/36628)) - -- Fix the bug that the MKLDNN does not support conv3d. ([#42055](https://github.com/PaddlePaddle/Paddle/pull/42055)) - -#### **Backend Capability Fixing** - -- Fix cuDNN default algorithm selection configuration for prediction, with using non-deterministic policies. ([#41491](https://github.com/PaddlePaddle/Paddle/pull/41491)) - -- Fix the bug with deformable_conv op in TensorRT plugin resource recovery handling error. ([#38374](https://github.com/PaddlePaddle/Paddle/pull/38374)) - -- Fix a serialization error in the TensorRT plugin for deformable_conv op. ([#38057](https://github.com/PaddlePaddle/Paddle/pull/38057)) - -- Adapt the new refactor engine and serialization API of TensorRT 8.0. ([#36769](https://github.com/PaddlePaddle/Paddle/pull/36769)) - -- Fix the bug that the Flatten2MatmulFusePass, Squeeze2MatmulFusePass, and Reshape2MatmulFusePass do not take effect. ([#37644](https://github.com/PaddlePaddle/Paddle/pull/37644)) - -- Fix the bug with TensorRT input data reporting errors. ([#37427](https://github.com/PaddlePaddle/Paddle/pull/37427)) - -- Add error message when input dimension is wrong. ([#38962](https://github.com/PaddlePaddle/Paddle/pull/38962)) - -- Fix the bug with EmbEltwiseLayernorm output type error. ([#40015](https://github.com/PaddlePaddle/Paddle/pull/40015)) - -- Remove conv_affine_channel_fuse_pass and the corresponding unit test. ([#39817](https://github.com/PaddlePaddle/Paddle/pull/39817)) - -- Fix an issue where the adaptive_pool2d pass incorrectly replaces the pool attribute. ([#39600](https://github.com/PaddlePaddle/Paddle/pull/39600)) - -- Fix the bug that shuffle_channel_detect_pass incorrectly generates shuffle_channel op. ([#39242](https://github.com/PaddlePaddle/Paddle/pull/39242)) - -- Fix transpose parameter error. ([#39006](https://github.com/PaddlePaddle/Paddle/pull/39006)) - -- Fix the crash bug when nearest_interp_v2 input scale dimension is less than 1. ([#38725](https://github.com/PaddlePaddle/Paddle/pull/38725)) - -- Fix the bug that the prelu does not support one-dimensional input in dynamic shape. ([#39389](https://github.com/PaddlePaddle/Paddle/pull/39389)) - -- Fix the bug in the kernel function of slice's special_slice_plugin. ([#39875](https://github.com/PaddlePaddle/Paddle/pull/39875)) - -- Temporarily disable int8 branch under skip_layernorm variable length to prevent accuracy degradation. ([#39991](https://github.com/PaddlePaddle/Paddle/pull/39991)) - -- Fix some bugs regarding support for preln_ernie models. ([#39733](https://github.com/PaddlePaddle/Paddle/pull/39733)) - -- Fix the bug that slice may exceed threads limit in ERNIE. Fix the bug that the spacial_slice is incorrectly triggered. ([#39096](https://github.com/PaddlePaddle/Paddle/pull/39096)) - -- Fix the bug that the elementwise does not support broadcast when the dimension is the same. ([#37908](https://github.com/PaddlePaddle/Paddle/pull/37908)) - -- Fix the problem that the underlying implementation is different in the nearest_interp op when align_corners is True and TensorRT layer results and native op have diff. ([#37525](https://github.com/PaddlePaddle/Paddle/pull/37525)) - -- Fix qkv_plugin: Kernel function computation error. ([#37096](https://github.com/PaddlePaddle/Paddle/pull/37096)) - -- Fix the bug with inference pass for dynamic quantization. ([#35879](https://github.com/PaddlePaddle/Paddle/pull/35879)) - -- Reuse directly when Tensor requests less memory than the allocated size. ([#37880](https://github.com/PaddlePaddle/Paddle/pull/37880)) - -- Fix the hang bug when ERNIE fixed-length model is enabled with TensorRT. ([#37839](https://github.com/PaddlePaddle/Paddle/pull/37839)) - -- Fix the crash bug when TensorRT int8 lacks of dynamic range information. ([#36900](https://github.com/PaddlePaddle/Paddle/pull/36900)) - -- Fix the bug with slice deserialization code. ([#36588](https://github.com/PaddlePaddle/Paddle/pull/36588)) - -- Fix yolo box calculation formula error. ([#36240](https://github.com/PaddlePaddle/Paddle/pull/36240)) - -- Fix the crash bug when the earlier version model uses a later version of roi_align. ([#38788](https://github.com/PaddlePaddle/Paddle/pull/38788)) External Developers - -- Fix the bug of a large performance difference of softmax between python and C++. ([#37130](https://github.com/PaddlePaddle/Paddle/pull/37130)) - -- Fix matmul inference failure on static shape 2-dimensional input and dynamic shape 3-dimensional input. ([#36849](https://github.com/PaddlePaddle/Paddle/pull/36849)) - -- Fix reshape_transpose_matmul_mkldnn_fuse_pass mishandling of shapes. ([#36731](https://github.com/PaddlePaddle/Paddle/pull/36731)) - -- Fix an issue where TensorRT gets 4 dimensions when the input is 2 dimensions. ([#36614](https://github.com/PaddlePaddle/Paddle/pull/36614)) - -- Fix the bug report when the interpolate_v2 MKLDNN operator is null in the scale attribute. ([#36623](https://github.com/PaddlePaddle/Paddle/pull/36623)) - -- Fix poor performance of the recurrent operator in multi-threaded scenarios. ([#36052](https://github.com/PaddlePaddle/Paddle/pull/36052)) - -- Remove restrictions of relu, sigmoid, tanh, relu6, batch_norm, clip, concat, gelu, hard_sigmoid, prelu, softmax, split, and swish on TensorRT 2-dimensional inputs. ([#37097](https://github.com/PaddlePaddle/Paddle/pull/37097)) - -- Fix reshape op to use TensorRT inference. ([#41090](https://github.com/PaddlePaddle/Paddle/pull/41090)) - -- Fix matmul related pass, which is compatible with matmul_v2. ([#36424](https://github.com/PaddlePaddle/Paddle/pull/36424)) - -- Support VALID and SAME attributes in the padding method of the conv2d operator when TensorRT is enabled. ([#38999](https://github.com/PaddlePaddle/Paddle/pull/38999)) - -- Fix MKLDNN multi-input operator quantization problem. ([#39593](https://github.com/PaddlePaddle/Paddle/pull/39593), [#39346](https://github.com/PaddlePaddle/Paddle/pull/39346), [#40717](https://github.com/PaddlePaddle/Paddle/pull/40717)) - -- Fix scale error of conv+activation in MKLDNN quantization scenarios. ([#38331](https://github.com/PaddlePaddle/Paddle/pull/38331)) - -- Fix the bug in MKLDNN quantization without parameters where the quantization of subsequent operators is handled differently. ([#39342](https://github.com/PaddlePaddle/Paddle/pull/39342)) - -- Fix a data type related issue in MKLDNN cpu_bfloat16_placement_pass. ([#38702](https://github.com/PaddlePaddle/Paddle/pull/38702)) - -- Fix a split operator execution issue in MKLDNN bfloat16 inference. ([#39548](https://github.com/PaddlePaddle/Paddle/pull/39548)) - -- Fix the bug with MKLDNN matmul_v2 operator not supporting 6 dimensions. ([#36342](https://github.com/PaddlePaddle/Paddle/pull/36342), [#38665](https://github.com/PaddlePaddle/Paddle/pull/38665)) - -- Fix MKLDNN DeviceContext error in MKLDNN matmul_v2_transpose_reshape. ([#38554](https://github.com/PaddlePaddle/Paddle/pull/38554)) - -- Fix incorrectly calculated results for segmentation models in MKLDNN inference scenarios. ([#37310](https://github.com/PaddlePaddle/Paddle/pull/37310)) - -- Fix MKLDNN bfloat16 placement operator list and add the missing operator. ([#36291](https://github.com/PaddlePaddle/Paddle/pull/36291)) - -- Fix the format bug of MKLDNN operators, including: FC, conv_transpose, 6-dimensional Tensor error reporting, and wrong output format of conv to NHWC input. ([#38890](https://github.com/PaddlePaddle/Paddle/pull/38890), [#37344](https://github.com/PaddlePaddle/Paddle/pull/37344), [#37175](https://github.com/PaddlePaddle/Paddle/pull/37175), [#38553](https://github.com/PaddlePaddle/Paddle/pull/38553), [#40049](https://github.com/PaddlePaddle/Paddle/pull/40049), [#39097](https://github.com/PaddlePaddle/Paddle/pull/39097)) - -- Fix MKLDNN multi-threaded reasoning scenario error due to cache mechanism. ([#36290](https://github.com/PaddlePaddle/Paddle/pull/36290), [#35884](https://github.com/PaddlePaddle/Paddle/pull/35884)) - -- Fix MKLDNN quantization model accuracy anomaly caused by matmul and FC. ([#38023](https://github.com/PaddlePaddle/Paddle/pull/38023), [#37618](https://github.com/PaddlePaddle/Paddle/pull/37618)) - -- Fix the abnormal quantization model accuracy issue in MKLDNN quantization conversion scripts caused by missing passes. ([#37619](https://github.com/PaddlePaddle/Paddle/pull/37619), [#40542](https://github.com/PaddlePaddle/Paddle/pull/40542),[#38912](https://github.com/PaddlePaddle/Paddle/pull/38912)) - -- Fix the crash bug in MKLDNN enabling volume op due to data type mismatch. ([#38133](https://github.com/PaddlePaddle/Paddle/pull/38133)) - -- Fix an issue where some MKLDNN ops need to change back to the original layout after modifying the layout. ([#39422](https://github.com/PaddlePaddle/Paddle/pull/39422)) - -- Fix the bug of Python API error report due to conflict with Ascend software stack, because the GIL lock is not released in the Ascend 910 inference scenario. ([#38605](https://github.com/PaddlePaddle/Paddle/pull/38605)) - - -## **5. Environment Adaptation** - -### **Compile and Install** - -- From version 2.3.0, PaddlePaddle has adjusted and upgraded the types of GPU architectures supported by the framework. (For more information, please refer to: [GPU architectures supported by PaddlePaddle](https://www.paddlepaddle.org.cn/documentation/docs/zh/2.3rc/install/Tables.html#gpu)) - - -Notes: - -- PIP source installation means downloading the installation package and dependency libraries from PIP official website with using `pip install paddlepaddle` or `pip install paddlepaddle-gpu`. This supports less architecture types, and lighter installation package,and only one CUDA version of the installation package is provided(compared with BOS source). - - - Prior to version 2.3, the PIP source installer (CUDA10.2) supports the following GPU architectures: 3.5, 5.0, 5.2, 6.0, 6.1, 7.0, and 7.5. - - - Later than version 2.3, the PIP source installer (CUDA11.0) supports the following GPU architectures: 6.0, 6.1, 7.0, 7.5, 8.0 - -- The BOS source is a way to download the installation package and dependency libraries from the official website of PaddlePaddle, which supports more GPU architectures. The download source is from China and it is much faster. (compared with PIP source, it supports more kinds of architectures and provides multiple CUDA versions of installation packages). - - - Prior to version 2.3, the GPU architectures supported by the bos source installer on the PaddlePaddle website: - - CUDA10: 3.5, 5.0, 5.2, 6.0, 6.1, 7.0, 7.5； +#### New Features - - CUDA11: 5.2，6.0，6.1，7.0，7.5，8.0。 +- Add the support for Hygon DCU K100. [#63535](https://github.com/PaddlePaddle/Paddle/pull/63535) +- Support the complex64/128 data type and fusion operators such as fused_bias_residual_layernorm, fused_bias_dropout_residual_layer_norm, and rms_norm. [#63217](https://github.com/PaddlePaddle/Paddle/pull/63217) - - Later than version 2.3, the GPU architectures supported by the bos source installer on the PaddlePaddle website: +#### Bug Fixing - - CUDA10: 3.5, 5.0, 5.2, 6.0, 6.1, 7.0, 7.5； +- Fix compilation error issues in DTK and ROCM version upgrades. [#62832](https://github.com/PaddlePaddle/Paddle/pull/62832),[#62931](https://github.com/PaddlePaddle/Paddle/pull/62931),[#61872](https://github.com/PaddlePaddle/Paddle/pull/61872),[#63738](https://github.com/PaddlePaddle/Paddle/pull/63738) - - CUDA11: 3.5, 5.0, 6.0, 6.1, 7.0, 7.5, 8.0。 +## Environment Updates -- Support Python 3.10. Fix compilation bugs caused by some PythonC API changes on Windows. ([#41180](https://github.com/PaddlePaddle/Paddle/pull/42180)) +In this PaddlePaddle version, we complete the release and update synchronization of the basic dependency libraries, and remove the old dependency libraries that are no longer updated. Complete a number of optimizations to improve compilation efficiency and compatibility, and improve the CI pipeline monitoring function to enhance the user installation experience. Fixe the several known compilation problems, improved the compilation system of paddle, and add some new features. Through the optimizations, the compilation and installation experience of the PaddlePaddle framework is further improved to bring developers a better use and development experience. -- The Windows platform supports the compilation through Visual Studio 2019. ([#38719](https://github.com/PaddlePaddle/Paddle/pull/38719)) +### New Support -- Eliminate various warnings when compiling on the Windows platform. ([#38034](https://github.com/PaddlePaddle/Paddle/pull/38034), [#37890](https://github.com/PaddlePaddle/Paddle/pull/37890), [#37442](https://github.com/PaddlePaddle/Paddle/pull/37442), [#37439](https://github.com/PaddlePaddle/Paddle/pull/37439), [#36857](https://github.com/PaddlePaddle/Paddle/pull/36857)) +- Support users to install paddle without relying on local cuda and cudnn, thus improving the user installation experience. [#60841](https://github.com/PaddlePaddle/Paddle/pull/60841)，[#61973](https://github.com/PaddlePaddle/Paddle/pull/61973),[#61862](https://github.com/PaddlePaddle/Paddle/pull/61862),[#61235](https://github.com/PaddlePaddle/Paddle/pull/61235),[#61209](https://github.com/PaddlePaddle/Paddle/pull/61209),[#61653](https://github.com/PaddlePaddle/Paddle/pull/61653),[#64083](https://github.com/PaddlePaddle/Paddle/pull/64083) +- Support CUDA 12.3 completely. Complete the retirement of cuda10.2. [#63356](https://github.com/PaddlePaddle/Paddle/pull/63356),[#60299](https://github.com/PaddlePaddle/Paddle/pull/60299),[#64171](https://github.com/PaddlePaddle/Paddle/pull/64171),[#62189](https://github.com/PaddlePaddle/Paddle/pull/62189),[#63392](https://github.com/PaddlePaddle/Paddle/pull/63392),[#64228](https://github.com/PaddlePaddle/Paddle/pull/64228),[#62498](https://github.com/PaddlePaddle/Paddle/pull/62498),[#64298](https://github.com/PaddlePaddle/Paddle/pull/64298) +- Support Python 3.12 completely, bringing more powerful language features and performance optimizations. Complete the retirement of python3.7. [#59875](https://github.com/PaddlePaddle/Paddle/pull/59875)，[#59877](https://github.com/PaddlePaddle/Paddle/pull/59877),[#59876](https://github.com/PaddlePaddle/Paddle/pull/59876) +- Upgrade of other paddle-dependent third-party libraries: [#63741](https://github.com/PaddlePaddle/Paddle/pull/63741),[#64447](https://github.com/PaddlePaddle/Paddle/pull/64447),[#60195](https://github.com/PaddlePaddle/Paddle/pull/60195),[#60110](https://github.com/PaddlePaddle/Paddle/pull/60110),[#61509](https://github.com/PaddlePaddle/Paddle/pull/61509) -- Fix jetson compilation issues introduced by the underlying data structure upgrade. ([#39669](https://github.com/PaddlePaddle/Paddle/pull/39669), [#39441](https://github.com/PaddlePaddle/Paddle/pull/39441)) +### Compilation Optimizations +- Optimize paddle's CMake codes, significantly improving compilation efficiency and experience. [##59995](https://github.com/PaddlePaddle/Paddle/pull/59995),[#60167](https://github.com/PaddlePaddle/Paddle/pull/60167)，[#61052](https://github.com/PaddlePaddle/Paddle/pull/61052)，[#59995](https://github.com/PaddlePaddle/Paddle/pull/59995)，[#59607](https://github.com/PaddlePaddle/Paddle/pull/59607),[#63093](https://github.com/PaddlePaddle/Paddle/pull/63093),[#63887](https://github.com/PaddlePaddle/Paddle/pull/63887),[#62969](https://github.com/PaddlePaddle/Paddle/pull/62969),[#64007](https://github.com/PaddlePaddle/Paddle/pull/64007),[#59811](https://github.com/PaddlePaddle/Paddle/pull/59811),[#63045](https://github.com/PaddlePaddle/Paddle/pull/63045),[#60235](https://github.com/PaddlePaddle/Paddle/pull/60235),[#60240](https://github.com/PaddlePaddle/Paddle/pull/60240),[#60235](https://github.com/PaddlePaddle/Paddle/pull/60235),[#61411](https://github.com/PaddlePaddle/Paddle/pull/61411),[#61944](https://github.com/PaddlePaddle/Paddle/pull/61944),[#61961](https://github.com/PaddlePaddle/Paddle/pull/61961),[#59990](https://github.com/PaddlePaddle/Paddle/pull/59990),[#59478](https://github.com/PaddlePaddle/Paddle/pull/59478),[#61501](https://github.com/PaddlePaddle/Paddle/pull/61501),[#60066](https://github.com/PaddlePaddle/Paddle/pull/60066),[#64133](https://github.com/PaddlePaddle/Paddle/pull/64133),[#64231](https://github.com/PaddlePaddle/Paddle/pull/64231),[#60087](https://github.com/PaddlePaddle/Paddle/pull/60087),[#60348](https://github.com/PaddlePaddle/Paddle/pull/60348),[#60737](https://github.com/PaddlePaddle/Paddle/pull/60737),[#61364](https://github.com/PaddlePaddle/Paddle/pull/61364),[#63214](https://github.com/PaddlePaddle/Paddle/pull/63214),[#62454](https://github.com/PaddlePaddle/Paddle/pull/62454),[#62473](https://github.com/PaddlePaddle/Paddle/pull/62473),[#63692](https://github.com/PaddlePaddle/Paddle/pull/63692),[#63950](https://github.com/PaddlePaddle/Paddle/pull/63950) +- Support C++ unit test link dynamic library under linux and windowx, greatly reducing the size of C++ unit test and the size of the entire build directory. [#60008](https://github.com/PaddlePaddle/Paddle/pull/60008),[#60960](https://github.com/PaddlePaddle/Paddle/pull/60960),[#60960](https://github.com/PaddlePaddle/Paddle/pull/60960),[#60961](https://github.com/PaddlePaddle/Paddle/pull/60961),[#60831](https://github.com/PaddlePaddle/Paddle/pull/60831),[#60832](https://github.com/PaddlePaddle/Paddle/pull/60832),[#60833](https://github.com/PaddlePaddle/Paddle/pull/60833),[#61372](https://github.com/PaddlePaddle/Paddle/pull/61372),[#60834](https://github.com/PaddlePaddle/Paddle/pull/60834),[#61374](https://github.com/PaddlePaddle/Paddle/pull/61374),[#61463](https://github.com/PaddlePaddle/Paddle/pull/61463),[#61376](https://github.com/PaddlePaddle/Paddle/pull/61376),[#60830](https://github.com/PaddlePaddle/Paddle/pull/60830),[#61373](https://github.com/PaddlePaddle/Paddle/pull/61373),[#61672](https://github.com/PaddlePaddle/Paddle/pull/61672),[#61375](https://github.com/PaddlePaddle/Paddle/pull/61375),[#61676](https://github.com/PaddlePaddle/Paddle/pull/61676),[#62036](https://github.com/PaddlePaddle/Paddle/pull/62036),[#61945](https://github.com/PaddlePaddle/Paddle/pull/61945),[#61675](https://github.com/PaddlePaddle/Paddle/pull/61675),[#61674](https://github.com/PaddlePaddle/Paddle/pull/61674),[#62773](https://github.com/PaddlePaddle/Paddle/pull/62773),[#61238](https://github.com/PaddlePaddle/Paddle/pull/61238),[#59988](https://github.com/PaddlePaddle/Paddle/pull/59988),[#60307](https://github.com/PaddlePaddle/Paddle/pull/60307),[#59612](https://github.com/PaddlePaddle/Paddle/pull/59612),[#59942](https://github.com/PaddlePaddle/Paddle/pull/59942),[#59968](https://github.com/PaddlePaddle/Paddle/pull/59968),[#59978](https://github.com/PaddlePaddle/Paddle/pull/59978),[#60121](https://github.com/PaddlePaddle/Paddle/pull/60121),[#60149](https://github.com/PaddlePaddle/Paddle/pull/60149),[#60161](https://github.com/PaddlePaddle/Paddle/pull/60161),[#60160](https://github.com/PaddlePaddle/Paddle/pull/60160),[#60230](https://github.com/PaddlePaddle/Paddle/pull/60230),[#60154](https://github.com/PaddlePaddle/Paddle/pull/60154),[#60356](https://github.com/PaddlePaddle/Paddle/pull/60356),[#60392](https://github.com/PaddlePaddle/Paddle/pull/60392),[#60517](https://github.com/PaddlePaddle/Paddle/pull/60517),[#61131](https://github.com/PaddlePaddle/Paddle/pull/61131),[#60959](https://github.com/PaddlePaddle/Paddle/pull/60959) +- Add the support for Clang compiler. Users can now use Clang to compile, enjoying faster compilation speed and better error message prompts. [#63382](https://github.com/PaddlePaddle/Paddle/pull/63382),[#63133](https://github.com/PaddlePaddle/Paddle/pull/63133),[#61705](https://github.com/PaddlePaddle/Paddle/pull/61705),[#63152](https://github.com/PaddlePaddle/Paddle/pull/63152),[#63373](https://github.com/PaddlePaddle/Paddle/pull/63373) -### **New Hardware Backend Extention** +### CI Pipeline Improvements -- Custom device support: provide a plug-in way to extend PaddlePaddle hardware backend. With this function, developers do not need to modify PaddlePaddle codes for specific hardware, but simply implement the standard interface and compile it into a dynamic link library that can be called by PaddlePaddle as a plug-in.This reduces the development effort of adding a new hardware backend to PaddlePaddle. Currently it supports custom Runtime and custom Kernel. +- Improve the merge-in code monitoring mechanism in the CI pipeline, to ensure higher code quality and stability. Add a function monitoring module, to monitor various indicators of the CI pipeline in real time, ensuring smooth execution of each stage, to identify and resolve issues in a timely manner. [#61384](https://github.com/PaddlePaddle/Paddle/pull/61384),[#62190](https://github.com/PaddlePaddle/Paddle/pull/62190),[#60758](https://github.com/PaddlePaddle/Paddle/pull/60758),[#60399](https://github.com/PaddlePaddle/Paddle/pull/60399),[#58623](https://github.com/PaddlePaddle/Paddle/pull/58623),[#62177](https://github.com/PaddlePaddle/Paddle/pull/62177),[#62361](https://github.com/PaddlePaddle/Paddle/pull/62361),[#62893](https://github.com/PaddlePaddle/Paddle/pull/62893),[#63705](https://github.com/PaddlePaddle/Paddle/pull/63705),[#64476](https://github.com/PaddlePaddle/Paddle/pull/64476),[#64752](https://github.com/PaddlePaddle/Paddle/pull/64752),[#64733](https://github.com/PaddlePaddle/Paddle/pull/64733),[#61914](https://github.com/PaddlePaddle/Paddle/pull/61914) -- Support Huawei NPU chip (Ascend910) training/inference. Support ResNet50, YoloV3, BERT, Transformer and many other models. Support static + dynamic graph and auto-mixed precision training. Support single card, and distribute training across multiple cards, multiple machines. +### Code Cleanup -- Support Graphcore IPU chip (including IPU Mk2 GC200 and Bow IPU) training/inference. Support ResNet50, BERT and other models. Support static graph training. Support single card, and distribute training across multiple cards, multiple machines. +- Remove some old codes. [#63580](https://github.com/PaddlePaddle/Paddle/pull/63580),[#62840](https://github.com/PaddlePaddle/Paddle/pull/62840),[#62886](https://github.com/PaddlePaddle/Paddle/pull/62886),[#63046](https://github.com/PaddlePaddle/Paddle/pull/63046),[#63004](https://github.com/PaddlePaddle/Paddle/pull/63004),[#63039](https://github.com/PaddlePaddle/Paddle/pull/63039),[#62733](https://github.com/PaddlePaddle/Paddle/pull/62733),[#62773](https://github.com/PaddlePaddle/Paddle/pull/62773),[#62768](https://github.com/PaddlePaddle/Paddle/pull/62768),[#62744](https://github.com/PaddlePaddle/Paddle/pull/62744),[#62861](https://github.com/PaddlePaddle/Paddle/pull/62861),[#62774](https://github.com/PaddlePaddle/Paddle/pull/62774),[#62851](https://github.com/PaddlePaddle/Paddle/pull/62851),[#62973](https://github.com/PaddlePaddle/Paddle/pull/62973),[#63273](https://github.com/PaddlePaddle/Paddle/pull/63273),[#62445](https://github.com/PaddlePaddle/Paddle/pull/62445),[#64382](https://github.com/PaddlePaddle/Paddle/pull/64382),[#64409](https://github.com/PaddlePaddle/Paddle/pull/64409),[#64391](https://github.com/PaddlePaddle/Paddle/pull/64391),[#64310](https://github.com/PaddlePaddle/Paddle/pull/64310),[#64348](https://github.com/PaddlePaddle/Paddle/pull/64348),[#64651](https://github.com/PaddlePaddle/Paddle/pull/64651),[#64709](https://github.com/PaddlePaddle/Paddle/pull/64709),[#61714](https://github.com/PaddlePaddle/Paddle/pull/61714),[#62109](https://github.com/PaddlePaddle/Paddle/pull/62109),[#61751](https://github.com/PaddlePaddle/Paddle/pull/61751),[#61691](https://github.com/PaddlePaddle/Paddle/pull/61691),[#61735](https://github.com/PaddlePaddle/Paddle/pull/61735) -- Support cambricon MLU chip (MLU370x4) training/inference. Support models such as ResNet50. Support static graph + dynamic graph training. Support auto-mixed precision training. Support single card, and distribute training across multiple cards, multiple machines. +### Bug Fixing -- Support KUNLUNXIN 2 chips (KUNLUNXIN AI acceleration cards R200, R300) training/inference. Support ResNet50, YoloV3, OCR-DB, SSD, MobilnetV3, UNet, BERT, Transformer, GPT-2, Wide&Deep, and DeepFM. Support static graph + dynamic graph training. Support auto-mixed precision training. Support single card, and distribute training across multiple cards, multiple machines. +- Fix several compilation issues of paddle framework. [#63297](https://github.com/PaddlePaddle/Paddle/pull/63297)，[#62994](https://github.com/PaddlePaddle/Paddle/pull/62994)，[#62651](https://github.com/PaddlePaddle/Paddle/pull/62651)，[#64408](https://github.com/PaddlePaddle/Paddle/pull/64408)，[#60934](https://github.com/PaddlePaddle/Paddle/pull/60934)，[#62899](https://github.com/PaddlePaddle/Paddle/pull/62899)，[#60528](https://github.com/PaddlePaddle/Paddle/pull/60528)，[#63158](https://github.com/PaddlePaddle/Paddle/pull/63158)，[#64549](https://github.com/PaddlePaddle/Paddle/pull/64549)，[#62351](https://github.com/PaddlePaddle/Paddle/pull/62351)，[#61259](https://github.com/PaddlePaddle/Paddle/pull/61259),[#61281](https://github.com/PaddlePaddle/Paddle/pull/61281),[#62304](https://github.com/PaddlePaddle/Paddle/pull/62304),[#60736](https://github.com/PaddlePaddle/Paddle/pull/60736),[#60811](https://github.com/PaddlePaddle/Paddle/pull/60811),[#63949](https://github.com/PaddlePaddle/Paddle/pull/63949),[#59892](https://github.com/PaddlePaddle/Paddle/pull/59892),[#60767](https://github.com/PaddlePaddle/Paddle/pull/60767),[#60856](https://github.com/PaddlePaddle/Paddle/pull/60856),[#61286](https://github.com/PaddlePaddle/Paddle/pull/61286),[#61638](https://github.com/PaddlePaddle/Paddle/pull/61638),[#62079](https://github.com/PaddlePaddle/Paddle/pull/62079),[#62142](https://github.com/PaddlePaddle/Paddle/pull/62142),[#62823](https://github.com/PaddlePaddle/Paddle/pull/62823),[#62814](https://github.com/PaddlePaddle/Paddle/pull/62814),[#62425](https://github.com/PaddlePaddle/Paddle/pull/62425),[#62619](https://github.com/PaddlePaddle/Paddle/pull/62619),[#60207](https://github.com/PaddlePaddle/Paddle/pull/60207),[#60765](https://github.com/PaddlePaddle/Paddle/pull/60765),[#61870](https://github.com/PaddlePaddle/Paddle/pull/61870),[#61923](https://github.com/PaddlePaddle/Paddle/pull/61923),[#62144](https://github.com/PaddlePaddle/Paddle/pull/62144),[#62426](https://github.com/PaddlePaddle/Paddle/pull/62426),[#63848](https://github.com/PaddlePaddle/Paddle/pull/63848),[#60682](https://github.com/PaddlePaddle/Paddle/pull/60682),[#61369](https://github.com/PaddlePaddle/Paddle/pull/61369),[#62882](https://github.com/PaddlePaddle/Paddle/pull/62882),[#63944](https://github.com/PaddlePaddle/Paddle/pull/63944),[#64812](https://github.com/PaddlePaddle/Paddle/pull/64812),[#60654](https://github.com/PaddlePaddle/Paddle/pull/60654),[#60887](https://github.com/PaddlePaddle/Paddle/pull/60887),[#62058](https://github.com/PaddlePaddle/Paddle/pull/62058),[#64639](https://github.com/PaddlePaddle/Paddle/pull/64639),[#60115](https://github.com/PaddlePaddle/Paddle/pull/60115),[#61940](https://github.com/PaddlePaddle/Paddle/pull/61940),[#62614](https://github.com/PaddlePaddle/Paddle/pull/62614),[#59914](https://github.com/PaddlePaddle/Paddle/pull/59914),[#63762](https://github.com/PaddlePaddle/Paddle/pull/63762),[#60145](https://github.com/PaddlePaddle/Paddle/pull/60145),[#60285](https://github.com/PaddlePaddle/Paddle/pull/60285),[#60378](https://github.com/PaddlePaddle/Paddle/pull/60378),[#60393](https://github.com/PaddlePaddle/Paddle/pull/60393),[#61057](https://github.com/PaddlePaddle/Paddle/pull/61057),[#61058](https://github.com/PaddlePaddle/Paddle/pull/61058),[#61151](https://github.com/PaddlePaddle/Paddle/pull/61151),[#61347](https://github.com/PaddlePaddle/Paddle/pull/61347),[#61554](https://github.com/PaddlePaddle/Paddle/pull/61554),[#61844](https://github.com/PaddlePaddle/Paddle/pull/61844),[#62915](https://github.com/PaddlePaddle/Paddle/pull/62915),[#61852](https://github.com/PaddlePaddle/Paddle/pull/61852),[#61704](https://github.com/PaddlePaddle/Paddle/pull/61704),[#61991](https://github.com/PaddlePaddle/Paddle/pull/61991),[#62264](https://github.com/PaddlePaddle/Paddle/pull/62264),[#62762](https://github.com/PaddlePaddle/Paddle/pull/62762),[#63820](https://github.com/PaddlePaddle/Paddle/pull/63820),[#63864](https://github.com/PaddlePaddle/Paddle/pull/63864),[#65017](https://github.com/PaddlePaddle/Paddle/pull/65017),[#61183](https://github.com/PaddlePaddle/Paddle/pull/61183),[#59866](https://github.com/PaddlePaddle/Paddle/pull/59866),[#61171](https://github.com/PaddlePaddle/Paddle/pull/61171),[#61290](https://github.com/PaddlePaddle/Paddle/pull/61290),[#61725](https://github.com/PaddlePaddle/Paddle/pull/61725),[#61614](https://github.com/PaddlePaddle/Paddle/pull/61614),[#61721](https://github.com/PaddlePaddle/Paddle/pull/61721),[#61494](https://github.com/PaddlePaddle/Paddle/pull/61494),[#61556](https://github.com/PaddlePaddle/Paddle/pull/61556),[#61689](https://github.com/PaddlePaddle/Paddle/pull/61689) +## Documentation-related Bug Fixing -## Thanks to our Contributors +- With the enhancement of API feature, some API documentations have been fixed and enhanced simultaneously. [#62875](https://github.com/PaddlePaddle/Paddle/pull/62875), [#59793](https://github.com/PaddlePaddle/Paddle/pull/59793), [#60002](https://github.com/PaddlePaddle/Paddle/pull/60002), [#59985](https://github.com/PaddlePaddle/Paddle/pull/59985), [#63365](https://github.com/PaddlePaddle/Paddle/pull/63365), [#60962](https://github.com/PaddlePaddle/Paddle/pull/60962), [#60942](https://github.com/PaddlePaddle/Paddle/pull/60942), [#64232](https://github.com/PaddlePaddle/Paddle/pull/64232), [#63255](https://github.com/PaddlePaddle/Paddle/pull/63255) +- Update/supplement API documentation. bernoulli_ ([#64504](https://github.com/PaddlePaddle/Paddle/pull/64504)), paddle.static.ctr_metric_bundle ([#60912](https://github.com/PaddlePaddle/Paddle/pull/60912)), LayerNorm ([#62928](https://github.com/PaddlePaddle/Paddle/pull/62928)), Sequential ([#63128](https://github.com/PaddlePaddle/Paddle/pull/63128)), paddle.summary ([#63121](https://github.com/PaddlePaddle/Paddle/pull/63121)), ShardOptimizer in AutoParallel ([#62933](https://github.com/PaddlePaddle/Paddle/pull/62933)), paddle.nccl.version ([#62480](https://github.com/PaddlePaddle/Paddle/pull/62480)) +- Update the Readme file. [#59883](https://github.com/PaddlePaddle/Paddle/pull/59883),[#60691](https://github.com/PaddlePaddle/Paddle/pull/60691),[#60749](https://github.com/PaddlePaddle/Paddle/pull/60749) +- Update mkldnn to onednn. [#63199](https://github.com/PaddlePaddle/Paddle/pull/63199),[#63202](https://github.com/PaddlePaddle/Paddle/pull/63202),[#63215](https://github.com/PaddlePaddle/Paddle/pull/63215),[#63209](https://github.com/PaddlePaddle/Paddle/pull/63209) +- Fix document rendering bugs. [#59725](https://github.com/PaddlePaddle/Paddle/pull/59725),[#60306](https://github.com/PaddlePaddle/Paddle/pull/60306) +- Fix a lot of typos in the codes to enhance source readability. [#60093](https://github.com/PaddlePaddle/Paddle/pull/60093),[#60603](https://github.com/PaddlePaddle/Paddle/pull/60603),[#60631](https://github.com/PaddlePaddle/Paddle/pull/60631),[#60679](https://github.com/PaddlePaddle/Paddle/pull/60679),[#60741](https://github.com/PaddlePaddle/Paddle/pull/60741),[#60770](https://github.com/PaddlePaddle/Paddle/pull/60770),[#60784](https://github.com/PaddlePaddle/Paddle/pull/60784),[#60825](https://github.com/PaddlePaddle/Paddle/pull/60825),[#60857](https://github.com/PaddlePaddle/Paddle/pull/60857),[#60891](https://github.com/PaddlePaddle/Paddle/pull/60891),[#60921](https://github.com/PaddlePaddle/Paddle/pull/60921),[#60920](https://github.com/PaddlePaddle/Paddle/pull/60920),[#60923](https://github.com/PaddlePaddle/Paddle/pull/60923),[#60928](https://github.com/PaddlePaddle/Paddle/pull/60928),[#60940](https://github.com/PaddlePaddle/Paddle/pull/60940),[#60936](https://github.com/PaddlePaddle/Paddle/pull/60936),[#60932](https://github.com/PaddlePaddle/Paddle/pull/60932),[#60935](https://github.com/PaddlePaddle/Paddle/pull/60935),[#60931](https://github.com/PaddlePaddle/Paddle/pull/60931),[#60951](https://github.com/PaddlePaddle/Paddle/pull/60951),[#60964](https://github.com/PaddlePaddle/Paddle/pull/60964),[#60965](https://github.com/PaddlePaddle/Paddle/pull/60965),[#60967](https://github.com/PaddlePaddle/Paddle/pull/60967),[#60972](https://github.com/PaddlePaddle/Paddle/pull/60972),[#60971](https://github.com/PaddlePaddle/Paddle/pull/60971),[#60980](https://github.com/PaddlePaddle/Paddle/pull/60980),[#60984](https://github.com/PaddlePaddle/Paddle/pull/60984),[#60985](https://github.com/PaddlePaddle/Paddle/pull/60985),[#60989](https://github.com/PaddlePaddle/Paddle/pull/60989),[#60990](https://github.com/PaddlePaddle/Paddle/pull/60990),[#60991](https://github.com/PaddlePaddle/Paddle/pull/60991),[#60992](https://github.com/PaddlePaddle/Paddle/pull/60992),[#60994](https://github.com/PaddlePaddle/Paddle/pull/60994),[#60995](https://github.com/PaddlePaddle/Paddle/pull/60995),[#60996](https://github.com/PaddlePaddle/Paddle/pull/60996),[#61001](https://github.com/PaddlePaddle/Paddle/pull/61001),[#61000](https://github.com/PaddlePaddle/Paddle/pull/61000),[#60999](https://github.com/PaddlePaddle/Paddle/pull/60999),[#60998](https://github.com/PaddlePaddle/Paddle/pull/60998),[#61026](https://github.com/PaddlePaddle/Paddle/pull/61026),[#61009](https://github.com/PaddlePaddle/Paddle/pull/61009),[#61034](https://github.com/PaddlePaddle/Paddle/pull/61034),[#61033](https://github.com/PaddlePaddle/Paddle/pull/61033),[#61020](https://github.com/PaddlePaddle/Paddle/pull/61020),[#61092](https://github.com/PaddlePaddle/Paddle/pull/61092),[#61066](https://github.com/PaddlePaddle/Paddle/pull/61066),[#61063](https://github.com/PaddlePaddle/Paddle/pull/61063),[#61089](https://github.com/PaddlePaddle/Paddle/pull/61089),[#61071](https://github.com/PaddlePaddle/Paddle/pull/61071),[#61129](https://github.com/PaddlePaddle/Paddle/pull/61129),[#61128](https://github.com/PaddlePaddle/Paddle/pull/61128),[#61126](https://github.com/PaddlePaddle/Paddle/pull/61126),[#61123](https://github.com/PaddlePaddle/Paddle/pull/61123),[#61113](https://github.com/PaddlePaddle/Paddle/pull/61113),[#61189](https://github.com/PaddlePaddle/Paddle/pull/61189),[#61175](https://github.com/PaddlePaddle/Paddle/pull/61175),[#61153](https://github.com/PaddlePaddle/Paddle/pull/61153),[#61198](https://github.com/PaddlePaddle/Paddle/pull/61198),[#61206](https://github.com/PaddlePaddle/Paddle/pull/61206),[#61256](https://github.com/PaddlePaddle/Paddle/pull/61256),[#61255](https://github.com/PaddlePaddle/Paddle/pull/61255),[#61251](https://github.com/PaddlePaddle/Paddle/pull/61251),[#61246](https://github.com/PaddlePaddle/Paddle/pull/61246),[#61245](https://github.com/PaddlePaddle/Paddle/pull/61245),[#61231](https://github.com/PaddlePaddle/Paddle/pull/61231),[#61247](https://github.com/PaddlePaddle/Paddle/pull/61247),[#61265](https://github.com/PaddlePaddle/Paddle/pull/61265),[#61264](https://github.com/PaddlePaddle/Paddle/pull/61264),[#61266](https://github.com/PaddlePaddle/Paddle/pull/61266),[#61267](https://github.com/PaddlePaddle/Paddle/pull/61267),[#61268](https://github.com/PaddlePaddle/Paddle/pull/61268),[#61270](https://github.com/PaddlePaddle/Paddle/pull/61270),[#61334](https://github.com/PaddlePaddle/Paddle/pull/61334),[#61392](https://github.com/PaddlePaddle/Paddle/pull/61392),[#61404](https://github.com/PaddlePaddle/Paddle/pull/61404),[#61318](https://github.com/PaddlePaddle/Paddle/pull/61318),[#61383](https://github.com/PaddlePaddle/Paddle/pull/61383),[#61306](https://github.com/PaddlePaddle/Paddle/pull/61306),[#61324](https://github.com/PaddlePaddle/Paddle/pull/61324),[#61426](https://github.com/PaddlePaddle/Paddle/pull/61426),[#61390](https://github.com/PaddlePaddle/Paddle/pull/61390),[#61419](https://github.com/PaddlePaddle/Paddle/pull/61419),[#61420](https://github.com/PaddlePaddle/Paddle/pull/61420),[#61408](https://github.com/PaddlePaddle/Paddle/pull/61408),[#61425](https://github.com/PaddlePaddle/Paddle/pull/61425),[#61557](https://github.com/PaddlePaddle/Paddle/pull/61557),[#61628](https://github.com/PaddlePaddle/Paddle/pull/61628),[#61652](https://github.com/PaddlePaddle/Paddle/pull/61652),[#61602](https://github.com/PaddlePaddle/Paddle/pull/61602),[#61558](https://github.com/PaddlePaddle/Paddle/pull/61558),[#61660](https://github.com/PaddlePaddle/Paddle/pull/61660),[#61423](https://github.com/PaddlePaddle/Paddle/pull/61423),[#61627](https://github.com/PaddlePaddle/Paddle/pull/61627),[#61685](https://github.com/PaddlePaddle/Paddle/pull/61685),[#61690](https://github.com/PaddlePaddle/Paddle/pull/61690),[#61727](https://github.com/PaddlePaddle/Paddle/pull/61727),[#61738](https://github.com/PaddlePaddle/Paddle/pull/61738),[#61740](https://github.com/PaddlePaddle/Paddle/pull/61740),[#61741](https://github.com/PaddlePaddle/Paddle/pull/61741),[#61743](https://github.com/PaddlePaddle/Paddle/pull/61743),[#61744](https://github.com/PaddlePaddle/Paddle/pull/61744),[#61745](https://github.com/PaddlePaddle/Paddle/pull/61745),[#61761](https://github.com/PaddlePaddle/Paddle/pull/61761),[#61762](https://github.com/PaddlePaddle/Paddle/pull/61762),[#61764](https://github.com/PaddlePaddle/Paddle/pull/61764),[#61767](https://github.com/PaddlePaddle/Paddle/pull/61767),[#61768](https://github.com/PaddlePaddle/Paddle/pull/61768),[#61774](https://github.com/PaddlePaddle/Paddle/pull/61774),[#61781](https://github.com/PaddlePaddle/Paddle/pull/61781),[#61783](https://github.com/PaddlePaddle/Paddle/pull/61783),[#61757](https://github.com/PaddlePaddle/Paddle/pull/61757),[#61732](https://github.com/PaddlePaddle/Paddle/pull/61732),[#61776](https://github.com/PaddlePaddle/Paddle/pull/61776),[#61780](https://github.com/PaddlePaddle/Paddle/pull/61780),[#61730](https://github.com/PaddlePaddle/Paddle/pull/61730),[#61728](https://github.com/PaddlePaddle/Paddle/pull/61728),[#61633](https://github.com/PaddlePaddle/Paddle/pull/61633),[#61720](https://github.com/PaddlePaddle/Paddle/pull/61720),[#61734](https://github.com/PaddlePaddle/Paddle/pull/61734),[#61779](https://github.com/PaddlePaddle/Paddle/pull/61779),[#61775](https://github.com/PaddlePaddle/Paddle/pull/61775),[#61773](https://github.com/PaddlePaddle/Paddle/pull/61773),[#61787](https://github.com/PaddlePaddle/Paddle/pull/61787),[#61687](https://github.com/PaddlePaddle/Paddle/pull/61687),[#61747](https://github.com/PaddlePaddle/Paddle/pull/61747),[#61760](https://github.com/PaddlePaddle/Paddle/pull/61760),[#61782](https://github.com/PaddlePaddle/Paddle/pull/61782),[#61800](https://github.com/PaddlePaddle/Paddle/pull/61800),[#61748](https://github.com/PaddlePaddle/Paddle/pull/61748),[#61772](https://github.com/PaddlePaddle/Paddle/pull/61772),[#61786](https://github.com/PaddlePaddle/Paddle/pull/61786),[#61880](https://github.com/PaddlePaddle/Paddle/pull/61880),[#61718](https://github.com/PaddlePaddle/Paddle/pull/61718),[#61742](https://github.com/PaddlePaddle/Paddle/pull/61742),[#61766](https://github.com/PaddlePaddle/Paddle/pull/61766),[#61835](https://github.com/PaddlePaddle/Paddle/pull/61835),[#61838](https://github.com/PaddlePaddle/Paddle/pull/61838),[#61754](https://github.com/PaddlePaddle/Paddle/pull/61754),[#61833](https://github.com/PaddlePaddle/Paddle/pull/61833),[#61749](https://github.com/PaddlePaddle/Paddle/pull/61749),[#61938](https://github.com/PaddlePaddle/Paddle/pull/61938),[#61919](https://github.com/PaddlePaddle/Paddle/pull/61919),[#61924](https://github.com/PaddlePaddle/Paddle/pull/61924),[#61778](https://github.com/PaddlePaddle/Paddle/pull/61778),[#61839](https://github.com/PaddlePaddle/Paddle/pull/61839),[#61879](https://github.com/PaddlePaddle/Paddle/pull/61879),[#61929](https://github.com/PaddlePaddle/Paddle/pull/61929),[#61801](https://github.com/PaddlePaddle/Paddle/pull/61801),[#61788](https://github.com/PaddlePaddle/Paddle/pull/61788),[#61999](https://github.com/PaddlePaddle/Paddle/pull/61999),[#61928](https://github.com/PaddlePaddle/Paddle/pull/61928),[#61958](https://github.com/PaddlePaddle/Paddle/pull/61958),[#61982](https://github.com/PaddlePaddle/Paddle/pull/61982),[#61996](https://github.com/PaddlePaddle/Paddle/pull/61996),[#61953](https://github.com/PaddlePaddle/Paddle/pull/61953),[#61998](https://github.com/PaddlePaddle/Paddle/pull/61998),[#62003](https://github.com/PaddlePaddle/Paddle/pull/62003),[#61921](https://github.com/PaddlePaddle/Paddle/pull/61921),[#61881](https://github.com/PaddlePaddle/Paddle/pull/61881),[#61746](https://github.com/PaddlePaddle/Paddle/pull/61746),[#61955](https://github.com/PaddlePaddle/Paddle/pull/61955),[#62002](https://github.com/PaddlePaddle/Paddle/pull/62002),[#62001](https://github.com/PaddlePaddle/Paddle/pull/62001),[#61997](https://github.com/PaddlePaddle/Paddle/pull/61997),[#61765](https://github.com/PaddlePaddle/Paddle/pull/61765),[#61956](https://github.com/PaddlePaddle/Paddle/pull/61956),[#62004](https://github.com/PaddlePaddle/Paddle/pull/62004),[#62044](https://github.com/PaddlePaddle/Paddle/pull/62044),[#62040](https://github.com/PaddlePaddle/Paddle/pull/62040),[#62043](https://github.com/PaddlePaddle/Paddle/pull/62043),[#62042](https://github.com/PaddlePaddle/Paddle/pull/62042),[#62041](https://github.com/PaddlePaddle/Paddle/pull/62041),[#62039](https://github.com/PaddlePaddle/Paddle/pull/62039),[#62019](https://github.com/PaddlePaddle/Paddle/pull/62019),[#61910](https://github.com/PaddlePaddle/Paddle/pull/61910),[#61882](https://github.com/PaddlePaddle/Paddle/pull/61882),[#61836](https://github.com/PaddlePaddle/Paddle/pull/61836),[#62013](https://github.com/PaddlePaddle/Paddle/pull/62013),[#62055](https://github.com/PaddlePaddle/Paddle/pull/62055),[#62047](https://github.com/PaddlePaddle/Paddle/pull/62047),[#62000](https://github.com/PaddlePaddle/Paddle/pull/62000),[#62048](https://github.com/PaddlePaddle/Paddle/pull/62048),[#62075](https://github.com/PaddlePaddle/Paddle/pull/62075),[#62038](https://github.com/PaddlePaddle/Paddle/pull/62038),[#62045](https://github.com/PaddlePaddle/Paddle/pull/62045),[#62105](https://github.com/PaddlePaddle/Paddle/pull/62105),[#62214](https://github.com/PaddlePaddle/Paddle/pull/62214),[#62212](https://github.com/PaddlePaddle/Paddle/pull/62212),[#62183](https://github.com/PaddlePaddle/Paddle/pull/62183)，[#62182](https://github.com/PaddlePaddle/Paddle/pull/62182),[#62181](https://github.com/PaddlePaddle/Paddle/pull/62181),[#62179](https://github.com/PaddlePaddle/Paddle/pull/62179),[#62178](https://github.com/PaddlePaddle/Paddle/pull/62178),[#62172](https://github.com/PaddlePaddle/Paddle/pull/62172),[#62168](https://github.com/PaddlePaddle/Paddle/pull/62168),[#62163](https://github.com/PaddlePaddle/Paddle/pull/62163),[#62162](https://github.com/PaddlePaddle/Paddle/pull/62162),[#62161](https://github.com/PaddlePaddle/Paddle/pull/62161),[#62160](https://github.com/PaddlePaddle/Paddle/pull/62160),[#62046](https://github.com/PaddlePaddle/Paddle/pull/62046),[#62175](https://github.com/PaddlePaddle/Paddle/pull/62175),[#62259](https://github.com/PaddlePaddle/Paddle/pull/62259),[#62258](https://github.com/PaddlePaddle/Paddle/pull/62258),[#62213](https://github.com/PaddlePaddle/Paddle/pull/62213),[#62260](https://github.com/PaddlePaddle/Paddle/pull/62260),[#62290](https://github.com/PaddlePaddle/Paddle/pull/62290),[#62288](https://github.com/PaddlePaddle/Paddle/pull/62288),[#62323](https://github.com/PaddlePaddle/Paddle/pull/62323),[#62319](https://github.com/PaddlePaddle/Paddle/pull/62319),[#62331](https://github.com/PaddlePaddle/Paddle/pull/62331),[#62330](https://github.com/PaddlePaddle/Paddle/pull/62330),[#62329](https://github.com/PaddlePaddle/Paddle/pull/62329),[#62324](https://github.com/PaddlePaddle/Paddle/pull/62324),[#62317](https://github.com/PaddlePaddle/Paddle/pull/62317),[#62311](https://github.com/PaddlePaddle/Paddle/pull/62311),[#62310](https://github.com/PaddlePaddle/Paddle/pull/62310),[#62308](https://github.com/PaddlePaddle/Paddle/pull/62308),[#62289](https://github.com/PaddlePaddle/Paddle/pull/62289),[#62307](https://github.com/PaddlePaddle/Paddle/pull/62307),[#62315](https://github.com/PaddlePaddle/Paddle/pull/62315),[#62406](https://github.com/PaddlePaddle/Paddle/pull/62406),[#62458](https://github.com/PaddlePaddle/Paddle/pull/62458),[#62459](https://github.com/PaddlePaddle/Paddle/pull/62459),[#62481](https://github.com/PaddlePaddle/Paddle/pull/62481),[#62465](https://github.com/PaddlePaddle/Paddle/pull/62465),[#62462](https://github.com/PaddlePaddle/Paddle/pull/62462),[#62453](https://github.com/PaddlePaddle/Paddle/pull/62453),[#62496](https://github.com/PaddlePaddle/Paddle/pull/62496),[#62457](https://github.com/PaddlePaddle/Paddle/pull/62457),[#62537](https://github.com/PaddlePaddle/Paddle/pull/62537),[#62514](https://github.com/PaddlePaddle/Paddle/pull/62514),[#62548](https://github.com/PaddlePaddle/Paddle/pull/62548),[#62544](https://github.com/PaddlePaddle/Paddle/pull/62544),[#62575](https://github.com/PaddlePaddle/Paddle/pull/62575),[#62463](https://github.com/PaddlePaddle/Paddle/pull/62463),[#62643](https://github.com/PaddlePaddle/Paddle/pull/62643),[#62803](https://github.com/PaddlePaddle/Paddle/pull/62803),[#62924](https://github.com/PaddlePaddle/Paddle/pull/62924),[#63037](https://github.com/PaddlePaddle/Paddle/pull/63037),[#63102](https://github.com/PaddlePaddle/Paddle/pull/63102),[#63139](https://github.com/PaddlePaddle/Paddle/pull/63139),[#63092](https://github.com/PaddlePaddle/Paddle/pull/63092),[#63147](https://github.com/PaddlePaddle/Paddle/pull/63147),[#60518](https://github.com/PaddlePaddle/Paddle/pull/60518),[#60485](https://github.com/PaddlePaddle/Paddle/pull/60485),[#61273](https://github.com/PaddlePaddle/Paddle/pull/61273),[#63429](https://github.com/PaddlePaddle/Paddle/pull/63429),[#61954](https://github.com/PaddlePaddle/Paddle/pull/61954) -This release contains contributions from the project core team as well as: +## Others -Adam Osewski, Allen Guo, arlesniak, chenenquan, chenyanlann, fengkuangxiaxia, fuqianya, fwenguang, guguguzi, helen88, houj04, Jacek Czaja, jakpiase, jianghaicheng, joanna.wozna.intel, joeqiao12, Leo Chen, Leo Guo, Li-fAngyU, lidanqing, Liyulingyue, Matsumoto GAO, maxhuiy, Ming-Xu Huang, Nyakku Shigure, piotrekobi, piotrekobiIntel, QingshuChen, qipengh, Skr Bang, Sylwester Fraczek, Sławomir Siwek, taixiurong, tanzhipeng, Tomasz Socha, TTerror, Webbley, yaozhixin, ykkk2333, yujun, Zhangjingyu06, zhangxiaoci, zhangyikun02, zhangyk0314, zlsh80826, zn, Zuza. +Non-user related changes, including deprecated code cleanup, useless unit test cleanup, debugging or upgrade of monitoring mechanism. [#63377](https://github.com/PaddlePaddle/Paddle/pull/63377),[#64106](https://github.com/PaddlePaddle/Paddle/pull/64106),[#64220](https://github.com/PaddlePaddle/Paddle/pull/64220),[#64293](https://github.com/PaddlePaddle/Paddle/pull/64293),[#64464](https://github.com/PaddlePaddle/Paddle/pull/64464),[#64944](https://github.com/PaddlePaddle/Paddle/pull/64944),[#63638](https://github.com/PaddlePaddle/Paddle/pull/63638),[#63732](https://github.com/PaddlePaddle/Paddle/pull/63732),[#63735](https://github.com/PaddlePaddle/Paddle/pull/63735),[#63826](https://github.com/PaddlePaddle/Paddle/pull/63826),[#63982](https://github.com/PaddlePaddle/Paddle/pull/63982),[#63737](https://github.com/PaddlePaddle/Paddle/pull/63737),[#64471](https://github.com/PaddlePaddle/Paddle/pull/64471),[#64574](https://github.com/PaddlePaddle/Paddle/pull/64574),[#64494](https://github.com/PaddlePaddle/Paddle/pull/64494),[#62775](https://github.com/PaddlePaddle/Paddle/pull/62775),[#63601](https://github.com/PaddlePaddle/Paddle/pull/63601),[#62564](https://github.com/PaddlePaddle/Paddle/pull/62564),[#63772](https://github.com/PaddlePaddle/Paddle/pull/63772),[#64719](https://github.com/PaddlePaddle/Paddle/pull/64719),[#61640](https://github.com/PaddlePaddle/Paddle/pull/61640),[#63459](https://github.com/PaddlePaddle/Paddle/pull/63459),[#64062](https://github.com/PaddlePaddle/Paddle/pull/64062),[#63480](https://github.com/PaddlePaddle/Paddle/pull/63480),[#63833](https://github.com/PaddlePaddle/Paddle/pull/63833)[#63673](https://github.com/PaddlePaddle/Paddle/pull/63673),[#63672](https://github.com/PaddlePaddle/Paddle/pull/63672),[#64131](https://github.com/PaddlePaddle/Paddle/pull/64131),[#64156](https://github.com/PaddlePaddle/Paddle/pull/64156),[#64155](https://github.com/PaddlePaddle/Paddle/pull/64155),[#64159](https://github.com/PaddlePaddle/Paddle/pull/64159),[#63902](https://github.com/PaddlePaddle/Paddle/pull/63902),[#64230](https://github.com/PaddlePaddle/Paddle/pull/64230),[#64229](https://github.com/PaddlePaddle/Paddle/pull/64229),[#64236](https://github.com/PaddlePaddle/Paddle/pull/64236),[#64260](https://github.com/PaddlePaddle/Paddle/pull/64260),[#64175](https://github.com/PaddlePaddle/Paddle/pull/64175),[#64250](https://github.com/PaddlePaddle/Paddle/pull/64250),[#64269](https://github.com/PaddlePaddle/Paddle/pull/64269),[#64238](https://github.com/PaddlePaddle/Paddle/pull/64238),[#64349](https://github.com/PaddlePaddle/Paddle/pull/64349),[#64394](https://github.com/PaddlePaddle/Paddle/pull/64394),[#64402](https://github.com/PaddlePaddle/Paddle/pull/64402),[#64401](https://github.com/PaddlePaddle/Paddle/pull/64401),[#64388](https://github.com/PaddlePaddle/Paddle/pull/64388),[#64329](https://github.com/PaddlePaddle/Paddle/pull/64329),[#64502](https://github.com/PaddlePaddle/Paddle/pull/64502),[#64501](https://github.com/PaddlePaddle/Paddle/pull/64501),[#64515](https://github.com/PaddlePaddle/Paddle/pull/64515),[#64503](https://github.com/PaddlePaddle/Paddle/pull/64503),[#64514](https://github.com/PaddlePaddle/Paddle/pull/64514),[#64601](https://github.com/PaddlePaddle/Paddle/pull/64601),[#64564](https://github.com/PaddlePaddle/Paddle/pull/64564),[#64012](https://github.com/PaddlePaddle/Paddle/pull/64012),[#64697](https://github.com/PaddlePaddle/Paddle/pull/64697),[#64682](https://github.com/PaddlePaddle/Paddle/pull/64682),[#64051](https://github.com/PaddlePaddle/Paddle/pull/64051),[#63267](https://github.com/PaddlePaddle/Paddle/pull/63267),[#63426](https://github.com/PaddlePaddle/Paddle/pull/63426),[#63626](https://github.com/PaddlePaddle/Paddle/pull/63626),[#63257](https://github.com/PaddlePaddle/Paddle/pull/63257),[#63266](https://github.com/PaddlePaddle/Paddle/pull/63266),[#63468](https://github.com/PaddlePaddle/Paddle/pull/63468),[#63262](https://github.com/PaddlePaddle/Paddle/pull/63262),[#63248](https://github.com/PaddlePaddle/Paddle/pull/63248),[#63241](https://github.com/PaddlePaddle/Paddle/pull/63241),[#63252](https://github.com/PaddlePaddle/Paddle/pull/63252),[#63258](https://github.com/PaddlePaddle/Paddle/pull/63258),[#63235](https://github.com/PaddlePaddle/Paddle/pull/63235),[#63399](https://github.com/PaddlePaddle/Paddle/pull/63399),[#63488](https://github.com/PaddlePaddle/Paddle/pull/63488),[#63487](https://github.com/PaddlePaddle/Paddle/pull/63487),[#63466](https://github.com/PaddlePaddle/Paddle/pull/63466),[#63464](https://github.com/PaddlePaddle/Paddle/pull/63464),[#63483](https://github.com/PaddlePaddle/Paddle/pull/63483),[#63486](https://github.com/PaddlePaddle/Paddle/pull/63486),[#63475](https://github.com/PaddlePaddle/Paddle/pull/63475),[#63489](https://github.com/PaddlePaddle/Paddle/pull/63489),[#63470](https://github.com/PaddlePaddle/Paddle/pull/63470),[#63457](https://github.com/PaddlePaddle/Paddle/pull/63457),[#63493](https://github.com/PaddlePaddle/Paddle/pull/63493),[#63561](https://github.com/PaddlePaddle/Paddle/pull/63561),[#63584](https://github.com/PaddlePaddle/Paddle/pull/63584),[#63587](https://github.com/PaddlePaddle/Paddle/pull/63587),[#63586](https://github.com/PaddlePaddle/Paddle/pull/63586),[#63569](https://github.com/PaddlePaddle/Paddle/pull/63569),[#63559](https://github.com/PaddlePaddle/Paddle/pull/63559),[#63558](https://github.com/PaddlePaddle/Paddle/pull/63558),[#63555](https://github.com/PaddlePaddle/Paddle/pull/63555),[#63543](https://github.com/PaddlePaddle/Paddle/pull/63543),[#63589](https://github.com/PaddlePaddle/Paddle/pull/63589),[#63583](https://github.com/PaddlePaddle/Paddle/pull/63583),[#63565](https://github.com/PaddlePaddle/Paddle/pull/63565),[#63564](https://github.com/PaddlePaddle/Paddle/pull/63564),[#63265](https://github.com/PaddlePaddle/Paddle/pull/63265),[#63562](https://github.com/PaddlePaddle/Paddle/pull/63562),[#63591](https://github.com/PaddlePaddle/Paddle/pull/63591),[#63460](https://github.com/PaddlePaddle/Paddle/pull/63460),[#63238](https://github.com/PaddlePaddle/Paddle/pull/63238),[#63631](https://github.com/PaddlePaddle/Paddle/pull/63631),[#63707](https://github.com/PaddlePaddle/Paddle/pull/63707),[#63714](https://github.com/PaddlePaddle/Paddle/pull/63714),[#63854](https://github.com/PaddlePaddle/Paddle/pull/63854),[#63929](https://github.com/PaddlePaddle/Paddle/pull/63929),[#63532](https://github.com/PaddlePaddle/Paddle/pull/63532),[#59628](https://github.com/PaddlePaddle/Paddle/pull/59628),[#62209](https://github.com/PaddlePaddle/Paddle/pull/62209),[#63742](https://github.com/PaddlePaddle/Paddle/pull/63742),[#60518](https://github.com/PaddlePaddle/Paddle/pull/60518),[#62078](https://github.com/PaddlePaddle/Paddle/pull/62078),[#62684](https://github.com/PaddlePaddle/Paddle/pull/62684),[#62723](https://github.com/PaddlePaddle/Paddle/pull/62723),[#64141](https://github.com/PaddlePaddle/Paddle/pull/64141),[#60404](https://github.com/PaddlePaddle/Paddle/pull/60404),[#64212](https://github.com/PaddlePaddle/Paddle/pull/64212),[#60652](https://github.com/PaddlePaddle/Paddle/pull/60652),[#64545](https://github.com/PaddlePaddle/Paddle/pull/64545),[#64477](https://github.com/PaddlePaddle/Paddle/pull/64477),[#64556](https://github.com/PaddlePaddle/Paddle/pull/64556),[#63160](https://github.com/PaddlePaddle/Paddle/pull/63160),[#63796](https://github.com/PaddlePaddle/Paddle/pull/63796),[#64693](https://github.com/PaddlePaddle/Paddle/pull/64693),[#64484](https://github.com/PaddlePaddle/Paddle/pull/64484),[#64677](https://github.com/PaddlePaddle/Paddle/pull/64677),[#64461](https://github.com/PaddlePaddle/Paddle/pull/64461),[#63189](https://github.com/PaddlePaddle/Paddle/pull/63189),[#63855](https://github.com/PaddlePaddle/Paddle/pull/63855),[#63896](https://github.com/PaddlePaddle/Paddle/pull/63896),[#63193](https://github.com/PaddlePaddle/Paddle/pull/63193),[#63200](https://github.com/PaddlePaddle/Paddle/pull/63200),[#63406](https://github.com/PaddlePaddle/Paddle/pull/63406),[#61283](https://github.com/PaddlePaddle/Paddle/pull/61283),[#63607](https://github.com/PaddlePaddle/Paddle/pull/63607),[#64486](https://github.com/PaddlePaddle/Paddle/pull/64486),[#64004](https://github.com/PaddlePaddle/Paddle/pull/64004),[#63132](https://github.com/PaddlePaddle/Paddle/pull/63132),[#63553](https://github.com/PaddlePaddle/Paddle/pull/63553),[#63572](https://github.com/PaddlePaddle/Paddle/pull/63572),[#63794](https://github.com/PaddlePaddle/Paddle/pull/63794),[#63919](https://github.com/PaddlePaddle/Paddle/pull/63919),[#63980](https://github.com/PaddlePaddle/Paddle/pull/63980),[#62917](https://github.com/PaddlePaddle/Paddle/pull/62917),[#64451](https://github.com/PaddlePaddle/Paddle/pull/64451),[#63541](https://github.com/PaddlePaddle/Paddle/pull/63541),[#63703](https://github.com/PaddlePaddle/Paddle/pull/63703),[#64536](https://github.com/PaddlePaddle/Paddle/pull/64536),[#63264](https://github.com/PaddlePaddle/Paddle/pull/63264),[#63335](https://github.com/PaddlePaddle/Paddle/pull/63335),[#63841](https://github.com/PaddlePaddle/Paddle/pull/63841),[#64628](https://github.com/PaddlePaddle/Paddle/pull/64628),[#63419](https://github.com/PaddlePaddle/Paddle/pull/63419),[#62210](https://github.com/PaddlePaddle/Paddle/pull/62210),[#63557](https://github.com/PaddlePaddle/Paddle/pull/63557),[#63064](https://github.com/PaddlePaddle/Paddle/pull/63064),[#61442](https://github.com/PaddlePaddle/Paddle/pull/61442),[#63537](https://github.com/PaddlePaddle/Paddle/pull/63537),[#63839](https://github.com/PaddlePaddle/Paddle/pull/63839),[#60927](https://github.com/PaddlePaddle/Paddle/pull/60927),[#60566](https://github.com/PaddlePaddle/Paddle/pull/60566),[#60842](https://github.com/PaddlePaddle/Paddle/pull/60842),[#64612](https://github.com/PaddlePaddle/Paddle/pull/64612),[#60047](https://github.com/PaddlePaddle/Paddle/pull/60047),[#63898](https://github.com/PaddlePaddle/Paddle/pull/63898),[#60415](https://github.com/PaddlePaddle/Paddle/pull/60415),[#60474](https://github.com/PaddlePaddle/Paddle/pull/60474),[#60439](https://github.com/PaddlePaddle/Paddle/pull/60439),[#60565](https://github.com/PaddlePaddle/Paddle/pull/60565),[#64414](https://github.com/PaddlePaddle/Paddle/pull/64414),[#62526](https://github.com/PaddlePaddle/Paddle/pull/62526),[#54183](https://github.com/PaddlePaddle/Paddle/pull/54183),[#64096](https://github.com/PaddlePaddle/Paddle/pull/64096),[#61325](https://github.com/PaddlePaddle/Paddle/pull/61325),[#60629](https://github.com/PaddlePaddle/Paddle/pull/60629),[#61051](https://github.com/PaddlePaddle/Paddle/pull/61051),[#62103](https://github.com/PaddlePaddle/Paddle/pull/62103),[#63594](https://github.com/PaddlePaddle/Paddle/pull/63594),[#60968](https://github.com/PaddlePaddle/Paddle/pull/60968),[#64613](https://github.com/PaddlePaddle/Paddle/pull/64613),[#64073](https://github.com/PaddlePaddle/Paddle/pull/64073),[#63816](https://github.com/PaddlePaddle/Paddle/pull/63816),[#64416](https://github.com/PaddlePaddle/Paddle/pull/64416),[#62499](https://github.com/PaddlePaddle/Paddle/pull/62499),[#64531](https://github.com/PaddlePaddle/Paddle/pull/64531),[#63827](https://github.com/PaddlePaddle/Paddle/pull/63827),[#59885](https://github.com/PaddlePaddle/Paddle/pull/59885),[#59949](https://github.com/PaddlePaddle/Paddle/pull/59949),[#63428](https://github.com/PaddlePaddle/Paddle/pull/63428),[#63218](https://github.com/PaddlePaddle/Paddle/pull/63218),[#63538](https://github.com/PaddlePaddle/Paddle/pull/63538),[#64497](https://github.com/PaddlePaddle/Paddle/pull/64497),[#63082](https://github.com/PaddlePaddle/Paddle/pull/63082),[#64395](https://github.com/PaddlePaddle/Paddle/pull/64395),[#60183](https://github.com/PaddlePaddle/Paddle/pull/60183),[#63691](https://github.com/PaddlePaddle/Paddle/pull/63691),[#64428](https://github.com/PaddlePaddle/Paddle/pull/64428),[#64648](https://github.com/PaddlePaddle/Paddle/pull/64648),[#64650](https://github.com/PaddlePaddle/Paddle/pull/64650),[#59926](https://github.com/PaddlePaddle/Paddle/pull/59926),[#59750](https://github.com/PaddlePaddle/Paddle/pull/59750),[#60080](https://github.com/PaddlePaddle/Paddle/pull/60080),[#60208](https://github.com/PaddlePaddle/Paddle/pull/60208),[#64124](https://github.com/PaddlePaddle/Paddle/pull/64124),[#64187](https://github.com/PaddlePaddle/Paddle/pull/64187),[#64166](https://github.com/PaddlePaddle/Paddle/pull/64166),[#64284](https://github.com/PaddlePaddle/Paddle/pull/64284),[#64253](https://github.com/PaddlePaddle/Paddle/pull/64253),[#64555](https://github.com/PaddlePaddle/Paddle/pull/64555),[#59878](https://github.com/PaddlePaddle/Paddle/pull/59878),[#64081](https://github.com/PaddlePaddle/Paddle/pull/64081) From cdefa909db1eade3975386fe6750077c805111a9 Mon Sep 17 00:00:00 2001 From: PuQing Date: Fri, 12 Jul 2024 16:04:58 +0800 Subject: [PATCH 10/16] add weight and density options for histogram (#6760) --- docs/api/paddle/histogram_cn.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/api/paddle/histogram_cn.rst b/docs/api/paddle/histogram_cn.rst index 8d0bce79468..24a68a70796 100644 --- a/docs/api/paddle/histogram_cn.rst +++ b/docs/api/paddle/histogram_cn.rst @@ -3,7 +3,7 @@ histogram ------------------------------- -.. py:function:: paddle.histogram(input, bins=100, min=0, max=0, name=None) +.. py:function:: paddle.histogram(input, bins=100, min=0, max=0, weight=None, density=False, name=None) 计算输入 Tensor 的直方图。以 min 和 max 为 range 边界，将其均分成 bins 个直条，然后将排序好的数据划分到各个直条(bins)中。如果 min 和 max 都为 0，则利用数据中的最大最小值作为边界。 @@ -14,11 +14,13 @@ histogram - **bins** (int，可选) - 直方图 bins(直条)的个数，默认为 100。 - **min** (int，可选) - range 的下边界(包含)，默认为 0。 - **max** (int，可选) - range 的上边界(包含)，默认为 0。 + - **weight** (Tensor，可选) - 权重 Tensor，维度和 input 相同。如果提供，输入中的每个值都将以对应的权重值进行计数(而不是 1)。默认为 None。 + - **density** (bool，可选) - 如果为 False，则返回直方图中每个 bin 的计数。如果为 True，则返回直方图中每个 bin 经过归一化后的概率密度函数的值，使得直方图的积分(即所有 bin 的面积)等于 1。默认为 False。 - **name** (str，可选) - 具体用法请参见 :ref:`api_guide_Name`，一般无需设置，默认值为 None。 返回 :::::::::::: -Tensor，数据为 int64 类型，维度为(nbins,)。 +Tensor，维度为(nbins,)。如果 density 为 True 或者 weight 不为 None，则返回的 Tensor 的数据类型为 float32；否则为 int64。 代码示例 :::::::::::: From f1f6bfe51bc59a72c480e1c859a971d6f6ea75c2 Mon Sep 17 00:00:00 2001 From: "runxuan.zhang" <58112832+ZaraXzz@users.noreply.github.com> Date: Wed, 17 Jul 2024 10:08:15 +0800 Subject: [PATCH 11/16] Update AvgPool3D_cn.rst (#6737) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Update AvgPool3D_cn.rst 更改了 stride 描述上的问题 * Update AvgPool3D_cn.rst * Update docs/api/paddle/nn/AvgPool3D_cn.rst --------- Co-authored-by: ooo oo <106524776+ooooo-create@users.noreply.github.com> --- docs/api/paddle/nn/AvgPool3D_cn.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/api/paddle/nn/AvgPool3D_cn.rst b/docs/api/paddle/nn/AvgPool3D_cn.rst index f90f1d10b58..061485d9395 100644 --- a/docs/api/paddle/nn/AvgPool3D_cn.rst +++ b/docs/api/paddle/nn/AvgPool3D_cn.rst @@ -32,12 +32,12 @@ AvgPool3D 参数 ::::::::: - **kernel_size** (int|list|tuple)：池化核大小。如果它是一个元组或列表，它必须包含三个整数值，(pool_size_Depth, pool_size_Height, pool_size_Width)。若为一个整数，则表示 D，H 和 W 维度上均为该值，比如若 pool_size=2，则池化核大小为[2,2,2]。 - - **stride** (int|list|tuple，可选)：池化层的步长。如果它是一个元组或列表，它将包含两个整数，(pool_stride_Height, pool_stride_Width)。若为一个整数，则表示 H 和 W 维度上 stride 均为该值。默认值为 None，这时会使用 kernel_size 作为 stride。 + - **stride** (int|list|tuple，可选)：池化层的步长。如果它是一个元组或列表，它将包含三个整数，(pool_stride_Depth, pool_stride_Height, pool_stride_Width)。若为一个整数，则表示 D，H 和 W 维度上 stride 均为该值。默认值为 None，这时会使用 kernel_size 作为 stride。 - **padding** (str|int|list|tuple，可选) 池化填充。如果它是一个元组或列表，它可以有 3 种格式：(1)包含 3 个整数值：[pad_depth, pad_height, pad_width]；(2)包含 6 个整数值：[pad_depth_front, pad_depth_back, pad_height_top, pad_height_bottom, pad_width_left, pad_width_right]；(3)包含 5 个二元组：当 data_format 为"NCDHW"时为[[0,0], [0,0], [pad_depth_front, pad_depth_back], [pad_height_top, pad_height_bottom], [pad_width_left, pad_width_right]]，当 data_format 为"NDHWC"时为[[0,0], [pad_depth_front, pad_depth_back], [pad_height_top, pad_height_bottom], [pad_width_left, pad_width_right], [0,0]]。若为一个整数，则表示 D、H 和 W 维度上均为该值。默认值：0。 - **ceil_mode** (bool，可选)：是否用 ceil 函数计算输出高度和宽度。如果是 True，则使用 `ceil` 计算输出形状的大小。默认为 False - **exclusive** (bool，可选)：是否在平均池化模式忽略填充值，默认是 `True`。 - **divisor_override** (int|float，可选)：如果指定，它将用作除数，否则根据`kernel_size`计算除数。默认`None`。 - - **data_format** (str，可选)：输入和输出的数据格式，可以是"NCDHW"和"NDHWC"。N 是批尺寸，C 是通道数，D 是特征深度，H 是特征高度，W 是特征宽度。默认值："NDCHW"。 + - **data_format** (str，可选)：输入和输出的数据格式，可以是"NCDHW"和"NDHWC"。N 是批尺寸，C 是通道数，D 是特征深度，H 是特征高度，W 是特征宽度。默认值："NCDHW"。 - **name** (str，可选) - 具体用法请参见 :ref:`api_guide_Name`，一般无需设置，默认值为 None。 From 95048cdd7a449d4ff29c6af9708f42466a1046b1 Mon Sep 17 00:00:00 2001 From: RedContritio Date: Thu, 18 Jul 2024 11:23:00 +0800 Subject: [PATCH 12/16] Rename PyTorch-Paddle mapping type (#6762) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * rename mapping type "仅 paddle 参数更多" * rename mapping type "仅参数默认值不一致" * update torch.Tensor.scatter --- .../api_difference/Tensor/torch.BoolTensor.md | 2 +- .../api_difference/Tensor/torch.ByteTensor.md | 2 +- .../api_difference/Tensor/torch.DoubleTensor.md | 2 +- .../api_difference/Tensor/torch.FloatTensor.md | 2 +- .../api_difference/Tensor/torch.HalfTensor.md | 2 +- .../api_difference/Tensor/torch.IntTensor.md | 2 +- .../api_difference/Tensor/torch.LongTensor.md | 2 +- .../api_difference/Tensor/torch.ShortTensor.md | 2 +- .../api_difference/Tensor/torch.Tensor.argwhere.md | 2 +- .../api_difference/Tensor/torch.Tensor.corrcoef.md | 4 ++-- .../Tensor/torch.Tensor.count_nonzero.md | 2 +- .../api_difference/Tensor/torch.Tensor.cov.md | 4 ++-- .../api_difference/Tensor/torch.Tensor.cummax.md | 2 +- .../api_difference/Tensor/torch.Tensor.cummin.md | 2 +- .../Tensor/torch.Tensor.fill_diagonal_.md | 2 +- .../api_difference/Tensor/torch.Tensor.fliplr.md | 2 +- .../api_difference/Tensor/torch.Tensor.flipud.md | 2 +- .../api_difference/Tensor/torch.Tensor.item.md | 2 +- .../Tensor/torch.Tensor.logcumsumexp.md | 2 +- .../Tensor/torch.Tensor.logical_and.md | 2 +- .../Tensor/torch.Tensor.logical_not.md | 2 +- .../Tensor/torch.Tensor.logical_or.md | 2 +- .../api_difference/Tensor/torch.Tensor.matmul.md | 4 ++-- .../api_difference/Tensor/torch.Tensor.median.md | 2 +- .../Tensor/torch.Tensor.nanmedian.md | 2 +- .../api_difference/Tensor/torch.Tensor.pinverse.md | 2 +- .../api_difference/Tensor/torch.Tensor.ravel.md | 2 +- .../api_difference/Tensor/torch.Tensor.scatter.md | 2 +- .../Tensor/torch.Tensor.scatter_add.md | 2 +- .../Tensor/torch.Tensor.scatter_add_.md | 2 +- .../api_difference/Tensor/torch.Tensor.softmax.md | 2 +- .../api_difference/Tensor/torch.Tensor.take.md | 2 +- .../api_difference/Tensor/torch.Tensor.trace.md | 2 +- .../api_difference/Tensor/torch.Tensor.uniform_.md | 2 +- .../Tensor/torch.Tensor.unique_consecutive.md | 2 +- .../api_difference/cuda/torch.cuda.BoolTensor.md | 2 +- .../api_difference/cuda/torch.cuda.ByteTensor.md | 2 +- .../api_difference/cuda/torch.cuda.DoubleTensor.md | 2 +- .../api_difference/cuda/torch.cuda.FloatTensor.md | 2 +- .../api_difference/cuda/torch.cuda.HalfTensor.md | 2 +- .../api_difference/cuda/torch.cuda.IntTensor.md | 2 +- .../api_difference/cuda/torch.cuda.LongTensor.md | 2 +- .../api_difference/cuda/torch.cuda.ShortTensor.md | 2 +- .../cuda/torch.cuda.amp.GradScaler.md | 2 +- .../api_difference/cuda/torch.cuda.amp.autocast.md | 2 +- .../torch.nn.functional.conv_transpose1d.md | 2 +- .../torch.nn.functional.conv_transpose2d.md | 2 +- .../torch.nn.functional.conv_transpose3d.md | 2 +- .../torch.nn.functional.fractional_max_pool2d.md | 2 +- .../functional/torch.nn.functional.group_norm.md | 2 +- .../functional/torch.nn.functional.hardsigmoid.md | 4 ++-- .../functional/torch.nn.functional.pad.md | 2 +- .../torch.nn.functional.pixel_shuffle.md | 2 +- .../torch.nn.functional.pixel_unshuffle.md | 2 +- .../functional/torch.nn.functional.prelu.md | 2 +- .../functional/torch.nn.functional.rrelu_.md | 2 +- .../functional/torch.nn.functional.upsample.md | 2 +- .../torch.nn.functional.upsample_bilinear.md | 2 +- .../torch.nn.functional.upsample_nearest.md | 5 +++-- .../api_difference/linalg/torch.linalg.diagonal.md | 4 ++-- .../api_difference/linalg/torch.linalg.vander.md | 2 +- .../nn/torch.nn.AdaptiveAvgPool2d.md | 2 +- .../nn/torch.nn.AdaptiveAvgPool3d.md | 2 +- .../api_difference/nn/torch.nn.ChannelShuffle.md | 2 +- .../api_difference/nn/torch.nn.ConstantPad1d.md | 2 +- .../api_difference/nn/torch.nn.ConstantPad2d.md | 2 +- .../api_difference/nn/torch.nn.ConstantPad3d.md | 2 +- .../nn/torch.nn.LocalResponseNorm.md | 2 +- .../api_difference/nn/torch.nn.MaxUnpool1d.md | 2 +- .../api_difference/nn/torch.nn.MaxUnpool2d.md | 2 +- .../api_difference/nn/torch.nn.MaxUnpool3d.md | 2 +- .../api_difference/nn/torch.nn.Module.bfloat16.md | 2 +- .../api_difference/nn/torch.nn.Module.cpu.md | 2 +- .../api_difference/nn/torch.nn.Module.cuda.md | 2 +- .../api_difference/nn/torch.nn.Module.double.md | 2 +- .../api_difference/nn/torch.nn.Module.float.md | 2 +- .../api_difference/nn/torch.nn.Module.half.md | 2 +- .../api_difference/nn/torch.nn.Module.md | 2 +- .../nn/torch.nn.Module.state_dict.md | 2 +- .../nn/torch.nn.MultiLabelMarginLoss.md | 2 +- .../api_difference/nn/torch.nn.PixelShuffle.md | 2 +- .../api_difference/nn/torch.nn.PixelUnshuffle.md | 2 +- .../api_difference/nn/torch.nn.Softmax2d.md | 2 +- .../api_difference/nn/torch.nn.Upsample.md | 2 +- .../nn/torch.nn.UpsamplingBilinear2d.md | 2 +- .../nn/torch.nn.UpsamplingNearest2d.md | 2 +- .../api_difference/nn/torch.nn.ZeroPad2d.md | 2 +- .../api_difference/ops/torch.Tensor__upper.md | 2 +- .../api_difference/ops/torch.alpha_dropout.md | 4 ++-- .../api_difference/ops/torch.argwhere.md | 2 +- .../api_difference/ops/torch.count_nonzero.md | 2 +- .../api_difference/ops/torch.cov.md | 5 +++-- .../api_difference/ops/torch.fliplr.md | 2 +- .../api_difference/ops/torch.flipud.md | 2 +- .../api_difference/ops/torch.nn.Module.modules.md | 2 +- .../api_difference/ops/torch.rrelu.md | 4 ++-- .../api_difference/ops/torch.scatter_add.md | 2 +- .../api_difference/ops/torch.selu.md | 2 +- .../api_difference/ops/torch.unique_consecutive.md | 2 +- .../api_difference/optimizer/torch.optim.LBFGS.md | 2 +- .../optimizer/torch.optim.Optimizer.md | 2 +- .../others/torch.autograd.Function.backward.md | 2 +- ...utograd.profiler.profile.export_chrome_trace.md | 2 +- .../others/torch.cpu.amp.autocast.md | 2 +- .../api_difference/others/torch.nansum.md | 2 +- .../pytorch_api_mapping_format_cn.md | 14 +++++++------- .../utils/torch.utils.data.BatchSampler.md | 2 +- .../apply_reference_from_api_difference.py | 4 ++-- .../validate_mapping_in_api_difference.py | 8 ++++---- 109 files changed, 130 insertions(+), 128 deletions(-) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.BoolTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.BoolTensor.md index 12f1fcc09c4..cb55031dfb5 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.BoolTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.BoolTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.BoolTensor +## [ paddle 参数更多 ] torch.BoolTensor ### [torch.BoolTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ByteTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ByteTensor.md index e0920c87463..78dac6e7fde 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ByteTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ByteTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.ByteTensor +## [ paddle 参数更多 ] torch.ByteTensor ### [torch.ByteTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.DoubleTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.DoubleTensor.md index b9a3208d925..452c22e8f97 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.DoubleTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.DoubleTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.DoubleTensor +## [ paddle 参数更多 ] torch.DoubleTensor ### [torch.DoubleTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.FloatTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.FloatTensor.md index fd17a906ee4..9736a0cd25d 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.FloatTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.FloatTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.FloatTensor +## [ paddle 参数更多 ] torch.FloatTensor ### [torch.FloatTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.HalfTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.HalfTensor.md index 5c5c5dd697f..ee78e829f7c 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.HalfTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.HalfTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.HalfTensor +## [ paddle 参数更多 ] torch.HalfTensor ### [torch.HalfTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.IntTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.IntTensor.md index 7a04e0ae881..ff612639a18 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.IntTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.IntTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.IntTensor +## [ paddle 参数更多 ] torch.IntTensor ### [torch.IntTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.LongTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.LongTensor.md index 40a291b023e..77a43a451b0 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.LongTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.LongTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.LongTensor +## [ paddle 参数更多 ] torch.LongTensor ### [torch.LongTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ShortTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ShortTensor.md index ced1c8d2fa4..deb9cf2fa07 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ShortTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.ShortTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.ShortTensor +## [ paddle 参数更多 ] torch.ShortTensor ### [torch.ShortTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.argwhere.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.argwhere.md index 668dc9236df..4428bbd7eee 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.argwhere.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.argwhere.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.argwhere +## [ paddle 参数更多 ]torch.Tensor.argwhere ### [torch.Tensor.argwhere](https://pytorch.org/docs/stable/generated/torch.Tensor.argwhere.html#torch.Tensor.argwhere) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.corrcoef.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.corrcoef.md index 53165ee5aab..827d3823903 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.corrcoef.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.corrcoef.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.Tensor.corrcoef +## [ paddle 参数更多 ]torch.Tensor.corrcoef ### [torch.Tensor.corrcoef](https://pytorch.org/docs/stable/generated/torch.Tensor.corrcoef.html#torch.Tensor.corrcoef) @@ -12,7 +12,7 @@ torch.Tensor.corrcoef() paddle.Tensor.corrcoef(rowvar=True) ``` -仅 paddle 参数更多，具体如下： +其中 Paddle 相比 PyTorch 支持更多其他参数，具体如下： ### 参数映射 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.count_nonzero.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.count_nonzero.md index 24df4729f2c..1b1ea8e4a9f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.count_nonzero.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.count_nonzero.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.Tensor.count_nonzero +## [ paddle 参数更多 ]torch.Tensor.count_nonzero ### [torch.Tensor.count_nonzero](https://pytorch.org/docs/stable/generated/torch.Tensor.count_nonzero.html#torch.Tensor.count_nonzero) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cov.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cov.md index f2ded12b6e9..87ef33abdf5 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cov.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cov.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.cov +## [ paddle 参数更多 ]torch.Tensor.cov ### [torch.Tensor.cov](https://pytorch.org/docs/stable/generated/torch.Tensor.cov.html#torch.Tensor.cov) ```python @@ -15,7 +15,7 @@ paddle.Tensor.cov(rowvar=True, name=None) ``` -仅 paddle 参数更多，具体如下： +其中 Paddle 相比 PyTorch 支持更多其他参数，具体如下： ### 参数映射 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummax.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummax.md index cc6d2596d4e..5ce33be790b 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummax.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummax.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.cummax +## [ paddle 参数更多 ]torch.Tensor.cummax ### [torch.Tensor.cummax](https://pytorch.org/docs/stable/generated/torch.Tensor.cummax.html?highlight=cummax#torch.Tensor.cummax) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummin.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummin.md index c60aa4e25bb..49f88d3219e 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummin.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.cummin.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.cummin +## [ paddle 参数更多 ]torch.Tensor.cummin ### [torch.Tensor.cummin](https://pytorch.org/docs/stable/generated/torch.Tensor.cummin.html?highlight=cummin#torch.Tensor.cummin) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fill_diagonal_.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fill_diagonal_.md index b034172e01e..fb6c5804269 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fill_diagonal_.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fill_diagonal_.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.fill_diagonal_ +## [ paddle 参数更多 ]torch.Tensor.fill_diagonal_ ### [torch.Tensor.fill_diagonal_](https://pytorch.org/docs/stable/generated/torch.Tensor.fill_diagonal_.html?highlight=fill_diagonal_#torch.Tensor.fill_diagonal_) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fliplr.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fliplr.md index 5ad27794162..cee3de4ddc4 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fliplr.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.fliplr.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.fliplr +## [ paddle 参数更多 ]torch.Tensor.fliplr ### [torch.Tensor.fliplr](https://pytorch.org/docs/stable/generated/torch.Tensor.fliplr.html?highlight=fliplr#torch.Tensor.fliplr) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.flipud.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.flipud.md index ab48dca83dd..85c4c89bf7e 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.flipud.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.flipud.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.flipud +## [ paddle 参数更多 ]torch.Tensor.flipud ### [torch.Tensor.flipud](https://pytorch.org/docs/stable/generated/torch.Tensor.flipud.html?highlight=flipud#torch.Tensor.flipud) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.item.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.item.md index 27f021023d7..f7848485440 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.item.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.item.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.Tensor.item +## [ paddle 参数更多 ] torch.Tensor.item ### [torch.Tensor.item](https://pytorch.org/docs/stable/generated/torch.Tensor.item.html#torch-tensor-item) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logcumsumexp.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logcumsumexp.md index 289836ffa96..9d00da784e3 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logcumsumexp.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logcumsumexp.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.logcumsumexp +## [ paddle 参数更多 ]torch.Tensor.logcumsumexp ### [torch.Tensor.logcumsumexp](https://pytorch.org/docs/stable/generated/torch.Tensor.logcumsumexp.html?highlight=logcumsumexp#torch.Tensor.logcumsumexp) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_and.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_and.md index 3321ae885be..41890d8aadc 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_and.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_and.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.Tensor.logical_and +## [ paddle 参数更多 ] torch.Tensor.logical_and ### [torch.Tensor.logical_and](https://pytorch.org/docs/stable/generated/torch.Tensor.logical_and.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_not.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_not.md index 0ece44ec6ab..221367958ad 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_not.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_not.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.Tensor.logical_not +## [ paddle 参数更多 ] torch.Tensor.logical_not ### [torch.Tensor.logical_not](https://pytorch.org/docs/stable/generated/torch.Tensor.logical_not.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_or.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_or.md index 843ceddf65d..89a998f8cab 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_or.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.logical_or.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.Tensor.logical_or +## [ paddle 参数更多 ] torch.Tensor.logical_or ### [torch.Tensor.logical_or](https://pytorch.org/docs/stable/generated/torch.Tensor.logical_or.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matmul.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matmul.md index 2d72140a390..df5e7f2bd93 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matmul.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matmul.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.matmul +## [ paddle 参数更多 ]torch.Tensor.matmul ### [torch.Tensor.matmul](https://pytorch.org/docs/stable/generated/torch.Tensor.matmul.html) @@ -12,7 +12,7 @@ torch.Tensor.matmul(other) paddle.Tensor.matmul(y, transpose_x=False, transpose_y=False, name=None) ``` -其中 PyTorch 和 Paddle 功能一致，仅 Paddle 参数更多，具体如下： +其中 Paddle 相比 PyTorch 支持更多其他参数，具体如下： ### 参数映射 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.median.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.median.md index e1854dc94c3..58c762949e6 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.median.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.median.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.median +## [ paddle 参数更多 ]torch.Tensor.median ### [torch.Tensor.median](https://pytorch.org/docs/stable/generated/torch.Tensor.median.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nanmedian.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nanmedian.md index 3e069b2b7f1..55d06f28928 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nanmedian.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.nanmedian.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.nanmedian +## [ paddle 参数更多 ]torch.Tensor.nanmedian ### [torch.Tensor.nanmedian](https://pytorch.org/docs/stable/generated/torch.Tensor.nanmedian.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.pinverse.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.pinverse.md index 9d8adb06581..97ba4cc5b2d 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.pinverse.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.pinverse.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.pinverse +## [ paddle 参数更多 ]torch.Tensor.pinverse ### [torch.Tensor.pinverse](https://pytorch.org/docs/stable/generated/torch.Tensor.pinverse.html#torch.Tensor.pinverse) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ravel.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ravel.md index d2567859c92..0f0c840b2ed 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ravel.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.ravel.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.ravel +## [ paddle 参数更多 ]torch.Tensor.ravel ### [torch.Tensor.ravel](https://pytorch.org/docs/stable/generated/torch.Tensor.ravel.html#torch.Tensor.ravel) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter.md index f6d58e7065e..2a8b9d533d1 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.Tensor.scatter +## [ 输入参数类型不一致 ]torch.Tensor.scatter ### [torch.Tensor.scatter](https://pytorch.org/docs/stable/generated/torch.Tensor.scatter.html#torch.Tensor.scatter) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add.md index 506848c0731..783eaa36473 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.scatter_add +## [ paddle 参数更多 ]torch.Tensor.scatter_add ### [torch.Tensor.scatter_add](https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_add.html#torch.Tensor.scatter_add) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add_.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add_.md index a669e647559..2b4ebe0cf18 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add_.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter_add_.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.scatter_add_ +## [ paddle 参数更多 ]torch.Tensor.scatter_add_ ### [torch.Tensor.scatter_add_](https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_add_.html#torch.Tensor.scatter_add_) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.softmax.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.softmax.md index 10fd4786a62..1847cb49c46 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.softmax.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.softmax.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.Tensor.softmax +## [ paddle 参数更多 ]torch.Tensor.softmax ### [torch.Tensor.softmax](https://pytorch.org/docs/stable/generated/torch.Tensor.softmax.html?highlight=softmax#torch.Tensor.softmax) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.take.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.take.md index 3c4467b85b1..9afeea57013 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.take.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.take.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.Tensor.take +## [ paddle 参数更多 ] torch.Tensor.take ### [torch.Tensor.take](https://pytorch.org/docs/stable/generated/torch.Tensor.take.html#torch.Tensor.take) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.trace.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.trace.md index 7acab3b852f..bae669e0cf5 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.trace.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.trace.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.Tensor.trace +## [ paddle 参数更多 ]torch.Tensor.trace ### [torch.Tensor.trace](https://pytorch.org/docs/stable/generated/torch.Tensor.trace.html#torch-tensor-trace) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.uniform_.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.uniform_.md index b038d4c82ce..af2554d7b95 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.uniform_.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.uniform_.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.Tensor.uniform_ +## [ paddle 参数更多 ]torch.Tensor.uniform_ ### [torch.Tensor.uniform_](https://pytorch.org/docs/stable/generated/torch.Tensor.uniform_.html#torch-tensor-uniform) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unique_consecutive.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unique_consecutive.md index 4dc4cc08622..a5a38f25bd0 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unique_consecutive.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.unique_consecutive.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.Tensor.unique_consecutive +## [ paddle 参数更多 ]torch.Tensor.unique_consecutive ### [torch.Tensor.unique_consecutive](https://pytorch.org/docs/stable/generated/torch.Tensor.unique_consecutive.html#torch.Tensor.unique_consecutive) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.BoolTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.BoolTensor.md index 3a32fe59f2e..fbaf4e3ce96 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.BoolTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.BoolTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.cuda.BoolTensor +## [ paddle 参数更多 ] torch.cuda.BoolTensor ### [torch.cuda.BoolTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ByteTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ByteTensor.md index 45c8a2520ef..914eb37cbbf 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ByteTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ByteTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.cuda.ByteTensor +## [ paddle 参数更多 ] torch.cuda.ByteTensor ### [torch.cuda.ByteTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.DoubleTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.DoubleTensor.md index 0150cb2fe0f..00a72e2893e 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.DoubleTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.DoubleTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.cuda.DoubleTensor +## [ paddle 参数更多 ] torch.cuda.DoubleTensor ### [torch.cuda.DoubleTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.FloatTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.FloatTensor.md index ade385c8bbf..af9960b9bae 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.FloatTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.FloatTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.cuda.FloatTensor +## [ paddle 参数更多 ] torch.cuda.FloatTensor ### [torch.cuda.FloatTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.HalfTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.HalfTensor.md index c543b687f65..c47910178dd 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.HalfTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.HalfTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.cuda.HalfTensor +## [ paddle 参数更多 ] torch.cuda.HalfTensor ### [torch.cuda.HalfTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.IntTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.IntTensor.md index fee0cc9c408..cdb69b55670 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.IntTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.IntTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.cuda.IntTensor +## [ paddle 参数更多 ] torch.cuda.IntTensor ### [torch.cuda.IntTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.LongTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.LongTensor.md index 7f6c37d4621..5a8e3d1498f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.LongTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.LongTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.cuda.LongTensor +## [ paddle 参数更多 ] torch.cuda.LongTensor ### [torch.cuda.LongTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ShortTensor.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ShortTensor.md index 4bedd6423fc..022ce4bc3f7 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ShortTensor.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.ShortTensor.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.cuda.ShortTensor +## [ paddle 参数更多 ] torch.cuda.ShortTensor ### [torch.cuda.ShortTensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.GradScaler.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.GradScaler.md index 346f5a89bec..27ceb550ac7 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.GradScaler.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.GradScaler.md @@ -1,4 +1,4 @@ -## [仅参数默认值不一致]torch.cuda.amp.GradScaler +## [ 参数默认值不一致 ]torch.cuda.amp.GradScaler ### [torch.cuda.amp.GradScaler](https://pytorch.org/docs/stable/amp.html#torch.cuda.amp.GradScaler) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.autocast.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.autocast.md index c98c8019f45..89ef19f9579 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.autocast.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.autocast.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.cuda.amp.autocast +## [ paddle 参数更多 ]torch.cuda.amp.autocast ### [torch.cuda.amp.autocast](https://pytorch.org/docs/stable/amp.html#torch.cuda.amp.autocast) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose1d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose1d.md index 5cdc2efbbd0..089084ad300 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose1d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose1d.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多 ]torch.nn.functional.conv_transpose1d +## [ paddle 参数更多 ]torch.nn.functional.conv_transpose1d ### [torch.nn.functional.conv_transpose1d](https://pytorch.org/docs/stable/generated/torch.nn.functional.conv_transpose1d.html?highlight=conv_trans#torch.nn.functional.conv_transpose1d) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose2d.md index 00d54843f41..8e46e5c15dc 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose2d.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多 ]torch.nn.functional.conv_transpose2d +## [ paddle 参数更多 ]torch.nn.functional.conv_transpose2d ### [torch.nn.functional.conv_transpose2d](https://pytorch.org/docs/stable/generated/torch.nn.functional.conv_transpose2d.html?highlight=conv_#torch.nn.functional.conv_transpose2d) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose3d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose3d.md index 32db4e735e6..478769cf0d2 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose3d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.conv_transpose3d.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多 ]torch.nn.functional.conv_transpose3d +## [ paddle 参数更多 ]torch.nn.functional.conv_transpose3d ### [torch.nn.functional.conv_transpose3d](https://pytorch.org/docs/stable/generated/torch.nn.functional.conv_transpose3d.html?highlight=conv_#torch.nn.functional.conv_transpose3d) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool2d.md index 65973bb3bc5..e428cefb705 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool2d.md @@ -20,7 +20,7 @@ PyTorch 参数更多，具体如下： | ------------- | ------------ | ------------------------------------------------------ | | input | x | 表示输入的 Tensor 。仅参数名不一致。 | | kernel_size | kernel_size | 表示核大小。参数完全一致。 | -| output_size | output_size | 表示目标输出尺寸，PyTorch 为可选参数，Paddle 为必选参数，仅参数默认值不一致。PyTorch 的 output_size 与 output_ratio 输入二选一，如不输入 output_size，则必须输入 output_ratio，此时需要转写。转写方式与下文 output_ratio 一致。 | +| output_size | output_size | 表示目标输出尺寸，PyTorch 为可选参数，Paddle 为必选参数，与 PyTorch 默认值不同。PyTorch 的 output_size 与 output_ratio 输入二选一，如不输入 output_size，则必须输入 output_ratio，此时需要转写。转写方式与下文 output_ratio 一致。 | | output_ratio | - | 表示目标输出比例。Paddle 无此参数，需要转写。 | | return_indices | return_mask | 表示是否返回最大值索引。仅参数名不一致。 | | _random_samples | random_u | 表示随机数。PyTorch 以列表形式的 Tensor 方式传入，Paddle 以 float 的方式传入，如果 PyTorch 的多个随机数相同，需要转写，如果 PyTorch 的多个随机数不同，暂无转写方式。 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.group_norm.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.group_norm.md index 04e40cc61e3..af97877a09d 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.group_norm.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.group_norm.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.functional.group_norm +## [ paddle 参数更多 ]torch.nn.functional.group_norm ### [torch.nn.functional.group_norm](https://pytorch.org/docs/stable/generated/torch.nn.functional.group_norm.html#torch.nn.functional.group_norm) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardsigmoid.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardsigmoid.md index 88360ab6825..70ca9208d3f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardsigmoid.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardsigmoid.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.functional.hardsigmoid +## [ paddle 参数更多 ]torch.nn.functional.hardsigmoid ### [torch.nn.functional.hardsigmoid](https://pytorch.org/docs/stable/generated/torch.nn.functional.hardsigmoid.html) @@ -12,7 +12,7 @@ torch.nn.functional.hardsigmoid(input, inplace=False) paddle.nn.functional.hardsigmoid(x, slope=0.1666667, offset=0.5, name=None) ``` -其中 PyTorch 和 Paddle 功能一致，仅 paddle 参数更多，具体如下： +其中 Paddle 相比 PyTorch 支持更多其他参数，具体如下： ### 参数映射 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pad.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pad.md index e7ba14f81a5..e48c2a80844 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pad.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pad.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.functional.pad +## [ paddle 参数更多 ]torch.nn.functional.pad ### [torch.nn.functional.pad](https://pytorch.org/docs/stable/generated/torch.nn.functional.pad.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_shuffle.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_shuffle.md index 84d15682857..af43beb0fc8 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_shuffle.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_shuffle.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.functional.pixel_shuffle +## [ paddle 参数更多 ]torch.nn.functional.pixel_shuffle ### [torch.nn.functional.pixel_shuffle](https://pytorch.org/docs/stable/generated/torch.nn.functional.pixel_shuffle.html?highlight=pixel_shuffle#torch.nn.functional.pixel_shuffle) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_unshuffle.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_unshuffle.md index b85ba90f11f..7c565269cb1 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_unshuffle.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.pixel_unshuffle.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.nn.functional.pixel_unshuffle +## [ paddle 参数更多 ] torch.nn.functional.pixel_unshuffle ### [torch.nn.functional.pixel_unshuffle](https://pytorch.org/docs/stable/generated/torch.nn.functional.pixel_unshuffle.html?highlight=pixel_unshuffle#torch.nn.functional.pixel_unshuffle) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.prelu.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.prelu.md index fae6eebce60..088c191d48c 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.prelu.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.prelu.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.functional.prelu +## [ paddle 参数更多 ]torch.nn.functional.prelu ### [torch.nn.functional.prelu](https://pytorch.org/docs/stable/generated/torch.nn.functional.prelu.html?highlight=prelu#torch.nn.functional.prelu) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.rrelu_.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.rrelu_.md index 182d7554cec..132fef77394 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.rrelu_.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.rrelu_.md @@ -1,4 +1,4 @@ -## [ 仅参数默认值不一致 ]torch.nn.functional.rrelu_ +## [ 参数默认值不一致 ]torch.nn.functional.rrelu_ ### [torch.nn.functional.rrelu\_](https://pytorch.org/docs/stable/generated/torch.nn.functional.rrelu_.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample.md index baa35710d3e..05ea752e076 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.nn.functional.upsample +## [ paddle 参数更多 ] torch.nn.functional.upsample ### [torch.nn.functional.upsample](https://pytorch.org/docs/stable/generated/torch.nn.functional.upsample.html?highlight=upsample#torch.nn.functional.upsample) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_bilinear.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_bilinear.md index d95c317ef36..9adfab88df9 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_bilinear.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_bilinear.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.functional.upsample_bilinear +## [ paddle 参数更多 ]torch.nn.functional.upsample_bilinear ### [torch.nn.functional.upsample_bilinear](https://pytorch.org/docs/stable/generated/torch.nn.functional.upsample_bilinear.html#torch.nn.functional.upsample_bilinear) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_nearest.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_nearest.md index ecdb1e6ca81..4ff35092c73 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_nearest.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.upsample_nearest.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.functional.upsample_nearest +## [ paddle 参数更多 ]torch.nn.functional.upsample_nearest ### [torch.nn.functional.upsample_nearest](https://pytorch.org/docs/stable/generated/torch.nn.functional.upsample_nearest.html#torch.nn.functional.upsample_nearest) @@ -12,7 +12,8 @@ torch.nn.functional.upsample_nearest(input, size=None, scale_factor=None) paddle.nn.functional.upsample(x, size=None, scale_factor=None, mode='nearest', align_corners=False, align_mode=0, data_format='NCHW', name=None) ``` -仅 paddle 参数更多，具体区别如下： +其中 Paddle 相比 PyTorch 支持更多其他参数，具体如下： + ### 参数映射 | PyTorch | PaddlePaddle | 备注 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.diagonal.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.diagonal.md index 349fb450188..04087e3813d 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.diagonal.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.diagonal.md @@ -1,4 +1,4 @@ -## [ 仅参数默认值不一致 ]torch.linalg.diagonal +## [ 参数默认值不一致 ]torch.linalg.diagonal ### [torch.linalg.diagonal](https://pytorch.org/docs/stable/generated/torch.linalg.diagonal.html#torch.linalg.diagonal) ```python @@ -15,7 +15,7 @@ paddle.diagonal(x, name=None) ``` -两者功能一致且参数用法一致，仅参数默认值不一致，具体如下： +两者功能一致且参数用法一致，参数默认值不一致，具体如下： ### 参数映射 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.vander.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.vander.md index 18a9131c6d4..ab9376bcf1d 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.vander.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/linalg/torch.linalg.vander.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.linalg.vander +## [ paddle 参数更多 ]torch.linalg.vander ### [torch.linalg.vander](https://pytorch.org/docs/stable/generated/torch.linalg.vander.html#torch.linalg.vander) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool2d.md index b59d7637594..555ae4b5b27 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool2d.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.nn.AdaptiveAvgPool2d +## [ paddle 参数更多 ] torch.nn.AdaptiveAvgPool2d ### [torch.nn.AdaptiveAvgPool2d](https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool3d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool3d.md index f0c012e9ee5..cbcadb20575 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool3d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.AdaptiveAvgPool3d.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.nn.AdaptiveAvgPool3d +## [ paddle 参数更多 ] torch.nn.AdaptiveAvgPool3d ### [torch.nn.AdaptiveAvgPool3d](https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool3d.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ChannelShuffle.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ChannelShuffle.md index 288689366d6..ec2e2f4556f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ChannelShuffle.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ChannelShuffle.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.ChannelShuffle +## [ paddle 参数更多 ]torch.nn.ChannelShuffle ### [torch.nn.ChannelShuffle](https://pytorch.org/docs/stable/generated/torch.nn.ChannelShuffle.html?highlight=channelshuffle#torch.nn.ChannelShuffle) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad1d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad1d.md index 2bf6c4540e5..00da0cdf0f3 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad1d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad1d.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.ConstantPad1d +## [ paddle 参数更多 ]torch.nn.ConstantPad1d ### [torch.nn.ConstantPad1d](https://pytorch.org/docs/stable/generated/torch.nn.ConstantPad1d.html?highlight=constantpad1d#torch.nn.ConstantPad1d) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad2d.md index 38da3a0fc1f..8371c066f18 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad2d.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.ConstantPad2d +## [ paddle 参数更多 ]torch.nn.ConstantPad2d ### [torch.nn.ConstantPad2d](https://pytorch.org/docs/stable/generated/torch.nn.ConstantPad2d.html?highlight=pad#torch.nn.ConstantPad2d) ```python torch.nn.ConstantPad2d(padding, diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad3d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad3d.md index 1f529aff1d1..54d4577441f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad3d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ConstantPad3d.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.ConstantPad3d +## [ paddle 参数更多 ]torch.nn.ConstantPad3d ### [torch.nn.ConstantPad3d](https://pytorch.org/docs/stable/generated/torch.nn.ConstantPad3d.html?highlight=pad#torch.nn.ConstantPad3d) ```python torch.nn.ConstantPad3d(padding, diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LocalResponseNorm.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LocalResponseNorm.md index 0feff098e4f..9c42de58985 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LocalResponseNorm.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.LocalResponseNorm.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多]torch.nn.LocalResponseNorm +## [ paddle 参数更多 ]torch.nn.LocalResponseNorm ### [torch.nn.LocalResponseNorm](https://pytorch.org/docs/stable/generated/torch.nn.LocalResponseNorm.html?highlight=localre#torch.nn.LocalResponseNorm) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool1d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool1d.md index 202430242ab..d6c18761c72 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool1d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool1d.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多 ]torch.nn.MaxUnpool1d +## [ paddle 参数更多 ]torch.nn.MaxUnpool1d ### [torch.nn.MaxUnpool1d](https://pytorch.org/docs/stable/generated/torch.nn.MaxUnpool1d.html?highlight=maxunpool1d#torch.nn.MaxUnpool1d) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool2d.md index 64aea998ad7..b14a51d46d5 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool2d.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多 ]torch.nn.MaxUnpool2d +## [ paddle 参数更多 ]torch.nn.MaxUnpool2d ### [torch.nn.MaxUnpool2d](https://pytorch.org/docs/stable/generated/torch.nn.MaxUnpool2d.html?highlight=maxunpool2d#torch.nn.MaxUnpool2d) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool3d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool3d.md index 1f6f7f885bc..70221412403 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool3d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MaxUnpool3d.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多 ]torch.nn.MaxUnpool3d +## [ paddle 参数更多 ]torch.nn.MaxUnpool3d ### [torch.nn.MaxUnpool3d](https://pytorch.org/docs/stable/generated/torch.nn.MaxUnpool3d.html?highlight=maxunpool3d#torch.nn.MaxUnpool3d) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.bfloat16.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.bfloat16.md index 6fc1cd3f15d..b344b5970a7 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.bfloat16.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.bfloat16.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.Module.bfloat16 +## [ paddle 参数更多 ]torch.nn.Module.bfloat16 ### [torch.nn.Module.bfloat16](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.bfloat16) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cpu.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cpu.md index 6f4b44248a6..07f59c44ce6 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cpu.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cpu.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.Module.cpu +## [ paddle 参数更多 ]torch.nn.Module.cpu ### [torch.nn.Module.cpu](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.cpu) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cuda.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cuda.md index 2cab422ce32..df39239ae0b 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cuda.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.cuda.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.Module.cuda +## [ paddle 参数更多 ]torch.nn.Module.cuda ### [torch.nn.Module.cuda](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.cuda) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.double.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.double.md index c1f6fd31a69..39916d8c389 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.double.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.double.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.Module.double +## [ paddle 参数更多 ]torch.nn.Module.double ### [torch.nn.Module.double](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.double) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.float.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.float.md index e6f15c9294a..0a8f6554c96 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.float.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.float.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.Module.float +## [ paddle 参数更多 ]torch.nn.Module.float ### [torch.nn.Module.float](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.float) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.half.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.half.md index 9c41bf5850a..efc539445b7 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.half.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.half.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.Module.half +## [ paddle 参数更多 ]torch.nn.Module.half ### [torch.nn.Module.half](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.half) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.md index 5ce0920d4e5..1f29ab98a0d 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.nn.Module +## [ paddle 参数更多 ] torch.nn.Module ### [torch.nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=torch+nn+module#torch.nn.Module) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.state_dict.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.state_dict.md index 6b69384f70a..3e98f6620d1 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.state_dict.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Module.state_dict.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.nn.Module.state_dict +## [ paddle 参数更多 ] torch.nn.Module.state_dict ### [torch.nn.Module.state_dict](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.state_dict) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MultiLabelMarginLoss.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MultiLabelMarginLoss.md index 4565dd09208..9bbe39b8149 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MultiLabelMarginLoss.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.MultiLabelMarginLoss.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.nn.MultiLabelMarginLoss +## [ paddle 参数更多 ]torch.nn.MultiLabelMarginLoss ### [torch.nn.MultiLabelMarginLoss](https://pytorch.org/docs/stable/generated/torch.nn.MultiLabelMarginLoss) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelShuffle.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelShuffle.md index fea4614aeb7..e1c7c083a2a 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelShuffle.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelShuffle.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.PixelShuffle +## [ paddle 参数更多 ]torch.nn.PixelShuffle ### [torch.nn.PixelShuffle](https://pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html?highlight=pixel#torch.nn.PixelShuffle) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelUnshuffle.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelUnshuffle.md index 4ca18c43d81..a4de0dab54f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelUnshuffle.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.PixelUnshuffle.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.PixelUnshuffle +## [ paddle 参数更多 ]torch.nn.PixelUnshuffle ### [torch.nn.PixelUnshuffle](https://pytorch.org/docs/stable/generated/torch.nn.PixelUnshuffle.html?highlight=pixel#torch.nn.PixelUnshuffle) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Softmax2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Softmax2d.md index ef2be99fdab..2a34266a2d6 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Softmax2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Softmax2d.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.nn.Softmax2d +## [ paddle 参数更多 ]torch.nn.Softmax2d ### [torch.nn.Softmax2d](https://pytorch.org/docs/stable/generated/torch.nn.Softmax2d.html?highlight=softmax2d#torch.nn.Softmax2d) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Upsample.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Upsample.md index caf64dd6666..7764642cb2f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Upsample.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.Upsample.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.Upsample +## [ paddle 参数更多 ]torch.nn.Upsample ### [torch.nn.Upsample](https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html?highlight=upsample#torch.nn.Upsample) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingBilinear2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingBilinear2d.md index f2101d7e030..3b68bdef5a0 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingBilinear2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingBilinear2d.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.UpsamplingBilinear2d +## [ paddle 参数更多 ]torch.nn.UpsamplingBilinear2d ### [torch.nn.UpsamplingBilinear2d](https://pytorch.org/docs/stable/generated/torch.nn.UpsamplingBilinear2d.html?highlight=upsamplingbilinear2d#torch.nn.UpsamplingBilinear2d) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingNearest2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingNearest2d.md index 0d888f78631..ff059d5ad52 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingNearest2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.UpsamplingNearest2d.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.UpsamplingNearest2d +## [ paddle 参数更多 ]torch.nn.UpsamplingNearest2d ### [torch.nn.UpsamplingNearest2d](https://pytorch.org/docs/stable/generated/torch.nn.UpsamplingNearest2d.html?highlight=upsampl#torch.nn.UpsamplingNearest2d) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ZeroPad2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ZeroPad2d.md index e3a99280fc7..3073555aeb3 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ZeroPad2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/nn/torch.nn.ZeroPad2d.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.ZeroPad2d +## [ paddle 参数更多 ]torch.nn.ZeroPad2d ### [torch.nn.ZeroPad2d](https://pytorch.org/docs/stable/generated/torch.nn.ZeroPad2d.html?highlight=zeropad#torch.nn.ZeroPad2d) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.Tensor__upper.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.Tensor__upper.md index ff946db584c..95136c4c6a3 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.Tensor__upper.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.Tensor__upper.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.Tensor +## [ paddle 参数更多 ] torch.Tensor ### [torch.Tensor](https://pytorch.org/docs/stable/tensors.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.alpha_dropout.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.alpha_dropout.md index c47d03232d9..baf9adf826a 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.alpha_dropout.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.alpha_dropout.md @@ -1,4 +1,4 @@ -## [ 仅参数默认值不一致 ]torch.alpha_dropout +## [ 参数默认值不一致 ]torch.alpha_dropout ### [torch.alpha\_dropout](https://pytorch.org/docs/master/generated/torch.nn.functional.alpha_dropout.html) @@ -12,7 +12,7 @@ torch.alpha_dropout(input, p=0.5, train=False) paddle.nn.functional.alpha_dropout(x, p=0.5, training=True, name=None) ``` -其中 PyTorch 和 Paddle 功能一致，仅参数默认值不一致，具体如下： +其中 PyTorch 和 Paddle 功能一致，参数默认值不一致，具体如下： ### 参数映射 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.argwhere.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.argwhere.md index f172fdd5685..23a548a63c7 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.argwhere.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.argwhere.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.argwhere +## [ paddle 参数更多 ]torch.argwhere ### [torch.argwhere](https://pytorch.org/docs/stable/generated/torch.argwhere.html#torch.argwhere) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.count_nonzero.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.count_nonzero.md index c00b583ceb1..987bd763a01 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.count_nonzero.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.count_nonzero.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.count_nonzero +## [ paddle 参数更多 ]torch.count_nonzero ### [torch.count_nonzero](https://pytorch.org/docs/stable/generated/torch.count_nonzero.html?highlight=count_nonzero#torch.count_nonzero) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cov.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cov.md index 26ac533e75c..835ed0e36c2 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cov.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.cov.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.cov +## [ paddle 参数更多 ]torch.cov ### [torch.cov](https://pytorch.org/docs/stable/generated/torch.cov.html?highlight=cov#torch.cov) ```python @@ -19,7 +19,8 @@ paddle.linalg.cov(x, name=None) ``` -仅 paddle 参数更多，具体如下： +其中 Paddle 相比 PyTorch 支持更多其他参数，具体如下： + ### 参数映射 | PyTorch | PaddlePaddle | 备注 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.fliplr.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.fliplr.md index a910b63a3ac..1813bdfe08f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.fliplr.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.fliplr.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.fliplr +## [ paddle 参数更多 ]torch.fliplr ### [torch.fliplr](https://pytorch.org/docs/stable/generated/torch.fliplr.html?highlight=fliplr#torch.fliplr) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.flipud.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.flipud.md index c5661414817..e59f6f47e78 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.flipud.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.flipud.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.flipud +## [ paddle 参数更多 ]torch.flipud ### [torch.flipud](https://pytorch.org/docs/stable/generated/torch.flipud.html?highlight=flipud#torch.flipud) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nn.Module.modules.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nn.Module.modules.md index b030a127f3b..004d279f37c 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nn.Module.modules.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.nn.Module.modules.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.nn.Module.modules +## [ paddle 参数更多 ]torch.nn.Module.modules ### [torch.nn.Module.modules](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.modules) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rrelu.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rrelu.md index 9b02f7f6bd7..fd833b013b8 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rrelu.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.rrelu.md @@ -1,4 +1,4 @@ -## [ 仅参数默认值不一致 ]torch.rrelu +## [ 参数默认值不一致 ]torch.rrelu ### [torch.rrelu](https://pytorch.org/docs/stable/generated/torch.nn.functional.rrelu.html#torch.nn.functional.rrelu) @@ -12,7 +12,7 @@ torch.rrelu(input, lower=1./8, upper=1./3, training=False) paddle.nn.functional.rrelu(x, lower=1./8, upper=1./3, training=True, name=None) ``` -其中 PyTorch 和 Paddle 功能一致，仅参数名与参数默认值不一致，具体如下： +其中 PyTorch 和 Paddle 功能一致，参数名与参数默认值不一致，具体如下： ### 参数映射 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter_add.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter_add.md index 4886e0a667a..d1dbeda5a68 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter_add.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter_add.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.scatter_add +## [ paddle 参数更多 ]torch.scatter_add ### [torch.scatter_add](https://pytorch.org/docs/stable/generated/torch.scatter_add.html#torch.scatter_add) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.selu.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.selu.md index 1eb744e4a21..29c88915b20 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.selu.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.selu.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.selu +## [ paddle 参数更多 ]torch.selu ### [torch.selu](https://pytorch.org/docs/stable/generated/torch.nn.functional.selu.html#torch.nn.functional.selu) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.unique_consecutive.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.unique_consecutive.md index 5862eac61b8..784f410aabd 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.unique_consecutive.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.unique_consecutive.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.unique_consecutive +## [ paddle 参数更多 ]torch.unique_consecutive ### [torch.unique_consecutive](https://pytorch.org/docs/stable/generated/torch.unique_consecutive.html?highlight=unique_consecutive#torch.unique_consecutive) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.LBFGS.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.LBFGS.md index cd7175cdf41..a257ad8b971 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.LBFGS.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.LBFGS.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.optim.LBFGS +## [ paddle 参数更多 ]torch.optim.LBFGS ### [torch.optim.LBFGS](https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Optimizer.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Optimizer.md index 2bcfe468d0e..7784bddc8d3 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Optimizer.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/optimizer/torch.optim.Optimizer.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.optim.Optimizer +## [ paddle 参数更多 ]torch.optim.Optimizer ### [torch.optim.Optimizer](https://pytorch.org/docs/stable/optim.html#torch.optim.Optimizer) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.Function.backward.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.Function.backward.md index 9264f121659..ec74c85b8d1 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.Function.backward.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.Function.backward.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.autograd.Function.backward +## [ paddle 参数更多 ]torch.autograd.Function.backward ### [torch.autograd.Function.backward](https://pytorch.org/docs/stable/generated/torch.autograd.Function.backward.html#torch.autograd.Function.backward) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.profiler.profile.export_chrome_trace.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.profiler.profile.export_chrome_trace.md index cb3f89c27b5..d72e43c9667 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.profiler.profile.export_chrome_trace.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.autograd.profiler.profile.export_chrome_trace.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.autograd.profiler.profile.export_chrome_trace +## [ paddle 参数更多 ]torch.autograd.profiler.profile.export_chrome_trace ### [torch.autograd.profiler.profile.export_chrome_trace](https://pytorch.org/docs/stable/generated/torch.autograd.profiler.profile.export_chrome_trace.html#torch.autograd.profiler.profile.export_chrome_trace) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.cpu.amp.autocast.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.cpu.amp.autocast.md index 3cbae40e72a..31f1aa9b8c1 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.cpu.amp.autocast.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.cpu.amp.autocast.md @@ -1,4 +1,4 @@ -## [仅 paddle 参数更多]torch.cpu.amp.autocast +## [ paddle 参数更多 ]torch.cpu.amp.autocast ### [torch.cpu.amp.autocast](https://pytorch.org/docs/stable/amp.html?highlight=autocast#torch.cpu.amp.autocast) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.nansum.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.nansum.md index 4d7f615420c..0a7461f66f6 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.nansum.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.nansum.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ] torch.nansum +## [ paddle 参数更多 ] torch.nansum ### [torch.nansum](https://pytorch.org/docs/stable/generated/torch.nansum.html) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md index bd6f94d3fd4..41cd4f619ba 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md @@ -9,8 +9,8 @@ 为了文档整体的一致性，我们统一了 API 映射关系的分类名称，共分为 7 大类： > 注：第 1~3 类均为 API 层面一对一映射，根据参数层面的映射关系将其细分为三类。 -* 第 1 类又分为五种情况：`无参数`、`参数完全一致`、`仅参数名不一致`、`仅 paddle 参数更多`、`仅参数默认值不一致`。 -> 注：分类优先级依次递增，即如果同时 `参数名不一致` + `仅 paddle 参数更多`，则写成后者 `仅 paddle 参数更多` 。 +* 第 1 类又分为五种情况：`无参数`、`参数完全一致`、`仅参数名不一致`、`paddle 参数更多`、`参数默认值不一致`。 +> 注：分类优先级依次递增，即如果同时 `参数名不一致` + `paddle 参数更多`，则写成后者 `paddle 参数更多` 。 * 第 2 类为 `torch 参数更多`。如果 torch 和 paddle 都支持更多参数，统一写成`torch 参数更多`。 @@ -55,9 +55,9 @@ Paddle API 签名 * 如果 `仅参数名不一致`，无需转写示例，需要在备注栏里对该参数加一句 `仅参数名不一致`。 -* 如果 `仅 paddle 参数更多`，无需转写示例，需要在备注栏加一句 `PyTorch 无此参数，（Paddle 应如何设置此参数）` 。如果默认无影响，则写 `PyTorch 无此参数，Paddle 保持默认即可`。 +* 如果 `paddle 参数更多`，无需转写示例，需要在备注栏加一句 `PyTorch 无此参数，（Paddle 应如何设置此参数）` 。如果默认无影响，则写 `PyTorch 无此参数，Paddle 保持默认即可`。 -* 如果 `仅参数默认值不一致`，无需转写示例，需要在备注栏里加一句 `与 PyTorch 默认值不同，（Paddle 应如何设置此参数）` 。 +* 如果 `参数默认值不一致`，无需转写示例，需要在备注栏里加一句 `与 PyTorch 默认值不同，（Paddle 应如何设置此参数）` 。 * 如果 `torch 参数更多`，对每个 torch 多的参数都需要转写示例，需要在备注栏里加一句 `Paddle 无此参数，需要转写` ；如确实无法转写，需要在备注里写 `Paddle 无此参数，暂无转写方式` ；若可直接删除，则需要写 `Paddle 无此参数，一般对网络训练结果影响不大，可直接删除` 。 @@ -177,7 +177,7 @@ paddle.dist(x, ## 模板 4 -### [ 仅 paddle 参数更多 ] torch.ZeroPad2d +### [ paddle 参数更多 ] torch.ZeroPad2d ### [torch.nn.ZeroPad2d](https://pytorch.org/docs/stable/generated/torch.nn.ZeroPad2d.html?highlight=zeropad#torch.nn.ZeroPad2d)(仅作为示例) @@ -209,7 +209,7 @@ Paddle 相比 PyTorch 支持更多其他参数，具体如下： ## 模板 5 -## [ 仅参数默认值不一致 ] torch.linalg.svd +## [ 参数默认值不一致 ] torch.linalg.svd ### [torch.linalg.svd](https://pytorch.org/docs/stable/generated/torch.linalg.svd.html?highlight=svd#torch.linalg.svd) @@ -223,7 +223,7 @@ torch.linalg.svd(A, full_matrices=True) paddle.linalg.svd(x, full_matrices=False, name=None) ``` -两者功能一致且参数用法一致，仅参数默认值不一致，具体如下： +两者功能一致且参数用法一致，参数默认值不一致，具体如下： ### 参数映射 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.BatchSampler.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.BatchSampler.md index 5e83743bcc2..823f6f473b9 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.BatchSampler.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.BatchSampler.md @@ -1,4 +1,4 @@ -## [ 仅 paddle 参数更多 ]torch.utils.data.BatchSampler +## [ paddle 参数更多 ]torch.utils.data.BatchSampler ### [torch.utils.data.BatchSampler](https://pytorch.org/docs/stable/data.html?highlight=batchsampler#torch.utils.data.BatchSampler) ```python diff --git a/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py b/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py index 6c34eda10ef..d49a8011bed 100644 --- a/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py +++ b/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py @@ -19,8 +19,8 @@ def mapping_type_to_description(mapping_type): "无参数", "参数完全一致", "仅参数名不一致", - "仅 paddle 参数更多", - "仅参数默认值不一致", + "paddle 参数更多", + "参数默认值不一致", ] if mapping_type in mapping_type_1: diff --git a/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py b/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py index 16bc1308308..49315ff5841 100644 --- a/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py +++ b/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py @@ -17,8 +17,8 @@ "无参数", "参数完全一致", "仅参数名不一致", - "仅 paddle 参数更多", - "仅参数默认值不一致", + "paddle 参数更多", + "参数默认值不一致", # type 2 "torch 参数更多", # type 3 @@ -346,8 +346,8 @@ def get_meta_from_diff_file(filepath): "无参数", "参数完全一致", "仅参数名不一致", - "仅 paddle 参数更多", - "仅参数默认值不一致", + "paddle 参数更多", + "参数默认值不一致", # type 2 "torch 参数更多", # type 3 From 434de4ca147ceb54158559b1cd05d41d7063ab48 Mon Sep 17 00:00:00 2001 From: zhwesky2010 <1183042833@qq.com> Date: Thu, 18 Jul 2024 14:28:15 +0800 Subject: [PATCH 13/16] update pytorch_api_mapping_format (#6761) --- .../Tensor/torch.Tensor.matmul.md | 4 +- .../Tensor/torch.Tensor.scatter.md | 8 +- .../cuda/torch.cuda.amp.GradScaler.md | 4 +- .../torch.distributed.rpc.init_rpc.md | 2 +- ...rch.nn.functional.fractional_max_pool2d.md | 2 +- .../api_difference/ops/torch.scatter.md | 2 +- .../api_difference/ops/torch.svd.md | 2 +- .../others/torch.multiprocessing.spawn.md | 2 +- .../pytorch_api_mapping_format_cn.md | 321 +++++++++--------- .../torch.utils.data.DistributedSampler.md | 2 +- 10 files changed, 168 insertions(+), 181 deletions(-) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matmul.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matmul.md index df5e7f2bd93..e3d73b71998 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matmul.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.matmul.md @@ -19,5 +19,5 @@ paddle.Tensor.matmul(y, transpose_x=False, transpose_y=False, name=None) | PyTorch | PaddlePaddle | 备注 | | ------- | ------------ | ---- | | other | y | 输入变量，仅参数名不一致。 | -| - | transpose_x | 相乘前是否转置 x，仅 Paddle 参数更多，保持默认即可。 | -| - | transpose_y | 相乘前是否转置 y，仅 Paddle 参数更多，保持默认即可。 | +| - | transpose_x | 相乘前是否转置 x，PyTorch 无此参数，Paddle 保持默认即可。 | +| - | transpose_y | 相乘前是否转置 y，PyTorch 无此参数，Paddle 保持默认即可。 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter.md index 2a8b9d533d1..39bd6b7744c 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/Tensor/torch.Tensor.scatter.md @@ -1,4 +1,4 @@ -## [ 输入参数类型不一致 ]torch.Tensor.scatter +## [ paddle 参数更多 ]torch.Tensor.scatter ### [torch.Tensor.scatter](https://pytorch.org/docs/stable/generated/torch.Tensor.scatter.html#torch.Tensor.scatter) @@ -21,6 +21,6 @@ paddle.Tensor.put_along_axis(indices, values, axis, reduce="assign", include_sel | dim | axis | 表示在哪一个维度 scatter ，仅参数名不一致。 | | index | indices | 表示输入的索引张量，仅参数名不一致。 | | src | values | 表示需要插入的值，仅参数名不一致。 | -| reduce | reduce | 归约操作类型 。 | -| - | include_self | 表示插入 values 时是否包含 arr 中的元素，PyTorch 无此参数。| -| - | broadcast | 表示是否需要广播 indices 矩阵，PyTorch 无此参数。 | +| reduce | reduce | 归约操作类型 。 | +| - | include_self | 表示插入 values 时是否包含 arr 中的元素，PyTorch 无此参数，Paddle 保持默认即可。| +| - | broadcast | 表示是否需要广播 indices 矩阵，PyTorch 无此参数，Paddle 保持默认即可。| diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.GradScaler.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.GradScaler.md index 27ceb550ac7..57909353d1e 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.GradScaler.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/cuda/torch.cuda.amp.GradScaler.md @@ -18,10 +18,10 @@ paddle.amp.GradScaler(enable=True, init_loss_scaling=65536.0, incr_ratio=2.0, de | PyTorch | PaddlePaddle | 备注 | | --------------- | ------------------------ |-----------------------------------------------------------------------------| -| init_scale | init_loss_scaling | 初始 loss scaling 因子。与 PyTorch 默认值不同, Paddle 需设为 65536.0。 | +| init_scale | init_loss_scaling | 初始 loss scaling 因子。Paddle 与 PyTorch 默认值不同，Paddle 应设置为 65536.0。 | | growth_factor | incr_ratio | 增大 loss scaling 时使用的乘数。 | | backoff_factor | decr_ratio | 减小 loss scaling 时使用的小于 1 的乘数。 | -| growth_interval | incr_every_n_steps | 连续 n 个 steps 的梯度都是有限值时，增加 loss scaling。与 PyTorch 默认值不同, Paddle 需设为 2000。 | +| growth_interval | incr_every_n_steps | 连续 n 个 steps 的梯度都是有限值时，增加 loss scaling。Paddle 与 PyTorch 默认值不同，Paddle 应设置为 2000。| | enabled | enable | 是否使用 loss scaling。 | | - | decr_every_n_nan_or_inf | 累计出现 n 个 steps 的梯度为 nan 或者 inf 时，减小 loss scaling，PyTorch 无此参数，Paddle 保持默认即可。 | | - | use_dynamic_loss_scaling | 是否使用动态的 loss scaling，PyTorch 无此参数，Paddle 保持默认即可。 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.rpc.init_rpc.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.rpc.init_rpc.md index cdb723a39ed..67615e86135 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.rpc.init_rpc.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.rpc.init_rpc.md @@ -20,7 +20,7 @@ PyTorch 相比 Paddle 支持更多其他参数，具体如下： | ------------------- | ------------------- | -- | | name | name | worker 名字。 | | backend | - | RPC 后台实现类型，Paddle 无此参数，暂无转写方式。 | -| rank | rank | worker 的 ID，仅参数默认值不同。 | +| rank | rank | worker 的 ID，Paddle 与 PyTorch 默认值不同，Paddle 应设置为 -1。 | | world_size | world_size | workers 的数量。 | | rpc_backend_options | - | 传递给 worker 创建时的配置项，Paddle 无此参数，暂无转写方式。 | | - | master_endpoint | master 的 IP 地址，PyTorch 无此参数，Paddle 保持默认即可。 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool2d.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool2d.md index e428cefb705..ac8cb44ea6f 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool2d.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.fractional_max_pool2d.md @@ -20,7 +20,7 @@ PyTorch 参数更多，具体如下： | ------------- | ------------ | ------------------------------------------------------ | | input | x | 表示输入的 Tensor 。仅参数名不一致。 | | kernel_size | kernel_size | 表示核大小。参数完全一致。 | -| output_size | output_size | 表示目标输出尺寸，PyTorch 为可选参数，Paddle 为必选参数，与 PyTorch 默认值不同。PyTorch 的 output_size 与 output_ratio 输入二选一，如不输入 output_size，则必须输入 output_ratio，此时需要转写。转写方式与下文 output_ratio 一致。 | +| output_size | output_size | 表示目标输出尺寸，PyTorch 为可选参数，Paddle 为必选参数，Paddle 与 PyTorch 默认值不同。PyTorch 的 output_size 与 output_ratio 输入二选一，如不输入 output_size，则必须输入 output_ratio，此时需要转写。转写方式与下文 output_ratio 一致。 | | output_ratio | - | 表示目标输出比例。Paddle 无此参数，需要转写。 | | return_indices | return_mask | 表示是否返回最大值索引。仅参数名不一致。 | | _random_samples | random_u | 表示随机数。PyTorch 以列表形式的 Tensor 方式传入，Paddle 以 float 的方式传入，如果 PyTorch 的多个随机数相同，需要转写，如果 PyTorch 的多个随机数不同，暂无转写方式。 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter.md index 91ca1c61dd3..1d20eb53219 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.scatter.md @@ -1,4 +1,4 @@ -## [torch 参数更多]torch.scatter +## [ torch 参数更多 ]torch.scatter ### [torch.scatter](https://pytorch.org/docs/2.0/generated/torch.scatter.html?highlight=torch+scatter#torch.scatter) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.svd.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.svd.md index 2ad83a1803e..48e09bb5606 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.svd.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.svd.md @@ -19,7 +19,7 @@ PyTorch 相比 Paddle 支持更多其他参数，具体如下： | PyTorch | PaddlePaddle | 备注 | | ------------- | ------------ | ------------------------------------------------------ | | input | x | 输入 Tensor ，仅参数名不一致。 | -| some | full_matrics | 表示需计算的奇异值数目。 与 PyTorch 默认值不同，需要转写。 | +| some | full_matrics | 表示需计算的奇异值数目。 Paddle 与 PyTorch 默认值不同，需要转写。 | | compute_uv | - | 表示是否计算 U 和 V 。Paddle 无此参数，暂无转写方式。 | | out | - | 表示输出的 Tensor 元组。 Paddle 无此参数，需要转写。 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.multiprocessing.spawn.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.multiprocessing.spawn.md index 70d40ba595f..282c010c500 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.multiprocessing.spawn.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.multiprocessing.spawn.md @@ -20,7 +20,7 @@ PyTorch 相比 Paddle 支持更多其他参数，具体如下： | ------------ | ------------ | ------------------------------------------------------------------- | | fn | func | Python 函数。 | | args | args | 函数 func 的输入参数。 | -| nprocs | nprocs | 启动进程的数目。 与 PyTorch 默认值不同， Paddle 应设置为 `1`。 | +| nprocs | nprocs | 启动进程的数目。Paddle 与 PyTorch 默认值不同，Paddle 应设置为 `1`。 | | join | join | 对所有启动的进程执行阻塞的 join，等待进程执行结束。 | | start_method | - | 启动方式。 PyTorch 已弃用，Paddle 无此参数，对网络训练结果无影响，可直接删除。 | | - | options | 其他初始化并行执行环境的配置选项。 PyTorch 无此参数， Paddle 保持默认即可。 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md index 41cd4f619ba..acad5135840 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md @@ -2,34 +2,30 @@ 请严格根据此格式规范来新增《API 映射关系》，不符合规范的文档将不予合入，具体如下: -# API 映射关系 - 格式规范 +# API 映射关系文档 - 规范 ### [分类名称] api 全称 -为了文档整体的一致性，我们统一了 API 映射关系的分类名称，共分为 7 大类： -> 注：第 1~3 类均为 API 层面一对一映射，根据参数层面的映射关系将其细分为三类。 - -* 第 1 类又分为五种情况：`无参数`、`参数完全一致`、`仅参数名不一致`、`paddle 参数更多`、`参数默认值不一致`。 -> 注：分类优先级依次递增，即如果同时 `参数名不一致` + `paddle 参数更多`，则写成后者 `paddle 参数更多` 。 - -* 第 2 类为 `torch 参数更多`。如果 torch 和 paddle 都支持更多参数，统一写成`torch 参数更多`。 - -* 第 3 类又分为三种情况：`输入参数类型不一致`、`输入参数用法不一致`、`返回参数类型不一致`。 -> 注意：这里的**不一致**都是从 torch 的角度来看，如果 paddle 可涵盖 torch，即 torch 是 paddle 的功能子集，则认定为一致（例如：torch 参数仅支持 list，paddle 参数支持 list/tuple），反之才认定为不一致。 - -* 第 4 类为 `组合替代实现` ，表示该 API 没有可直接对应的 API，需要通过多个 API 组合实现。 - -* 第 5 类为 `涉及上下文修改` ，表示涉及到上下文的分析，需要修改其他位置的代码。 -> 举例：所有的 `torch.optim.lr_scheduler.*`、`torch.nn.init.*`、`torch.nn.utils.clip*` 都为该类。此类 API 由于两者在设计上具有较大的差异，需要对上下文进行分析，涉及到上下文代码的联动修改。 - -* 第 6 类为 `可删除` 。表示可直接删除该 API，则无需写差异分析文档，仅标注即可 - -* 第 7 类为 `功能缺失` ，表示 Paddle 当前无对应 API，则无需写差异分析文档，仅标注即可。 +由于 API 映射关系的复杂性，为了保证文档格式的规范性，我们将所有 API 映射关系分为 12 类，并制定了统一的 **分类名称**： +1. 无参数 +2. 参数完全一致 +3. 仅参数名不一致 +4. paddle 参数更多 +5. 参数默认值不一致 +6. torch 参数更多 +7. 输入参数用法不一致 +8. 输入参数类型不一致 +9. 返回参数类型不一致 +10. 组合替代实现 +11. 可删除 +12. 功能缺失 > 注意： -> 1. 分类优先级依次递增，即如果同时 `第 2 类：torch 参数更多` 与 `第 3 类：参数不一致` ，则写成后者 `第 3 类：参数不一致` 。 -> 2. 所有的 Paddle API 无需关注 `name` 参数，直接忽略即可。 -> 3. 将类成员 API 映射为非类成员 API，则无需对比第一个参数。例如将 `torch.Tensor.outer(vec2)` 映射为 `paddle.outer(x, y)`，则忽略 paddle 的第一个参数，从 torch 的 `vec2` 和 paddle 的 `y` 开始对比。 +> 1. 分类的优先级依次递增，例如：如果同时 `仅参数名不一致` + `paddle 参数更多`，则分类为后者 `paddle 参数更多` ，如果同时 `paddle 参数更多` + `torch 参数更多`，则分类为后者 `torch 参数更多`。 +> 2. `输入参数用法不一致`、`输入参数类型不一致`、`返回参数类型不一致` 其中的**不一致**都是从 torch 的角度来看，只要 torch 能被 paddle 全覆盖，则将其视作一致（例如：torch 参数仅支持 list 用法，而 paddle 参数支持 list/tuple 用法，则视作用法一致），如果 torch 无法被 paddle 全覆盖，才认定为 **不一致**。 +> 3. `可删除` 表示转写时可直接删除该 API，并不会对代码运行结果有影响，无需写映射文档，仅标注即可。`功能缺失` 表示 Paddle 当前无对应 API 功能，则无需写映射文档，仅标注即可。 +> 4. 所有的 Paddle API 无需关注 `name` 参数，直接忽略即可。 +> 5. 将类成员 API 映射为非类成员 API，则无需对比第一个参数。例如将 `torch.Tensor.outer(vec2)` 映射为 `paddle.outer(x, y)`，则忽略 paddle 的第一个参数 `x` ，直接从 torch 的 `vec2` 和 paddle 的 `y` 开始对比参数。 ### [pytorch api 全称] (pytorch api 链接) @@ -43,33 +39,33 @@ PyTorch API 签名 Paddle API 签名 ``` -**一句话总结**。整体概述总结两个 API 的差异。例如 `第 3 类：参数不一致` ，需要简述下有哪些不一致的地方。在描写参数时，需要用 \` ` 来加深其底色。 +**一句话总结**。整体概述总结两个 API 的差异。例如 `输入参数用法不一致` ，需要简述下有参数哪些用法不一致的地方。在描写参数时，需要用 \` ` 来加深其底色。 ### 参数映射 -参数映射表的左边是`PyTorch` 对应参数，右边是`Paddle`对应参数，表格参数顺序按 `PyTorch` 参数顺序来。 +参数映射以表格的形式呈现，表格的第 1 列是`PyTorch` 所有参数，第 2 列是`Paddle`对应参数，表格顺序按第 1 列 `PyTorch` 的参数顺序来。 -* 如果 `无参数`，则没有参数映射这一栏。 +1. **无参数**：无需参数映射与转写示例。 -* 如果 `参数完全一致`，无需转写示例。 +2. **参数完全一致**：无需转写示例。 -* 如果 `仅参数名不一致`，无需转写示例，需要在备注栏里对该参数加一句 `仅参数名不一致`。 +3. **仅参数名不一致**：无需转写示例，但需要在备注列里注明哪些参数 `仅参数名不一致`。 -* 如果 `paddle 参数更多`，无需转写示例，需要在备注栏加一句 `PyTorch 无此参数，（Paddle 应如何设置此参数）` 。如果默认无影响，则写 `PyTorch 无此参数，Paddle 保持默认即可`。 +4. **paddle 参数更多**：无需转写示例，但需要在备注列里注明 `PyTorch 无此参数，[Paddle 应如何设置此参数]`。如果无需特殊设置，则写 `PyTorch 无此参数，Paddle 保持默认即可`。 -* 如果 `参数默认值不一致`，无需转写示例，需要在备注栏里加一句 `与 PyTorch 默认值不同，（Paddle 应如何设置此参数）` 。 +5. **参数默认值不一致**：无需转写示例，但需要在备注列里注明 `Paddle 与 PyTorch 默认值不同，[Paddle 应如何设置此参数，设置为多少]`。 -* 如果 `torch 参数更多`，对每个 torch 多的参数都需要转写示例，需要在备注栏里加一句 `Paddle 无此参数，需要转写` ；如确实无法转写，需要在备注里写 `Paddle 无此参数，暂无转写方式` ；若可直接删除，则需要写 `Paddle 无此参数，一般对网络训练结果影响不大，可直接删除` 。 +6. **torch 参数更多**：对每个 torch 多的参数都需要**转写示例**，同时需要在备注列里注明 `Paddle 无此参数，需要转写` ；如确实无法转写，则注明 `Paddle 无此参数，暂无转写方式` ；若可直接删除，则需要写 `Paddle 无此参数，一般对网络训练结果影响不大，可直接删除` 。 -* 如果 `参数不一致`，对每个不一致的参数都需要转写示例，需要在备注栏里写 `（说明不一致的用法），需要转写`；如确实无法转写，需要在备注里写 `（说明不一致的用法），暂无转写方式`。 +7. **输入参数用法不一致、输入参数类型不一致、返回参数类型不一致**：对每个不一致的参数都需要**转写示例**，同时需要在备注列里注明 `[不一致的用法说明]，需要转写`；如确实无法转写，需要在备注里写 `[不一致的用法说明]，暂无转写方式`。 -* 每个备注都需要`以句号结尾`。 +8. 每个备注都需要`以句号结尾`。 ### 转写示例 -**除第 1 类 API 映射关系较为简单，无需写转写示例，其他类 API 都需要写转写示例，否则需说明：Paddle 暂无转写方式。** +第 1-5 类不需要转写示例，第 6-10 类需要写转写示例，第 11-12 类无需写本映射文档。 -转写示例需要写得精简和一目了然。一般情形下只需写两行代码，无需打印各种结果，并且要保证转写前后的输出结果是一致的。另外需要先描述下待写的是该 torch api 的哪个参数及其功能。 +转写示例需要写得精简和一目了然。一般情形下只需写两行代码，无需打印各种结果，需要保证转写前后的输出结果是一致的。另外需要先描述下转写的是 torch api 的哪个参数及其功能。 #### 参数名 1：参数功能 1 ```python @@ -89,11 +85,12 @@ torch.xxx() paddle.xxx() ``` --------------- +-------------------------------------------------------- -# API 映射关系 - 模板 +# API 映射关系文档 - 模板 -## 模板 1 + +## 分类 1：无参数 ### [ 无参数 ] torch.Tensor.t @@ -111,7 +108,8 @@ paddle.Tensor.t() 两者功能一致，无参数。 -## 模板 2 + +## 分类 2：参数完全一致 ### [ 参数完全一致 ] torch.Tensor.clip @@ -121,14 +119,13 @@ paddle.Tensor.t() torch.Tensor.clip(min=None, max=None) ``` -### [paddle.Tensor.clip](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/Tensor_cn.html#clip-min-none-max-none-name-none) +### [paddle.Tensor.clip](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/Tensor_cn.html#clip-min-none-max-none-name-none) ```python paddle.Tensor.clip(min=None, max=None, name=None) ``` 两者功能一致，参数完全一致，具体如下： - ### 参数映射 | PyTorch | PaddlePaddle | 备注 | @@ -137,11 +134,12 @@ paddle.Tensor.clip(min=None, max=None, name=None) | max | max | 裁剪的最大值，输入中大于该值的元素将由该元素代替。 | -## 模板 3 -### [ 仅参数名不一致 ] torch.dist +## 分类 3：仅参数名不一致 + +### [ 仅参数名不一致 ]torch.dist -### [torch.dist](https://pytorch.org/docs/stable/generated/torch.dist.html?highlight=dist#torch.dist)(仅作为示例) +### [torch.dist](https://pytorch.org/docs/stable/generated/torch.dist.html?highlight=dist#torch.dist) ```python torch.dist(input, @@ -149,7 +147,7 @@ torch.dist(input, p=2) ``` -### [paddle.dist](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/dist_cn.html#dist)(仅作为示例) +### [paddle.dist](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/dist_cn.html#dist) ```python paddle.dist(x, @@ -158,34 +156,26 @@ paddle.dist(x, ``` 两者功能一致且参数用法一致，仅参数名不一致，具体如下： - ### 参数映射 -| PyTorch | Paddle | 备注 | -| --------- | ----------- | ------------------------------------------------------------ | -| input | x | 表示输入的 Tensor ，仅参数名不一致。 | -| other | y | 表示输入的 Tensor ，仅参数名不一致。 | -| dim | axis | 表示进行运算的轴，仅参数名不一致。 | -| dtype | dtype | 表示数据类型。 | -| size | shape | 表示输出形状大小。 | -| n | num_rows | 生成 2-D Tensor 的行数，仅参数名不一致。 | -| m | num_columns | 生成 2-D Tensor 的列数， 仅参数名不一致。 | -| start_dim | start_axis | 表示 flatten 展开的起始维度。 | -| end_dim | stop_axis | 表示 flatten 展开的结束维度。 | -| ndarray | data | 表示需要转换的数据， PyTorch 只能传入 numpy.ndarray ， Paddle 可以传入 scalar 、 list 、 tuple 、 numpy.ndarray 、 paddle.Tensor 。 | +| PyTorch | PaddlePaddle | 备注 | +| ------------- | ------------ | ------------------------------------------------------ | +| input | x | 表示输入的 Tensor ，仅参数名不一致。 | +| other | y | 表示输入的 Tensor ，仅参数名不一致。 | +| p | p | 表示需要计算的范数 | -## 模板 4 -### [ paddle 参数更多 ] torch.ZeroPad2d +## 分类 4：paddle 参数更多 -### [torch.nn.ZeroPad2d](https://pytorch.org/docs/stable/generated/torch.nn.ZeroPad2d.html?highlight=zeropad#torch.nn.ZeroPad2d)(仅作为示例) +### [ paddle 参数更多 ]torch.nn.ZeroPad2d +### [torch.nn.ZeroPad2d](https://pytorch.org/docs/stable/generated/torch.nn.ZeroPad2d.html?highlight=zeropad#torch.nn.ZeroPad2d) ```python torch.nn.ZeroPad2d(padding) ``` -### [paddle.nn.ZeroPad2D](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/nn/ZeroPad2D_cn.html)(仅作为示例) +### [paddle.nn.ZeroPad2D](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/nn/ZeroPad2D_cn.html) ```python paddle.nn.ZeroPad2D(padding, @@ -193,34 +183,33 @@ paddle.nn.ZeroPad2D(padding, name=None) ``` -Paddle 相比 PyTorch 支持更多其他参数，具体如下： - +其中 Paddle 相比 PyTorch 支持更多其他参数，具体如下： ### 参数映射 -| PyTorch | Paddle | 备注 | -| ------- | ------------- | ------------------------------------------------------------ | -| - | axis | 指定进行运算的轴， PyTorch 无此参数， Paddle 保持默认即可。 | -| - | keepdim | 是否在输出 Tensor 中保留减小的维度， PyTorch 无此参数， Paddle 保持默认即可。 | -| - | dtype | 输出 Tensor 的数据类型， PyTorch 无此参数， Paddle 保持默认即可。 | -| - | dtype | 表示数据类型， PyTorch 无此参数， Paddle 保持默认即可。 | -| - | place | 表示 Tensor 存放位置， PyTorch 无此参数， Paddle 需设置为 paddle.CPUPlace()。 | -| - | stop_gradient | 表示是否阻断梯度传导， PyTorch 无此参数， Paddle 保持默认即可。 | +| PyTorch | PaddlePaddle | 备注 | +| ------------- | ------------ | ------------------------------------------------------ | +| padding | padding | 表示填充大小。 | +| - | data_format | 指定输入的 format， PyTorch 无此参数， Paddle 保持默认即可。 | -## 模板 5 -## [ 参数默认值不一致 ] torch.linalg.svd +## 分类 5：参数默认值不一致 -### [torch.linalg.svd](https://pytorch.org/docs/stable/generated/torch.linalg.svd.html?highlight=svd#torch.linalg.svd) +### [ 参数默认值不一致 ]torch.linalg.diagonal +### [torch.linalg.diagonal](https://pytorch.org/docs/stable/generated/torch.linalg.diagonal.html#torch.linalg.diagonal) ```python -torch.linalg.svd(A, full_matrices=True) +torch.linalg.diagonal(A, *, offset=0, dim1=-2, dim2=-1) ``` -### [paddle.linalg.svd](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/linalg/svd_cn.html) +### [paddle.diagonal](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/diagonal_cn.html#diagonal) ```python -paddle.linalg.svd(x, full_matrices=False, name=None) +paddle.diagonal(x, + offset=0, + axis1=0, + axis2=1, + name=None) ``` 两者功能一致且参数用法一致，参数默认值不一致，具体如下： @@ -229,15 +218,18 @@ paddle.linalg.svd(x, full_matrices=False, name=None) | PyTorch | PaddlePaddle | 备注 | | ------------- | ------------ | ------------------------------------------------------ | -| A | x | 输入 Tensor，仅参数名不一致。 | -| full_matrices | full_matrices | 是否计算完整的 U 和 V 矩阵，PyTorch 为 True，Paddle 为 False，Paddle 需设置为与 PyTorch 一致。 | +| A | x | 表示输入的 Tensor ，仅参数名不一致。 | +| offset | offset | 表示对角线偏移量。 | +| dim1 | axis1 | 获取对角线的二维平面的第一维，Paddle 与 PyTorch 默认值不同，Paddle 应设置为`-2`。| +| dim2 | axis2 | 获取对角线的二维平面的第二维，Paddle 与 PyTorch 默认值不同，Paddle 应设置为`-1`。| -## 模板 6 + +## 分类 6：torch 参数更多 ### [ torch 参数更多 ] torch.abs -### [torch.abs](https://pytorch.org/docs/stable/generated/torch.abs.html?highlight=abs#torch.abs)(仅作为示例) +### [torch.abs](https://pytorch.org/docs/stable/generated/torch.abs.html?highlight=abs#torch.abs) ```python torch.abs(input, @@ -245,14 +237,14 @@ torch.abs(input, out=None) ``` -### [paddle.abs](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/abs_cn.html#abs)(仅作为示例) +### [paddle.abs](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/abs_cn.html#abs) ```python paddle.abs(x, name=None) ``` -PyTorch 相比 Paddle 支持更多其他参数，具体如下： +PyTorch 相比 Paddle 支持更多其他参数，具体如下：（注：这里额外列举了一些其他常见 Pytorch 的参数的转写方式，与 torch.abs 无关） ### 参数映射 @@ -273,15 +265,6 @@ PyTorch 相比 Paddle 支持更多其他参数，具体如下： | antialias | - | 是否使用 anti-aliasing，Paddle 无此参数，暂无转写方式。 | ### 转写示例 -#### size：输出形状大小 -```python -# PyTorch 写法 -torch.empty(3, 5) - -# Paddle 写法 -paddle.empty([3, 5]) -``` - #### out：指定输出 ```python # PyTorch 写法 @@ -336,11 +319,11 @@ else: ``` -## 模板 7 +## 分类 7：输入参数用法不一致 -### [ 输入参数用法不一致 ] torch.transpose +## [ 输入参数用法不一致 ] torch.transpose -### [torch.transpose](https://pytorch.org/docs/stable/generated/torch.transpose.html?highlight=transpose#torch.transpose)(仅作为示例) +### [torch.transpose](https://pytorch.org/docs/stable/generated/torch.transpose.html?highlight=transpose#torch.transpose) ```python torch.transpose(input, @@ -348,7 +331,7 @@ torch.transpose(input, dim1) ``` -### [paddle.transpose](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/transpose_cn.html#transpose)(仅作为示例) +### [paddle.transpose](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/transpose_cn.html#transpose) ```python paddle.transpose(x, @@ -356,120 +339,124 @@ paddle.transpose(x, name=None) ``` -PyTorch 的 `tensors` 参数与 Paddle 的 `inputs` 参数用法不同，具体如下： +其中 PyTorch 的 `dim0、dim1` 与 Paddle 用法不一致，具体如下： ### 参数映射 -| PyTorch | Paddle | 备注 | -| ------- | ------------- | ------------------------------------------------------------ | -|*tensors | inputs | 一组输入 Tensor，PyTorch 的 tensors 为可变参数，Paddle 的 inputs 为 list(Tensor) 或 tuple(Tensor) 用法，需要转写。 | -| 返回值 | 返回值 | 返回参数类型不一致, PyTorch 返回 torch.ByteTensor，Paddle 返回 GeneratorState 对象，暂无转写方式。 | +| PyTorch | PaddlePaddle | 备注 | +| ------------- | ------------ | ------------------------------------------------------ | +| input | x | 输入 Tensor。 | +| dim0 | - | PyTorch 转置的第一个维度，Paddle 无此参数，需要转写。 | +| dim1 | - | PyTorch 转置的第二个维度，Paddle 无此参数，需要转写。 | +| - | perm | Paddle 可通过 perm 参数，等价的实现 torch 的 dim0、dim1 的功能。| ### 转写示例 -#### *tensors: 一组输入 Tensor，可变参数用法 -```python -# PyTorch 写法 -torch.broadcast_tensors(x, y) -# Paddle 写法 -paddle.broadcast_tensors([x, y]) -``` +#### dim0、dim1 参数： 转置的维度设置 +``` python +# PyTorch 写法: +torch.transpose(x, dim0=0, dim1=1) -#### affine：是否进行反射变换 +# Paddle 写法: +paddle.transpose(x, perm=[1, 0, 2]) -```python -affine=False 时，表示不更新： +# 注：x 为 3D Tensor +``` -# PyTorch 写法 -m = torch.nn.BatchNorm1D(24, affine=False) -# Paddle 写法 -weight_attr = paddle.ParamAttr(learning_rate=0.0) -bias_attr = paddle.ParamAttr(learning_rate=0.0) -m = paddle.nn.BatchNorm1D(24, weight_attr=weight_attr, bias_attr=bias_attr) +## 分类 8：输入参数类型不一致 -affine=True 时，表示更新： +### [ 输入参数类型不一致 ]torch.broadcast_tensors -# PyTorch 写法 -m = torch.nn.BatchNorm1D(24) +### [torch.broadcast_tensors](https://pytorch.org/docs/stable/generated/torch.broadcast_tensors.html?highlight=broadcast_tensors#torch.broadcast_tensors) -# Paddle 写法 -m = paddle.nn.BatchNorm1D(24) +```python +torch.broadcast_tensors(*tensors) ``` -## 模板 8 - -### [ 组合替代实现 ] torch.addcmul - -### [torch.addcmul](https://pytorch.org/docs/master/generated/torch.addcmul.html#torch.addcmul) +### [paddle.broadcast_tensors](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/broadcast_tensors_cn.html#broadcast-tensors) ```python -torch.addcmul(input, tensor1, tensor2, *, value=1, out=None) +paddle.broadcast_tensors(inputs, + name=None) ``` -用于实现矩阵 `tensor1` 与矩阵 `tensor2` 相乘，再加上输入 `input` ，公式为： +两者功能一致但参数类型不一致，具体如下： +### 参数映射 -$ out = input + value * tensor1 * tensor2 $ +| PyTorch | PaddlePaddle | 备注 | +| ------------- | ------------ | ------------------------------------------------------ | +| *tensors | inputs | 一组输入 Tensor ， PyTorch 参数 tensors 为可变参, Paddle 参数 inputs 为 list(Tensor) 或 tuple(Tensor) 的形式。 | -PaddlePaddle 目前无对应 API，可使用如下代码组合替代实现: ### 转写示例 - +#### *tensors: 一组输入 Tensor ```python # PyTorch 写法 -torch.addcmul(input, tensor1, tensor2, value=value) +torch.broadcast_tensors(x, y) # Paddle 写法 -paddle.add(input, value * tensor1 * tensor2) +paddle.broadcast_tensors([x, y]) ``` -## 模板 9 -### [ 涉及上下文修改 ] torch.nn.utils.clip_grad_value_ +## 分类 9：返回参数类型不一致 -### [torch.nn.utils.clip_grad_value_](https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_value_.html?highlight=clip_grad_value_#torch.nn.utils.clip_grad_value_)(仅作为示例) +### [ 返回参数类型不一致 ]torch.equal +### [torch.equal](https://pytorch.org/docs/stable/generated/torch.equal.html?highlight=equal#torch.equal) ```python -torch.nn.utils.clip_grad_value_(parameters, - clip_value) +torch.equal(input, + other) ``` -### [paddle.nn.ClipGradByValue](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/nn/ClipGradByValue_cn.html#clipgradbyvalue)(仅作为示例) +### [paddle.equal_all](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/equal_all_cn.html#equal-all) ```python -paddle.nn.ClipGradByValue(max, - min=None) +paddle.equal_all(x, + y, + name=None) ``` -其中 PyTorch 与 Paddle 对该 API 的设计思路与⽤法不同，需要分析上下⽂并联动修改： - +两者功能一致但返回参数类型不同，具体如下： ### 参数映射 -| PyTorch | PaddlePaddle | 备注 | -| ------- | ------------ | ---- | -| parameters | - | 表示要操作的 Tensor， PyTorch 属于原位操作， PaddlePaddle ⽆此参数，需要实例化之后在 optimizer 中设置才可以使⽤。需要上下⽂分析与联动修改。| -| clip_value | max | 表示裁剪梯度的范围，范围为 [-clip_value, clip_vale] ； PaddlePaddle 的 max 参数可实现该参数功能，直接设置为与 clip_value 一致。| -| - | min | 表示裁剪梯度的最⼩值， PyTorch ⽆此参数， Paddle 保持默认即可。 | +| PyTorch | PaddlePaddle | 备注 | +| ------------- | ------------ | ------------------------------------------------------ | +| input | x | 表示输入的 Tensor，仅参数名不一致。 | +| other | y | 表示输入的 Tensor，仅参数名不一致。 | +| 返回值 | 返回值 | PyTorch 返回 bool 类型，Paddle 返回 0-D bool Tensor，需要转写。| ### 转写示例 +#### 返回值 +``` python +# PyTorch 写法 +out = torch.equal(x, y) + +# Paddle 写法 +out = paddle.equal_all(x, y) +out = out.item() +``` + + +## 分类 10：组合替代实现 + +### [ 组合替代实现 ]torch.aminmax + +### [torch.aminmax](https://pytorch.org/docs/stable/generated/torch.aminmax.html#torch.aminmax) ```python -# torch ⽤法 -net = Model() -sgd = torch.optim.SGD(net.parameters(), lr=0.1) -for i in range(10): - loss = net(x) - loss.backward() - torch.nn.utils.clip_grad_value_(net.parameters(), 1.) - sgd.step() - -# paddle ⽤法 -net = Model() -sgd = paddle.optim.SGD(net.parameters(), lr=0.1, -grad_clip=paddle.nn.ClipGradByValue(), 1.) -for i in range(10): - loss = net(x) - loss.backward() - sgd.step() +torch.aminmax(input, *, dim=None, keepdim=False, out=None) +``` +Paddle 无此 API，需要组合实现。 + +### 转写示例 + +```python +# PyTorch 写法 +y = torch.aminmax(input, dim=-1, keepdim=True) + +# Paddle 写法 +y = tuple([paddle.amin(input, axis=-1, keepdim=True), paddle.amax(input, axis=-1, keepdim=True)]) ``` diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.DistributedSampler.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.DistributedSampler.md index a24b4b54254..76a3c4451af 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.DistributedSampler.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/utils/torch.utils.data.DistributedSampler.md @@ -29,6 +29,6 @@ PyTorch 参数更多，具体如下： | - | batch_size | 每 mini-batch 中包含的样本数，PyTorch 无此参数，Paddle 需设置为 1。 | | num_replicas | num_replicas | 分布式训练时的进程个数。 | | rank | rank | num_replicas 个进程中的进程序号。 | -| shuffle | shuffle | 是否需要在生成样本下标时打乱顺序。与 PyTorch 默认值不同， Paddle 应设置为 `True`。 | +| shuffle | shuffle | 是否需要在生成样本下标时打乱顺序。Paddle 与 PyTorch 默认值不同，Paddle 应设置为 `True`。 | | seed | - | 如果 shuffle=True，则使用随机种子对采样器进行随机排序,此数字在分布式组中的所有进程中应相同，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。 | | drop_last | drop_last | 是否需要丢弃最后无法凑整一个 mini-batch 的样本。 | From 8c9894cb714e57f3f7144f123fa29fe9fada95f7 Mon Sep 17 00:00:00 2001 From: zhwesky2010 <1183042833@qq.com> Date: Thu, 18 Jul 2024 17:20:56 +0800 Subject: [PATCH 14/16] update pytorch_api mapping format (#6767) --- .../pytorch_api_mapping_format_cn.md | 23 ++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md index acad5135840..835c2f81d7c 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/pytorch_api_mapping_format_cn.md @@ -59,11 +59,11 @@ Paddle API 签名 7. **输入参数用法不一致、输入参数类型不一致、返回参数类型不一致**：对每个不一致的参数都需要**转写示例**，同时需要在备注列里注明 `[不一致的用法说明]，需要转写`；如确实无法转写，需要在备注里写 `[不一致的用法说明]，暂无转写方式`。 -8. 每个备注都需要`以句号结尾`。 +8. **组合替代实现**：无需参数映射，需要写 **转写示例** 。 -### 转写示例 +9. 每个备注都需要`以句号结尾`。 -第 1-5 类不需要转写示例，第 6-10 类需要写转写示例，第 11-12 类无需写本映射文档。 +### 转写示例 转写示例需要写得精简和一目了然。一般情形下只需写两行代码，无需打印各种结果，需要保证转写前后的输出结果是一致的。另外需要先描述下转写的是 torch api 的哪个参数及其功能。 @@ -85,6 +85,23 @@ torch.xxx() paddle.xxx() ``` +是否需要 **参数映射** 与 **转写示例** 可查阅表格： + +第 1、10 类不需要参数映射，其他类均需要写参数映射。第 1-5 类不需要转写示例，第 6-10 类需要写转写示例。第 11-12 类则无需新建文档，仅标注即可。 + +| 分类序号 | 分类名称 | 参数映射 | 转写示例 | +| ------- | ----------------- | --------------- | --------------- | +| 1 | 无参数 | ❌ | ❌ | +| 2 | 参数完全一致 | ✅ | ❌ | +| 3 | 仅参数名不一致 | ✅ | ❌ | +| 4 | paddle 参数更多 | ✅ | ❌ | +| 5 | 参数默认值不一致 | ✅ | ❌ | +| 6 | torch 参数更多 | ✅ | ✅ | +| 7 | 输入参数用法不一致 | ✅ | ✅ | +| 8 | 输入参数类型不一致 | ✅ | ✅ | +| 9 | 返回参数类型不一致 | ✅ | ✅ | +| 10 | 组合替代实现 | ❌ | ✅ | + -------------------------------------------------------- # API 映射关系文档 - 模板 From 56781ca1374f4cd76967c386cf198d383507014f Mon Sep 17 00:00:00 2001 From: xuxinyi389 <104957571+xuxinyi389@users.noreply.github.com> Date: Thu, 18 Jul 2024 21:20:12 +0800 Subject: [PATCH 15/16] LLama convert docs (#6754) * add_docs * fix * fix_all * fix_typo * fix * fix * fix * fix_typo * fix_typo * remove * fix_typo * move --- .../torch.distributed.is_initialized.md | 24 ++++++++++ .../ops/torch.set_default_tensor_type.md | 8 ++-- .../others/torch.inference_mode.md | 44 +++++++++++++++++++ ...llel.initialize.get_model_parallel_rank.md | 15 +++++++ ...nitialize.get_model_parallel_world_size.md | 14 ++++++ ...el.initialize.initialize_model_parallel.md | 29 ++++++++++++ ...nitialize.model_parallel_is_initialized.md | 19 ++++++++ ...el_parallel.layers.ColumnParallelLinear.md | 30 +++++++++++++ ...model_parallel.layers.ParallelEmbedding.md | 30 +++++++++++++ ...model_parallel.layers.RowParallelLinear.md | 30 +++++++++++++ .../apply_reference_from_api_difference.py | 41 +++++++++++++++++ .../pytorch_api_mapping_cn.md | 20 +++++++-- .../validate_mapping_in_api_difference.py | 10 +++++ 13 files changed, 307 insertions(+), 7 deletions(-) create mode 100644 docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.is_initialized.md create mode 100644 docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.inference_mode.md create mode 100644 docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_rank.md create mode 100644 docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_world_size.md create mode 100644 docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.initialize_model_parallel.md create mode 100644 docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.model_parallel_is_initialized.md create mode 100644 docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ColumnParallelLinear.md create mode 100644 docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ParallelEmbedding.md create mode 100644 docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.RowParallelLinear.md diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.is_initialized.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.is_initialized.md new file mode 100644 index 00000000000..2a69704e9c2 --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/distributed/torch.distributed.is_initialized.md @@ -0,0 +1,24 @@ +## [无参数]torch.distributed.is_initialized + +### [torch.distributed.is_initialized](https://pytorch.org/docs/stable/distributed.html#torch.distributed.is_initialized) + +```python +torch.distributed.is_initialized() +``` + +### [paddle.distributed.is_initialized](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/distributed/is_initialized_cn.html#is-initialized) + +```python +paddle.distributed.is_initialized() +``` + +两者功能一致，无参数。 + +### 转写示例 +```python +# PyTorch 写法 +torch.distributed.is_initialized() + +# Paddle 写法 +paddle.distributed.is_initialized() +``` diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.set_default_tensor_type.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.set_default_tensor_type.md index acf827505a3..a8d750aaecf 100644 --- a/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.set_default_tensor_type.md +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/ops/torch.set_default_tensor_type.md @@ -1,4 +1,4 @@ -## [ 输入参数用法不一致 ]torch.set_default_tensor_type +## [ 输入参数类型不一致 ]torch.set_default_tensor_type ### [torch.set\_default\_tensor\_type](https://pytorch.org/docs/stable/generated/torch.set_default_tensor_type.html) @@ -12,17 +12,17 @@ torch.set_default_tensor_type(t) paddle.set_default_dtype(d) ``` -其中 PyTorch 与 Paddle 的参数类型不一致，具体如下： +两者功能一致但输入参数类型不一致，具体如下： ### 参数映射 | PyTorch | PaddlePaddle | 备注 | | ------- | ------------ | -- | -| t | d | 指定的默认张量类型，参数类型不一致。PyTorch 支持张量类型或其名称字符串（如 `torch.FloatTensor`，Paddle 支持直接指定 `dtype`（如 `paddle.float32`），需要转写。 | +| t | d | 浮点张量类型或其名称。PyTorch 支持 `torch.FloatTensor` 或 "torch.FloatTensor"，Paddle 只支持 `dtype`（如 `paddle.float32`），需要转写。 | ### 转写示例 -#### t 张量类型 +#### t 浮点张量类型或其名称 ```python # PyTorch diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.inference_mode.md b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.inference_mode.md new file mode 100644 index 00000000000..d2b54b1262c --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference/others/torch.inference_mode.md @@ -0,0 +1,44 @@ +## [ 仅参数名不一致 ] torch.inference_mode + +### [torch.inference_mode](https://pytorch.org/docs/stable/generated/torch.no_grad.html) + +```python +torch.inference_mode(mode=True) +``` + +### [paddle.no_grad](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/no_grad_cn.html) + +```python +paddle.no_grad(func=None) +``` + +inference_mode 会额外禁用视图跟踪和版本计数器，提高推理性能，其他功能一致。此外 mode 参数额外支持 bool 类型，具体如下： + +### 参数映射 + +| PyTorch | PaddlePaddle | 备注 | +| ----------- | ------------ | ----------------------------------------------------------------------------------------- | +| mode | func | mode 为函数时，仅参数名不同；mode 为 bool 时，作为上下文管理器使用，其中 mode=True 可忽略该参数，mode=False 时，直接删除该代码。 | + +### 转写示例 +#### mode : 函数或 bool 值 +```python +# PyTorch 写法 +@torch.inference_mode() +def doubler(x): + return x * 2 + +# Paddle 写法 +@paddle.no_grad() +def doubler(x): + return x * 2 + +# PyTorch 写法 +@torch.inference_mode(False) +def doubler(x): + return x * 2 + +# Paddle 写法 +def doubler(x): + return x * 2 +``` diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_rank.md b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_rank.md new file mode 100644 index 00000000000..b982129fcae --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_rank.md @@ -0,0 +1,15 @@ +## [无参数]fairscale.nn.model_parallel.initialize.get_model_parallel_rank + +### [fairscale.nn.model_parallel.initialize.get_model_parallel_rank](https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/initialize.py#L155) + +```python +fairscale.nn.model_parallel.initialize.get_model_parallel_rank() +``` + +### [paddle.distributed.fleet.base.topology._HYBRID_PARALLEL_GROUP.get_model_parallel_rank](https://github.com/PaddlePaddle/Paddle/blob/ddac1b431483ddc0f1ee600e799aa31fc0a75961/python/paddle/distributed/fleet/base/topology.py#L463) + +```python +paddle.distributed.fleet.base.topology._HYBRID_PARALLEL_GROUP.get_model_parallel_rank() +``` + +两者功能一致，均无参数。 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_world_size.md b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_world_size.md new file mode 100644 index 00000000000..9f0ebf461a0 --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_world_size.md @@ -0,0 +1,14 @@ +## [无参数]fairscale.nn.model_parallel.initialize.get_model_parallel_world_size + +### [fairscale.nn.model_parallel.initialize.get_model_parallel_world_size](https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/initialize.py#L150) + +```python +fairscale.nn.model_parallel.initialize.get_model_parallel_size() +``` + +### [paddle.distributed.fleet.base.topology._HYBRID_PARALLEL_GROUP._mp_degree](https://github.com/PaddlePaddle/Paddle/blob/ddac1b431483ddc0f1ee600e799aa31fc0a75961/python/paddle/distributed/fleet/base/topology.py#L185) + +```python +paddle.distributed.fleet.base.topology._HYBRID_PARALLEL_GROUP._mp_degree +``` +两者功能一致，均无参数。 diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.initialize_model_parallel.md b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.initialize_model_parallel.md new file mode 100644 index 00000000000..26d5a111f53 --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.initialize_model_parallel.md @@ -0,0 +1,29 @@ +## [组合替代实现]fairscale.nn.model_parallel.initialize.initialize_model_parallel + +### [fairscale.nn.model_parallel.initialize.initialize_model_parallel](https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/initialize.py#L41) + +```python +fairscale.nn.model_parallel.initialize.initialize_model_parallel() +``` + +对模型并行设置进行初始化; Paddle 无此 API，需要组合实现。 + +### 转写示例 + +```python +# Pytorch 写法 +fairscale.nn.model_parallel.initialize.initialize_model_parallel(model_parallel_size_=model_parallel_size_,pipeline_length=pipeline_length) + +# Paddle 写法 +world_size = paddle.distributed.get_world_size() +rank = paddle.distributed.get_rank() +model_parallel_size = int(min(world_size,model_parallel_size_)) +data_parallel_size = int(world_size/ (model_parallel_size * pipeline_length)) +Strategy = paddle.distributed.fleet.DistributedStrategy() +Strategy_dict = dict() +Strategy_dict["dp_degree"] = data_parallel_size +Strategy_dict["mp_degree"] = model_parallel_size +Strategy_dict["pp_degree"] = pipeline_length +Strategy.hybrid_configs = Strategy_dict +paddle.distributed.fleet.init(is_collective=True, strategy=Strategy) +``` diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.model_parallel_is_initialized.md b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.model_parallel_is_initialized.md new file mode 100644 index 00000000000..cdb8b3fca81 --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.model_parallel_is_initialized.md @@ -0,0 +1,19 @@ +## [组合替代实现]fairscale.nn.model_parallel.initialize.model_parallel_is_initialized + +### [fairscale.nn.model_parallel.initialize.model_parallel_is_initialized](https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/initialize.py#L119) + +```python +fairscale.nn.model_parallel.initialize.model_parallel_is_initialized() +``` + +返回模型并行初始化设置是否完成; Paddle 无此 API，需要组合实现。 + +### 转写示例 + +```python +# Pytorch 写法 +fairscale.nn.model_parallel.initialize.model_parallel_is_initialized() + +# Paddle 写法 +paddle.distributed.fleet.base.topology._HYBRID_PARALLEL_GROUP is not None +``` diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ColumnParallelLinear.md b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ColumnParallelLinear.md new file mode 100644 index 00000000000..e31b57d75d3 --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ColumnParallelLinear.md @@ -0,0 +1,30 @@ +## [torch 参数更多]fairscale.nn.model_parallel.layers.ColumnParallelLinear + +### [fairscale.nn.model_parallel.layers.ColumnParallelLinear](https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/layers.py#L218) + +```python +fairscale.nn.model_parallel.initialize.ColumnParallelLinear(in_features: int, out_features: int, bias: bool = True, gather_output: bool = True, init_method: Callable[[torch.Tensor], torch.Tensor] = init.xavier_normal_， stride: int = 1, keep_master_weight_for_test: bool = False) +``` +### [paddle.distributed.meta_parallel.parallel_layers.mp_layers.ColumnParallelLinear](https://github.com/PaddlePaddle/Paddle/blob/016766cc89fabc10181453ce70b701dd8ed019f6/python/paddle/distributed/fleet/layers/mpu/mp_layers.py#L153) + +```python +paddle.distributed.meta_parallel.parallel_layers.mp_layers.ColumnParallelLinear(in_features, out_features, weight_attr=None, has_bias=None, gather_output=True, fuse_matmul_bias=False, mp_group=None, name=None) +``` + +PyTorch 相比 Paddle 支持更多其他参数，具体如下： + +### 参数映射 + +| fairscale | PaddlePaddle | 备注 | +| ----------------------------| ---------------- | -------- | +| in_features | in_features | 输入特征数。 | +| out_features | out_features | 输出特征数。 | +| bias | has_bias | 是否增加 bias。 | +| gather_output | gather_output | 是否对每个 rank 的输出 allgather。 | +| init_method | - | 参数初始化方法，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。 | +| - | weight_attr | 网络层参数属性，PyTorch 无此参数，Paddle 保持默认即可。 | +| stride | - | 线性层切分后参数块的 stride, 用于特殊的存储格式，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。 ｜ +| keep_master_weight_for_test | - | 返回主参数用于测试，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。 | +| - | fuse_matmul_bias | 是否融合矩阵乘和加 bias 操作，PyTorch 无此参数，Paddle 保持默认即可。 | +| - | mp_group | 模型并行组，PyTorch 无此参数，Paddle 保持默认即可。 | +| - | name | 网络层名称，PyTorch 无此参数，Paddle 保持默认即可。 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ParallelEmbedding.md b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ParallelEmbedding.md new file mode 100644 index 00000000000..efc3fb13ef3 --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ParallelEmbedding.md @@ -0,0 +1,30 @@ +## [torch 参数更多]fairscale.nn.model_parallel.layers.ParallelEmbedding + +### [fairscale.nn.model_parallel.layers.ParallelEmbedding](https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/layers.py#L152) + +```python +fairscale.nn.model_parallel.initialize.ParallelEmbedding(num_embeddings: int, embedding_dim: int ,padding_idx: Optional[int] = None, max_norm: Optional[float] = None, norm_type: float = 2.0, scale_grad_by_freq: bool = False, sparse: bool = False, init_method: Callable[[torch.Tensor], torch.Tensor] = init.xavier_normal_, keep_master_weight_for_test: bool = False) +``` +### [paddle.distributed.meta_parallel.parallel_layers.mp_layers.VocabParallelEmbedding](https://github.com/PaddlePaddle/Paddle/blob/016766cc89fabc10181453ce70b701dd8ed019f6/python/paddle/distributed/fleet/layers/mpu/mp_layers.py#L37) + +```python +paddle.distributed.meta_parallel.parallel_layers.mp_layers.VocabParallelEmbedding(num_embeddings, embedding_dim, weight_attr=None, mp_group=None, name=None) +``` + +两者功能大体一致，但内部实现细节不一样，ParallelEmbedding 的切分方向沿着 embedding 方向，VocabParallelEmbedding 的切分方向沿着 vocab(词汇表)方向，故在多卡训练时，load 参数时需手动修改以匹配参数切分方式的不同。 + +### 参数映射 + +| fairscale | PaddlePaddle | 备注 | +| ---------------------------- | -------------- | -------- | +| num_embeddings | num_embeddings | 词汇表大小。 | +| embedding_dim | embedding_dim | embedding 的维度大小。| +| padding_idx | - | 填充数值，Paddle 无此参数，暂无转写方式。 | +| max_norm | - | 范数大于 maxnorm 的数值被设置为 maxnorm, Paddle 无此参数，暂无转写方式。 | +| norm_type | - | 设置 p 范数，Paddle 无此参数，暂无转写方式。 | +| sparse | - | 是否为稀疏向量，Paddle 无此参数，暂无转写方式。 | +| scale_grad_by_freq | - | 是否根据 batch 内单词的频数的倒数缩放梯度，Paddle 无此参数，暂无转写方式。| +| init_method | - | 参数初始化方法，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。 | +| keep_master_weight_for_test | - | 返回主参数用于测试，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。 | +| - | mp_group | 模型并行组，PyTorch 无此参数，Paddle 保持默认即可。 | +| - | name | 网络层名称，PyTorch 无此参数，Paddle 保持默认即可。 | diff --git a/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.RowParallelLinear.md b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.RowParallelLinear.md new file mode 100644 index 00000000000..fb824fd2c32 --- /dev/null +++ b/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.RowParallelLinear.md @@ -0,0 +1,30 @@ +## [torch 参数更多]fairscale.nn.model_parallel.layers.RowParallelLinear + +### [fairscale.nn.model_parallel.layers.RowParallelLinear](https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/layers.py#L299) + +```python +fairscale.nn.model_parallel.initialize.RowParallelLinear(in_features: int, out_features: int, bias: bool = True, input_is_parallel: bool = False, init_method: Callable[[torch.Tensor], torch.Tensor] = init.xavier_normal_, stride: int = 1, keep_master_weight_for_test: bool = False) +``` +### [paddle.distributed.meta_parallel.parallel_layers.mp_layers.RowParallelLinear](https://github.com/PaddlePaddle/Paddle/blob/016766cc89fabc10181453ce70b701dd8ed019f6/python/paddle/distributed/fleet/layers/mpu/mp_layers.py#L291) + +```python +paddle.distributed.meta_parallel.parallel_layers.mp_layers.RowParallelLinear(in_features, out_features, weight_attr=None, has_bias=True, input_is_parallel=False, fuse_matmul_bias=False, mp_group=None, name=None) +``` + +PyTorch 相比 Paddle 支持更多其他参数，具体如下： + +### 参数映射 + +| fairscale | PaddlePaddle | 备注 | +| ----------------------------| ---------------- | -------- | +| in_features | in_features | 输入特征数。 | +| out_features | out_features | 输出特征数。 | +| bias | has_bias | 是否增加 bias。 | +| input_is_parallel | input_is_parallel| 输入是否在 GPUs 上进行过分割，如果是就不再分割。 | +| init_method |- | 参数初始化方法，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。 | +|- | weight_attr | 网络层参数属性，PyTorch 无此参数，Paddle 保持默认即可。| +| stride |- | 线性层切分后参数块的 stride, 用于特殊的存储格式，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。 | +| keep_master_weight_for_test |- | 返回主参数用于测试，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。 | +|- | fuse_matmul_bias | 是否融合矩阵乘和加 bias 操作，PyTorch 无此参数，Paddle 保持默认即可。 | +|- | mp_group | 模型并行组，PyTorch 无此参数，Paddle 保持默认即可。| +|- | name | 网络层名称，PyTorch 无此参数，Paddle 保持默认即可。| diff --git a/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py b/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py index d49a8011bed..213c8174f28 100644 --- a/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py +++ b/docs/guides/model_convert/convert_from_pytorch/apply_reference_from_api_difference.py @@ -78,6 +78,9 @@ def mapping_type_to_description(mapping_type): NOT_IMPLEMENTED_PATTERN = re.compile( r"^\| *NOT-IMPLEMENTED-ITEM\( *(?P[^,]+) *, *(?P.+) *\) *\|$" ) +MANUAL_MAINTAINING_PATTERN = re.compile( + r"^\| *MANUAL_MAINTAINING-ITEM\(*(?P[^,]+) *,*(?P[^,]+) *, *(?P[^,]+) *,*(?P[^,]+) *, *(?P[^,]+) *, *(?P.+) *\) *\|$" +) DOCS_REPO_BASEURL = "https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/" @@ -158,6 +161,7 @@ def apply_reference_to_row_ex(line, metadata_dict, context, line_idx): reference_table_match = REFERENCE_TABLE_PATTERN.match(line) alias_match = ALIAS_PATTERN.match(line) not_implemented_match = NOT_IMPLEMENTED_PATTERN.match(line) + manual_maintaining_match = MANUAL_MAINTAINING_PATTERN.match(line) row_idx_s = str(context["table_row_idx"]) @@ -208,7 +212,44 @@ def apply_reference_to_row_ex(line, metadata_dict, context, line_idx): mapping_column, mapping_url_column, ] + output = "| " + " | ".join(content) + " |\n" + return [output] + elif manual_maintaining_match: + torch_api = ( + manual_maintaining_match["torch_api"].strip("`").replace(r"\_", "_") + ) + torch_url = ( + manual_maintaining_match["torch_url"].strip("`").replace(r"\_", "_") + ) + paddle_api = ( + manual_maintaining_match["paddle_api"] + .strip("`") + .replace(r"\_", "_") + ) + paddle_url = ( + manual_maintaining_match["paddle_url"] + .strip("`") + .replace(r"\_", "_") + ) + mapping_column = ( + manual_maintaining_match["mapping_type_desc"] + .strip() + .replace(r"\_", "_") + ) + diff_page_url = ( + manual_maintaining_match["diff_url"].strip("`").replace(r"\_", "_") + ) + mapping_url_column = f"[详细对比]({diff_page_url})" + torch_api_column = f"[`{torch_api}`]({torch_url})" + paddle_api_column = f"[`{paddle_api}`]({paddle_url})" + content = [ + row_idx_s, + torch_api_column, + paddle_api_column, + mapping_column, + mapping_url_column, + ] output = "| " + " | ".join(content) + " |\n" return [output] else: diff --git a/docs/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.md b/docs/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.md index 65b47bb117a..eacd5212df4 100644 --- a/docs/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.md +++ b/docs/guides/model_convert/convert_from_pytorch/pytorch_api_mapping_cn.md @@ -28,7 +28,8 @@ | [torch.profiler.XX](#id21) | 主要为`torch.profiler.XX`类 API | | [torch.optim.XX](#id22) | 主要为`torch.optim.XX`类 API | | [torch.sparse.XX](#id12) | 主要为`torch.sparse.XX`类 API | -| [其他](#id13) | 其他 API | +| [torch 其他](#id13) | PyTorch 其他 API | +| [fairscale.xx](#id23) | 第三方库 fairscale API | ## torch.XX API 映射列表 @@ -201,9 +202,9 @@ ***持续更新...*** -## 其他类 API 映射列表 +## PyTorch 其他类 API 映射列表 -梳理了其他类 API 的 PyTorch-PaddlePaddle API 映射列表。 +梳理了 PyTorch 其他类 API 的 PyTorch-PaddlePaddle API 映射列表。 | 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | | ----- | ----------- | ----------------- | ----------- | ------- | @@ -415,4 +416,17 @@ | NOT-IMPLEMENTED-ITEM(`torch.utils.cpp_extension.include_paths`, https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.include_paths) | | NOT-IMPLEMENTED-ITEM(`torch.utils.cpp_extension.load_inline`, https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.load_inline) | + + ## fairscale.XX API 映射列表 + +| 序号 | Pytorch 最新 release | Paddle develop | 映射关系分类 | 备注 | +| ----- | ----------- | ----------------- | ----------- | ------- | +|MANUAL_MAINTAINING-ITEM(`fairscale.nn.model_parallel.initialize.get_model_parallel_rank`,https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/initialize.py#L155, `paddle.distributed.fleet.base.topology._HYBRID_PARALLEL_GROUP.get_model_parallel_rank`, https://github.com/PaddlePaddle/Paddle/blob/ddac1b431483ddc0f1ee600e799aa31fc0a75961/python/paddle/distributed/fleet/base/topology.py#L463, 无参数 , https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_rank.md) | +|MANUAL_MAINTAINING-ITEM(`fairscale.nn.model_parallel.initialize.get_model_parallel_world_size`,https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/initialize.py#L150, `paddle.distributed.fleet.base.topology._HYBRID_PARALLEL_GROUP._mp_degree`,https://github.com/PaddlePaddle/Paddle/blob/ddac1b431483ddc0f1ee600e799aa31fc0a75961/python/paddle/distributed/fleet/base/topology.py#L185, 无参数 , https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.get_model_parallel_world_size.md) | +|MANUAL_MAINTAINING-ITEM(`fairscale.nn.model_parallel.initialize.initialize_model_parallel`,https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/initialize.py#L41, ` `, , 组合替代实现 , https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.initialize_model_parallel.md) | +|MANUAL_MAINTAINING-ITEM(`fairscale.nn.model_parallel.initialize.model_parallel_is_initialized`, https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/initialize.py#L119,` `, , 组合替代实现 , https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.initialize.model_parallel_is_initialized.md) | +|MANUAL_MAINTAINING-ITEM(`fairscale.nn.model_parallel.layers.ColumnParallelLinear`,https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/layers.py#L218, `paddle.distributed.meta_parallel.parallel_layers.mp_layers.ColumnParallelLinear`,https://github.com/PaddlePaddle/Paddle/blob/016766cc89fabc10181453ce70b701dd8ed019f6/python/paddle/distributed/fleet/layers/mpu/mp_layers.py#L153, torch 参数更多 , https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ColumnParallelLinear.md) | +|MANUAL_MAINTAINING-ITEM(`fairscale.nn.model_parallel.layers.ParallelEmbedding`,https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/layers.py#L152, `paddle.distributed.meta_parallel.parallel_layers.mp_layers.VocabParallelEmbedding`,https://github.com/PaddlePaddle/Paddle/blob/016766cc89fabc10181453ce70b701dd8ed019f6/python/paddle/distributed/fleet/layers/mpu/mp_layers.py#L37, torch 参数更多 , https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.ParallelEmbedding.md) | +|MANUAL_MAINTAINING-ITEM(`fairscale.nn.model_parallel.layers.RowParallelLinear`,https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/layers.py#L299, `paddle.distributed.meta_parallel.parallel_layers.mp_layers.RowParallelLinear`,https://github.com/PaddlePaddle/Paddle/blob/016766cc89fabc10181453ce70b701dd8ed019f6/python/paddle/distributed/fleet/layers/mpu/mp_layers.py#L291,torch 参数更多 , https://github.com/PaddlePaddle/docs/tree/develop/docs/guides/model_convert/convert_from_pytorch/api_difference_third_party/fairscale/fairscale.nn.model_parallel.layers.RowParallelLinear.md) | + ***持续更新...*** diff --git a/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py b/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py index 49315ff5841..f18bf4f552e 100644 --- a/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py +++ b/docs/guides/model_convert/convert_from_pytorch/validate_mapping_in_api_difference.py @@ -407,6 +407,7 @@ def validate_mapping_table_macro_row(columns, row_idx, line_idx): "REFERENCE-MAPPING-ITEM", "NOT-IMPLEMENTED-ITEM", "REFERENCE-MAPPING-TABLE", + "MANUAL_MAINTAINING_PATTERN", ]: print(f"Unknown macro type: {macro_type} at line {line_idx}.") return False @@ -674,6 +675,11 @@ def auto_fill_index_from_api_diff(basedir, meta_dict) -> None: "NOT-IMPLEMENTED-ITEM" ): pass + # if before is MANUAL_MAINTAINING_PATTERN, replace + elif target[api_type][torch_api].startswith( + "MANUAL_MAINTAINING_PATTERN" + ): + pass # if before is X2Paddle, skip elif ( "https://github.com/PaddlePaddle/X2Paddle" @@ -758,10 +764,14 @@ def auto_fill_index_from_api_diff(basedir, meta_dict) -> None: for api, ref in od_apis.items(): if ref.startswith("NOT-IMPLEMENTED-ITEM"): f.write(f"| {ref} |\n") + for api, ref in od_apis.items(): + if ref.startswith("MANUAL_MAINTAINING_PATTERN"): + f.write(f"| {ref} |\n") for api, ref in od_apis.items(): if not ( ref.startswith("REFERENCE-MAPPING-ITEM") or ref.startswith("NOT-IMPLEMENTED-ITEM") + or ref.startswith("MANUAL_MAINTAINING_PATTERN") ): f.write(f"| {ref} |\n") f.write("\n") From fab87996c1bc922d0b2f7dd6b93152cf2d08c501 Mon Sep 17 00:00:00 2001 From: unknown Date: Fri, 19 Jul 2024 20:32:26 +0800 Subject: [PATCH 16/16] =?UTF-8?q?=E4=BF=AE=E6=94=B9=E4=BA=86=E6=96=87?= =?UTF-8?q?=E6=A1=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/api/paddle/unbind_cn.rst | 5 +++++ docs/images/api_legend/unbind.png | Bin 0 -> 166219 bytes 2 files changed, 5 insertions(+) create mode 100644 docs/images/api_legend/unbind.png diff --git a/docs/api/paddle/unbind_cn.rst b/docs/api/paddle/unbind_cn.rst index 075f247429f..250e9546fd3 100644 --- a/docs/api/paddle/unbind_cn.rst +++ b/docs/api/paddle/unbind_cn.rst @@ -10,6 +10,11 @@ unbind 将输入 Tensor 按照指定的维度分割成多个子 Tensor。 +下面展示了将一个形状为[3,4,5]的向量通过 unbind 操作，沿着 Axis=0 维度分割，转为为三个形状为[4,5]的二维张量的例子。通过比较，可以清晰地看到转变前后各个元素的对应关系。值得注意的是，沿着 Axis=0 维度分割，仅能 unbind 出三个张量，同理，沿着 Axis=1，仅能 unbind 出四个张量，以此类推。 + +.. image:: ../../images/api_legend/unbind.png + :alt: 图例 + 参数 ::::::::: - **input** (Tensor) - 输入变量，数据类型为 float16、loat32、float64、int32、int64、complex64、complex128 的多维 Tensor。 diff --git a/docs/images/api_legend/unbind.png b/docs/images/api_legend/unbind.png new file mode 100644 index 0000000000000000000000000000000000000000..28d8188c19ebaea826db2b869b653db17d6e01d3 GIT binary patch literal 166219 zcmeFZWmuH&);^3%cQ|x|lt?MkF*HaFFi1!Wh#=jK4BaK5fV46UN=c`b3?MBf4TGe_ z(EY#By?=ZEkLUgJetF+7&vA^0;tbb)-|JdyU2C1^xh4vtsZ4}Vi;sbUL8Pjppo4*d z3CF;=Gk}W?{LlRf<}Ki_TW&haau}rp^y|P6m^NSyFa}0N0>Om^7VtBkvx=b`1_l+` z&7WJn9Q!gD7}pPUrn*IIO9j`jk*p5m9x;nSHRk6YCfw`71euGWN{)7E&9Rh^^c_G2- z5Snlc>VIBh(Re)nd0T*xO<53Z{hznPEl^w>@c&vsAeO!hGxfg~2u~__n=AC+pK!fP z0sUuv;2%oy&tbj)*G*{u?@j!*2l9U{@h|85UrYQug#N#?gjM8^A3rYV86@aqrrky3 zs~xWR8Rc*G1L-h%I4KaT@(qae!jMmDlvgFHNc2kI;jl8(`uR?)zN;(xO5ZIFkAman z!q?yoF}Gh=`tc8RbaW1WzACG?tg-H+JT{V(gS{Bc{!C@+4UVDWA3z^>ET*-3?U=^V z3ajzAUQ~K+O5p!vpif2FzG?~5XK)F| zii(Q!piCcn)s`=>CipJTHp_fbNy-NYqgtC&HaTNWKKpOpzO4(w?+}Q`q<|_Zz<3ur zPGDFBk8DTSkMsOd$Rt{sz$U-A`7m-Ss#R3?+Hk?)YEIBbg3Q%WBpKVI7vHHhvLy{; zC?AXryq7rEdLMXob?zi0)0ZI~aCW7CO+It8Gi+Z3| zX{ISMs%QUHQZ|04dv~$JWShbFj{9tb2jB7c9GO;4(M8PKxSf`BM+E}K4kRIvn#u4g zaq5Tb-E8#AOijWFsGjD@g+XiV4c~js`J7$s4_dKp}b;^-GMNxS1ZZ{vB9`9#5 zwG^wR^3?a1TIxYkxwPe|Ox%-R^k=M?#zaStTwlrU>IoSm-be6em+J!16R5=?gw zlVhEzbuvrkH#6-^d+_k@0VjqbRQPJ5l+S6mq?9dbKJZ*a4*}JVAZ9q|Ad#ud?2PD> zaB4Yykzz8rzpqbyWO}WO+@=h#i{kO=Img-0I*M8Ba875*h+tf@T(t{)=%`s z6pHs2+CmP2D>N=bYG*w&5qc%R_^gU@0vlS+w;M0r7#Itm0w13ZBhpr^7XWFt3CwIn zab`o3SQTg7Bden*IJd7IzRAA%{#LN#o|L`F$|y}3>NP$+=93p8S**q0%W)z{t!Ssj zFnv>>4-%B@_hs)Swp*_Qz5HiO1 zIl?K?hkzm7=OY>`a8Sw>>SrgRT%J+!J zbPjOP;OHv5F*UuCrygTcxHyPUZ*7*p#g#^odO5)1a+MI)EJlB@-X{Y=5vx+Kow;jr znNN*CSeDM5pDZVMdAY0f69}}E=z(#oTN+r(l1#Y7%cxmHy z!)4h8o!Wd7@63AsoAZ~*jmteePPkmpTIEBbW|;@g{4){?sv&f^0$a^8QVR`RN>dAz3>AH-v201*sLjDFzQE|ec!7D%rq zne?Ugc+Su9O$QE>zd29_#ghZY5!r1jNJqP`B!+7GkX@@w%wJph8Q5udMy-vFW=@X&M1K>yHV!o+JG$c0|{ zF&0j3%R9y6dEKLc_wOiB0m-S+YiIl8ar)`++IjJ$z2yWMyUVk~($i%kk(pe;6;@uy zZqB#1PJ7B&$Zlx|E0Ms5(64-749_HQFj$!zMUjED=?RiU>)lqjgASSCyDYaBG(jDA z$wCF(*C4e!gcO9P(&WjVivton?jxJy&zw9!&Y!xOG<&2EHYWyuGs#cOoo;;@JV2#r zF~Y}hkArsvl5_mF>$@44(litq`fwS3(P%-B=_n zGY3U?C7k(+kL1bXuS_$nZ2}&I5B%5lO1jxh^?S+J7qN>k^O&{a%u7GjrBSPdDD_3e zO8r!?knKC!%?Ka{W=)FfO;bb|5J41ToTn^e)YDoX6u)`aOx{hMz@i=0N=CAlA6p?w z(`}NjA=W{4b{7zTu*Xl^qTfOH0-R^orK|zh5W=}}4NNLg=}%ip7pf&k|p zwB+dnP76_D_3k~f^Q|u=?1c7T%xL>lZ2S`i4kYuRmpn=5(}aA$qVf!-iNeA&=0bvf z%|Kj`uXqaAqAXu0Ie+aXh~=;4MKHQ{Je~?{886Wv)cds8z+WJ`Gh_Q|q%exYA+KDD zu$8b>e90jBBI&#H%;xxKzlC@Pyy+}4w<9eY=B!-SVwPgdMjBUq_9T0{)pz?U<+Ju* zWIFdpp72XS(1R5p(WSPdA4@k9*4I9aH~F}w+hZR;TE1Y>N=l(OOVeQe7U*nj$f3%- zxnObY%EgMNNrbFmi)n}IDD{|nn%wAtO7VFQXUaXRU;0f>fL{TP9DD6$J z;s}#`P4{TV9bMLnVn>k}chnCnJi(ELn7H=b<1r}DZM6010_oTJhg`(HD!%>-N?VtA zY;!lh?!l_ZcI)Se$Tj1vf4=LH{up%dG(P}KWY2(Ow9>zH50?wE6|$O_L9pH-KIwQA zVtcB_*B00Hy{V>1AM)9tqP_a}0ygg{*7NwRO%+T~J!QS*Zy8c8`#U1JBw;{{hBUr8 zm`U2hqDj7`xlu)R2FFl<6x*l@5Ab<9$*6piZ)bm}GE$|$harsXT{Q)?t91}iZqBfa zq8jIj7y>?_NC{(h5?&b~a||nV5rGU2T9*cBsq`^Z_#H?UWxI%4+1A0-0ef9kn*J4% zt+82ZTs2X7o+@+-I6C-vVQ&|o$8}m>ErbRuCE?_KX|eGcynEgx<(Ew) zlgbJ-n>ce{R(SuAihiUVB3E<=n#D5e9pqg5=9mkGo5Vk>WSXNF1-V3W(#<-Z>lk+je$(@0J$SsU!W z?{J3U2e&iyUSIA6ZLi9^u*0I4skix9Mq(w>XG{qB+O!gK)i#^kPgk?F|UhgdqTl)h@LVN@p@d z&zgWYz(i_^dNos(uZK~eNa8~+l9Ff{D6P)eqalhYlD8&0#W=;*iD8KiER}WaAkvoU zmNs^lPjc8(O~_?+TJ9M^DPP4sL8r#z^Q6hgD@@-B$8)uen@Of~pyx#IJ9ke2Rk{@) z2Rt@1B2OKcYMXp-kzTM}aZLpaMD;sc%CA8};Egp9_AVksC(eUiS<(0SeC;V<5iv6I z}5trdn4li=^Ktj%QNQBwg>^9$!aV*VG%6NG^o`uPB%~nH6uhm3V zntF4#hDoRRJ5&L!BiOd>42q!)A12$nA7|yt!JHYnbhTpG$e@eNE{oof_nkg+KTisk zAz>7zUtOuOdl{b0;Y4H*U(6@{&^`>cA`D;7AOm-usU+8T&}u$Kx5!Z^zdh>^Dc+ES zXkRh$k_FrGl0)X`+Ca`X5KmGjiwrvs^EG~&z@jWf@Bun<^M!Mlp!^}Sgp@UqAfn3L zfJGz)SquxK6wzK+?TOQBE&D(sLfvz#2kX)_<)GEdh^H2rS=W-XpBUqEsXV~^?(IMs z`bAEU6TEyCThFMC$Pn>%QYNPZj|Wi=k6sO@23fVpJ`Z}x12>Sqo#3Cka~Z%k&o0AW zmS-gGE&=37rC$Kr6zN?zPDVs4h8NoUTdO(E1_Z`U) zNNO=Ag08MKk`uw3Wk+>IN& zG)~M4-4vM!z|<=rVr5$5U>yyAid{jhL|smdh65!R7{W z4EV%6ePPQTDzF6yv8>$ZlWeC{Y)r6y2Qx?#1>*zW^u^dMx(9dVoJ!vTRsZ)J)G+k* ziR^b~iHjJaLVZXQlv$7ZG1A}!IKdQ%oh?49zuUWbts;L)cD`4rBKlka;WUOxGD*i+Zjh_$%r_8Bw@Tr6lpi3 zzl&1J?;ESBEhUK~l@vg#wB3=GaZTtHLQGKWavioz{V*pK;-Xp1`@Ehv0W%IW!76z< zUIA6gsil1&Mh@;G)rc2NkW>l{JFcLZIUqslkM~hbO=5@J zo_ZmH$Kwlqw4oF!@I6|`3X^R5i7;hyj1RRqE)YWOn2*8;m5pH03enk$D#XcmLTNrs z(}Zzs&V(&fLudy)Ss#<~wJi;CX62$6i$xDY2NbmAa@$w$J;n`jObNoH zMdS1Mv%Iy26y5}|pUQhC9k_f?l9cj2aMpu9;ymnu=Pr58uqbluJ<3=}_F_7F;}SUK z*m6V=&n5M$$XoFLhi!>7fIpN@|?E+ z142hakyNB&D8mD-d>RoEk;^$v6kY^5^kR#Er}X?QsYEu0`aoyOTU$|0K9w%frI?FE zk6mZ#Yw>L$UF$wF#sZOKSoom)zv}w*kR;mLb4tmrThh{RQB5!LCrnTuCDd80nHdG6 zSaP1zNZaD^_$s)hBwoZCM|0v*pc<<-xuEdGzRx}D>m+nMi$b?y(E&D-mTUduK;@rQ z58&vmY4TBQe2@ISSKAY~&?WYD=4Y_pptEOOycC9=SPVWXjKCiC$r3xeSc?(PqJbPU zX^POVyR*v4Bh4GLLB+S-27=DKlZR-}x^n^57DH4n>muu7UPeUG7$izDS+Pr7iKQ)n zk)u6=^W3F~tZi9lh2w1%rXJV9x!gs0z*lL3t=MJinaP=mfV>@06p8b9GTo+cDK>Hz zL1Ha!UfV*fa4r{{NyhUIh9P-8?K9=wK;~kiuwFsb-7)>P6OED4XJAF0?peJ#`K$!U zq+>WlN8{IVuFzl1Wj%xr0u@Wz*Cr#O0YL)~r7u}~9>6RnnNs!gV7)tFMUXq(okgTj zJc`b@DiMR7yyJK@bN;p%%PHli#icFQ`Jjd&d#V=TbnD27OY8lyYevIPH17DIE|Gl1 zFD$pZrlgAmYe9dZM_xC+{--Pgo?lr{L-Zt3a|Mrzl`mHMCVdnW+qOBEbV!K}07>A~ zxjRaY-R!mV9w8@OMS&W7G8cSjLS-~ULNSuv7Yx99-|n%>hEaNrtp&>#3-v#g7t;Z3 zJM0yiyjQGS)FAN^_c5Z8)QbkIc!l6GimWmQfp7+J6?V2fJffkPXzP(L1foyvIaLYq z-jZ}jNAr+NCo7PXGSjfQSo*={+0?a?RdoI4Wb9+KlrTX(%sVWIE`zzeT%!aXzwJQm zjw|N;N$fNV!b=KKnS*BBds(@+ulac5v@g*frlkT8h@CbZ*7S?&2Mk!P%mc8Z2ep{cbUf~Uh;J|WR(Ij`C2uGO%kg}~QZiv{M+$B~m#poOX=L!-|c;fBj`JmPm zOz3oCWH~<+XAdP}h3D0MfzKm5Q_gfjyeNMv5)~n%5M!zZJn>9`yklY=Gx#D#GJ)>J zOGY7_ykxSq-~|#YR2K76JJC)E&|x4Lc?w7)WUfvM?&B1#bHc*sNVag8M@|@6NnF}@ zrIJYzHKejF?g#hbqMU@sY*Ph~aymm#eXf~bz>QgxdR~(7$oAeKvxnlcLKVjBG^4niFpJDdt==V{ok~gaXGc+10EBbeopd-l8hx0! z<-+KbQ^8o1B9R?t7~~^JV3?wUAYe%O^0ShM8VZrkD2*D zI$-W&tq#omU}aL*N{mR<#h%A~@&uC9C`dA`%(sOD@zIi2lOAxtN|a|#XD!OBZ^$Y6 z5_pt3On{1&%zNvfLI#S3v64*P9(p-6Lz)ep zuW5g{+H*S5rw1$en$jb%&frE{dN&wLRG!wtJ~&#CsC!ha1x_v(>Kreo??0%IXd>lh z2T3DY(}E3HSgefVS(XTH$-@*nAm-;&#PJk6bmS68aREia5=-&R5esMwvq|ak-^X_? zFAs{BJhrXcL-8oGB$tK9d;CU8=5n$gcBb-&ic)3a#7m71H4;ws)a;pZoDyR1bv|ZY zpVWuV-BsbAGyC4%5Rzh${9xN&$p5v&(q;RpjVVu$T#kPva+x|ABV11FezM$ExahB& z+L{b6QN($faB*=p(f?iq3Y$&n8x=3%Asf!3ue#QN`P#!CB85O@kA$GlAI~y+y{+y} z*!U{5b|yk4xii++P1dyr%UX@>O(aG!^2K&FY zq$0k4QE;>M^nbVlDiXdc|HMFm)R>KMy$x<-xEC%S)~@(2h6)|sX!a-yweEPKt~USx z+$M@KTDw?M4^;Tzp7~MBiwndw?V$+%P(uO2V)Mfv5A|~~ zh0iROPUOYkUs;>!C(s;M;YhS(ccrI-E!wGX_7yLz3_C@!huE>hmcK(Fg>2md&`xLi zx7brX2M1aCBgDhqzTaYLz6ABkqC5QJP+d;m1Mg)*@9wa)BBCV{cq3L=P?JtF@Qr2S zdwsjLIc0CYxW#&tjIyUDdm-{CkB3;2w)m&}uM&4F)UEDK_gDOyNr5M0;4smV%cID> z9Nr~Z_E@HO5P?zgd_ffSeUM3+IRf4YiL}34?%dBgxI~Mr&Up3s^v=Y>5KdJhXQE}2 zNa)=6MH@7=D^6+gRW`wh@mv9azIypozYbYn1o)oM|tTJ~F0J3#6 zA*b+A=^sR3S5Y)^=`!l}rcj->WNd#7LA3AJxg*Z3~ylr<9e4yZGqT$B$txF?hh-+8SRR&T(uIKMhohQ?!T1RjU+({lZ+QG^kswl zVV+KG4JDEZR)gOr#s3gL6%Dl%MD4xILqtbEtcl^&9Hof0yI=Azvf&SW|L52D%t)bC zbh*uU0t#?L$A<0igP^#D-{G#$X(VwYmG+=s!xV4wl4)!$EvUb>{+AAd3_K(fRkUvk zAV=Mwa^;SOL>Xxmb}uVN_+I~B0E4e;$Y^X)j?_xtKtSB=|!)Itf^< z;x%Bu8*|(M&bW&P#d3gT0{8z(NfB#|@729UJSEf3LLRTo{=4L6G1OyX4+fIUDN*08 zx3u3XqHZ=L%>HMUs+RO=A&(vcl`TMCT)rbR;~b)v##^lOL9N(+s?w;*d+%!g#e=D& zgOkn4FE$B&v|8_ktlfLLi{JoT=et$WUFEi_dF=P+w&ER@mX;3M3HadvskpuF3-C>& z>1Kh&K!t+{ygc(=|B$}bkgdAw{uSV4dsli&o__v$jj*8ZzJ1eMP*pn4iqE?CJ?Ks0 zOyJPSsJb3QTB1=scGESVH6`~HJa*Cj3bCZYWAh`x&Q3}Na7fut*Yw}ZPdD*=C+@zMyFVbI1(2&= zW~cQwgV{eTt^2NuR~X-207&8u88QJ7!oLS3&O2QHSX{IwDj{KFv#hmrW2~gax-X5- z#^+#IdA2&BGm>n(LDR7k;2bX9tQfVmwa*SG%w{#DTCU4v1D_Bp4`gbOmzr#%+2c3c z!w8OB?f$iu$godlt)>7UTGY6!3E1?yBniihVNh~kUHIjY>_eLYfY|GsVG9m;`pFfY^!sS2V2wLTjwnX`W3J+i(!zWVQ!x1q;joRL#}=$9~`r$0{t9Ma%&B zehi^^zt&+!Cx^M{))N*|MO;g_0O^(9q`nhg)~B>n`?M+nbN)N`T{cgdsh)ov1e_ z@J#ilxSIX`FxHarrWYX)1cNZZip7$uOdt01m|=7N`x@zQ0stmqTl1dRvT5L{wXnm_ z9}KN!tye$wxUApg1YH@LG`J6|a=u5w6zt|(0syl9G5ZI=gX#h0K+&MgwXxf;?=GuJ z(Y!6_hhEP*_5el(E*FNsJ&^l1HbjqD26GwO!l+EwP*5W)Pl8*R)Taa06Q-N!{A+xb zYS)$C^`PtXc|dAFzo}?V-AxQQ-m;9+^XK`EqcxV@#9k9Pcol#)K*T>*jTRRK{oY|B z`q4&Otxlf*+ePfbLaEMf<*8T}p{kmi>*$_e+Mv+8Y;%*gRJFYF&-Pzhi9eEXCdN~m zL9{`sa~t<^DlEGjmc7RG^cf!U3P`bYa+Uyeap_1gKs$;rhEl|LM0LDnL1F6a}jQ#x+#Yi5aR1D0uer7SzJw)!^W)swGH>dIH$kCT^Ae9U*nCzS(_ zf>u}K@=p59WiEcNoG)CAM`R`8`?M~_I+**W^;hx6EO)f*e`%c|qI*m)S{gXldvwd} zG61*V&oI1|c-%p*PoR<**yKGX%ah$j%?#%WxbtFrDfeUf);ifydIlI4NKb?*@FXuY zTrE2qbbsS$+Ua^?NSLo_CaeLK{KS|9Gp&DCGWkPU%h`rm&#h*B1hhCmhOYtXFgois z|GVbH%i;V|67oyJc2yqoPgG|9!$gLbKc9C+7XxwTyZYX1?t_4J{ z(9ijW6fJ4(8;V zo~WL`{$+tuT$SsJTCFkKtY!zWkB1nbXb6~*4D9*|IE}}&Lv4N28|lI}5zU@EQ(7OH zB)y#*_Qg5`gyMm)I7#^Wo`L)|kWk@vV~B>O==%pZfZo&(06`QfjFEowHS7Z5vt6mS z0XA#mp4dgU9iuNbhzF_y5WN!2Rj*_CN7i5jd_(;r^ILo~GaB+wlW=$0Y53NpKM@22 z`VOgdqyW=2%z!0nsdT*DqJy}BK(P{aogF@}@?bnn($erIr+rjQ1rTHvpSuj0yqV)iQvCZ0sET_gfri0 zLwpZ6K14A3B4@X5^EA|3C=IHkxpGPF{U*?qI)sXl)HC`JxJDP9C6^_cxhyBrN79D| zK9N+%(%uL7>xLCxgYvIZvf&Uu9^oB8Xpuhel9J*v zEEL>C!riq)ofhHE8bS1#)MgeX9?ADnk+MPqm(5k@dQo7e+_D}eE?NDS{%yqUP!U0e z!kj9ImZXP;Q}32-7depbu|yPk>v(_rfi&MmXmfF1(}kq-nE)$xrcM(RW8-0jv2 zbl=xQ`T41t9P}IW0INEAW?Zp4$A)2U3uaQU%>Ncw`8XZ zluI=QD{_lQG3i7GfPZM;QoJn;Xp!z%Z$yL?Mk2Vk$&R&IUvX`k&LdXui=rQixz4rG zsLSvUAekEZkXnl>yNV7hV))h#aM$(z%K|<#YH-@A^>-)vhoQD| z`t6n^dKNF0Xcbu%d;?0aTK_r%B^cO4@1>l)t|mGDuRLD@QXZsms|={`7cmv+8Jkk< zkl#-8YR!7Q#I|z}dASrjG1Q%bk4txT+rXJM4e;gXozms>3Z!1kYdK#7zdTr9zg8Gg zNmfy$S^cI}1H;lY@?smk)1B&I>BFwJsF3rwtT=H3Z^Lqzl;}VT(h(F;!hW`N{P2sA znW&9>jt$zy3V{laDl6JtP*WB-sZyZK$Bsonl0!W*@^x_9pE_HGuRTuHu>nh7SSnQ? z+JYlr+DoKY6OoyV(xKo`ckVbgpw!!LgE%PZo!-c`Ze&wKf|$vfaa+1@3;6bX@r2t7Ev5Bek$)-Gh z7zcr}nlC^gszfIS*Uq^1ftwqMv3f|O$eY<<_ulct;Z{$a06RH5?gKR$keRP@*~kCY zVO`WB;ko&}5rLXjcbU!baEL}|ZYvYZwJ1qo^1$lIXn5`8l^_*?Jh2yZ2GR!)6nCA! z@Ozuc5@2G2M0i6iTGxfU9zR444U$}%8@ADFLWbQAt41~5x*xmK0bVOz6~{cEj~|;g zE)cub2K00n5xL0`s89qK&&#?x-4^vR;(q>1wU6dtkR&e&ptS<@Qsvir^_Y>rR%RTQ z@-=q2!^s1Is-4f)!tg*B&5v%$&G))1l>!4pv|ZzJ^JygZcME0G!2B3#r@&!Xhshxd zUwi(kl|Ngaa;kfJ~~E8m9b3=P2`6 z7JC?b=6V3DnW({IJeKYNKqYf-c6p=B`4fvv0Ea6-FXyCi1_>Legl7~fV&;c=428VG zsXL+noZGPeF_ZWp5WR<`_i~H^fPhg+Q0Q=ER`>>CpFI9>O>p@7j409PI<fH4yXZYQvSj}4`c-H68Gj6*gvtnR;5g|m85(Z&7RB`-h_qIL1I|M z@B9+3n)0OgM=|B{W*$t)Gn-jHL?ehU0oc>|M=y$u;-n$A!O((>%*&PHaLKNOmos%0 zPZ)g|e7fFC(!L9yv;B_yrgiis<8Zgwzx&pUT2UPUZ}8*-W1-{;D+q6G!J?H`#Q;)D%8)YdXAid_qYvFFvk!%cLeCzsXTu@UbisL;1c;$4<3xZ@B?;KB zU!CbU%Z^tG?CEd=>a&U)DR1SCTrP=V!KKsK>N4;K*k$L2o-kX&(RkSJ6v#U(0gAQg z4!YYZS(E1Rn0A}Mo4AdFi_`r^#C>z$vwK5j7!_%ZlavYs`H>QKwpdXF9h>ne?!d*| zVhN(zDdkpJubCs){imOt6C__qOc7_iB4^Kt6pm-ekG?Fb}qg}-`X_lHOlxO!$)lZ$^$3d$AFq7TjtL#bT7sxr)NExvYd?Eq&dzk}I*eSNFc zHyUvcSzA<{>xlbm{?g;?RYXZ{zUo;L6p|F;B01%0WB{@K=AxXaZSjqwny3S5^fx&q z6QKyA@Tqh&gh$v@6n#)nbkFF?V)?FD^x ziteO>#tzzC9=v_-m&l<)ZCZ4OH+<()2mtavu%hlCp3=kw#5$MeDi-C;Y-9^w_JfPC z!f2lm#UG29plP)(&@|Fl{)uVu_DVtpR2He^dm(ws?Xbw zn$9*#9K~)}x>Xqa52kRWi4-|YvSe!%*Xp~*2xv(KVOpQyb3@6ggwPm;RJ(d-FRXv@ z6G1WG??O?wD3V87oMtRR%rb1X3m;iP$A2_`cHFLnt}3D5flAc4ML|=}(Tz*!8D!6> z-gU(|hr6b4kaNm3yIh@7k>s;^10Zh9u2SwJ_{mdF6j(OM#;~L;-P5n>SbioZEW({+ z3C(sXI07VN02yX&=NDK!^f}QpCj|E$AV{cBa>@Cy% zMY;Y_J$PFF;2RRHDNAZv*N_eRP|%0*H(KY*yCHrM!KT>GTmTT+9v}zlvTGSnsv6W$ zge%}_5~{!_HBq3x!ZHjkjzVq@JxCMd4AtVx8*=81*m36?VY|h$%hlVd}c&y0raAjVvnNS2lAn3(R14N znIs}K#j>27%lD_B%yqhN;hZ!542gb9z9l2Si@ayc4JPgsWGk{ZVEhasz6ZaMoT7pf z{$7q6nfclbv_vGmbcTHOd=B);S>W6g|LfZ9cvK#IM}$=O0XZE(A(mG#>>QgJ?$~9r{JCO1Ee4!m;gmw!zG5=YX1n`LX}%K2E&(N;++9w-+=D6b$$L>^xBK5@o*;zsE1 z>@{fcf@@}FsYmEjgaRg6hbyV7mek=7FbJyL2Y9tt7yK9NDT@UgHwzx?EpUc5n9{c> zRsqo0z8`Sb=#^i5>$>Ir^ZA+|fDk;h9Akws!o_)BG3#UFW?pj0zJeKkpf=r7)7O)o5#7>W7~*DZ?gYlbZv zINu>rRn+X;Yuu{rGd=`qJuXD!v!t7XyRMiU4H!?~q@S_1^~DI_8B!%)wpKp&UkpELL1>)+l=kxoC$a z8}b}y>lwo9f&@x0nvjSq2K;!MK9oNur^Hu5c074gEdH2A?~CvC)}}QvzX=%Hsqy=+ z0a)$d@sEopD&3EyaK&IkQ@TDx7cLkl7)NEjmSmZ64(g;7dz~|XQcvk)Qn%g{f#iRe zwivAbL^F+2awgpWt|4Uxe^N=g1MkDTXhHP1A-ZJyJAKb*0|Wc2Fgqxwj?6_Lqq5bN zvlRr0y2At5%LqXl(rrGK^ZLE>znu3^>K{&b`M-NtLL2V#r?(MhO}zg`!fS1a#s0P{ zuk4frELy_!K-}EhVvPdl;q4yM%j!B^Ws(QvU#oCI&)X6w1AqdglpGKc@dc~#^TzaY z0}4g6E&dLSgkQG7e_&sFztFx1BfTQ=)3Ju1=3+ZC>^i3 z?RA^`T1m6oarS3wikr=r$|4_~_42__{FV+Mc8H-Oh1QuVgQhzJF(xRWB++H-zWgnk zw0BRd{~5gdRuOtr-Nr}KJbY2tq5*(~eLq^X-{EQWaHkh-Wmej&)h9gVWr%?Ubd{?f zCFIsVtxNt3^WFf^mY~1)BRLz^jcMjStf+bWAKV(4Ubq?8i)Djt(1#}u6JEYNn?gCj9+Zrkf1VYXU}zp)40yQ`RgZsX~%%$ork0s?}VYx_q4;40y@4Yjk| zxVm;!a%}&HFi^k{G4(3x{qeoPCVYH!wDpJXHA}rjHl5LvIGjUBRN!&(_lI7Jnv!1W znwy?E8y;%~4Ge5-8^)8$Ag2a1kYlqM$SEoo?!12o@uM-p$ltlzD8M(>C?G3lIc3{U z>V!6R!#88&Y1!wKlatHqvDlgaSSJ$0*Tkz4QHw0$AFm!+7&HCUJGs=Tpw&6rWZnk=_|=T4f#!T+$ocIL@$>0?}yi_e?-n z-+?zof0=q@IM?#g=lB(jhWh>R;n8!od7In2^3S+uEbdo4#0#H;DMZ>?t989hi}4iH zMhtuy%@cGa*QxM<-xRVeVWu5Zf{l&Ra7=iGjWA3IKF3O>(o$3s1s@*Bnh4N>J<|E; zy}Y*L_Pk2=fx*$F{}3gS80rv3rMJ1!u>hQeQu?olG>7pBwPWg80WrWP@}J>SRt(_x z$vEJwY`|y##@rupob1-$ML;xwA0gxZ`~Lqdftyi>EYLy(h`-=(4|>IAl-uyE@- z?uJF87K{2Hw<$!#M+r>U@g4o1E?QfA+11%um$SUQoM2?&6EhNuH}Yjrt#*#bW0S+g zEzW&DVB~oI2@q^KLw|^@pueX0uZ@T;2uE;FR$9mW>d!1%=}Uh#oUdq9ZI}AkaRv@( z+C(itye06;rKRMoEPwN>j*7qlSxsSkE@4|%VY|C0zCUes`qD;T-$IYq20i}+r4#?% z=l|X!Wh{`k3J0PfQFlJ*x>+fK@%M9pXv%nK6Mv%!vl=@)+^n`86HKJN%Vq9 zcAmUMbzo9)yfcbI7?^5sKjdKYRf;5e(NmY@>K+(|{T?`O7J*r%x6d)bH zqb(P+BnVXNfpHx~_oMnSGKuYd`58bn8K;8E%AVC~0^8?4dLCS!pP=V^rtcG@`(9jU zw&Aajv(Zm+!>?+aK46zQe&8Rk8BgE5-FBSw*&gJ1_9+s$yAU%dFh!@GT848)Vb-!` zN+J)V`j9?tJBu_)>}K)wL$HC}HbF26rR>~BruDE_vYcWosQ(R!-j ziO++Aq}whQ?7um0I-IBQY*8nmfRcs$c`nRkWV&u7@FcF$7XG6}T3+*P8Hu`ti}C_{ zpsCJ-gW`PtF@Pd7$>HHaaHzk{)h`Zc*@{k^V_3O*UY+vZuiBwXyK}>sqNN; z{Z<)n4;vv8Q3=DxqAi*xmAMIHk;FO1xb=IDm}9FY&(;gD`aMPo#yq~A-qZVVrHYuV zD<5e7byo^)$PiFpno41?_^bar@+mm%_1VQeJ?~`~P~(X)2(GJf%JUv^A$-5&1hiVx zo|#~iS=dVk9L89(>o4<{deZudr{<^g@DUD_;dNl_KF+<1a@FRc}UKQvd;< zz5)eE7b?q=cCl*>Y7>wSRI*|zS%pBY@}19$xSQuv$yD`keIn6^6F;t6`(Z&8i0wr1E5uD z%#>7SEvuA=5Sfmt?F`s|oC~po+g+yosC;Exz2g)8(}Taj*CD!ic&h90N5bJ@&l05} zUvZ@_MDKJ1^j>_y^u4>MF38^CN!Xwrf|bFjcz3EF+T_QXN_XPSM+rq12a8`;7C}Gx zGcb5vMDai?VTjt_#c30*G(rxPfOjE^=AlzDWlyCGp~WeBm4{+H!%MvU1JItwtHr$z zBn=w~=lvI+R++i9Zs~Tj!%BLkFAI6n{am`%5nBOLhclyZfYL&5%~;(C53Yu4)}^lRz(KUi-A-3;0XV{WS*w;H_;|8E%E#&avvP)bwG20LJ;l4 z?GzrFU5n#$z#58HwYhe~`@+KYXhS{GOF0m~nLcMKVy$Idbicuv3qChgoNihIOGOl0 zO&=kS(cQs$0fSA&Mwey;5?W6cYmRd9H-yJ*mzu?|2j}b#eqf37UE_&guEzsKgeK<~ z;tbA*9hiEZH{e#RJZxq!;(#R)?wy|z#i)bJ#(sAsT4`N9BKLVDlb7Ep8@yeyylUxw z$L->BIRlo?s}sw&;l!7^G8Y0S^MQ>DPR+t*&Ay+}$;Yz|D`}$rZ!7v7R$BtD9~mE? z>yvSOx^c^Ed>{*#0Nq?JF@ty=PU6tu-aDIY?cm#3#`i!IwXul|u#J+|>9r`Q%@W6w zd^bn zqdeC?3wW+K*btF#&*t|vpN3_M+-H@*0tI}`Fxii`d+!&6rsDqGusZ*FWGth>B2f6} zFD3Bd3t5OBJpXyzGsL%$LB!8yEyTAwgYaJzcbX4vateD*!5)LbYeLlt1E*KmsZvGZ z(a`%J9=5LK@r`bi$e6td`^OJM@!2s+@*|24;us{xd$_lM>Iir3Tp)SD9(B2}&7Ib3 z&$uz$@&St5%CRTl1ZnXnV+*~U@~zvo2TjkThc*7y_bJ{?K*A>70KeJ?f}&fIOEa3c7c%V3(N ziDq6c+PX~qc6ryIkqj`z#&KD3rydoL*@`QVE0_)>UXUGs|7oY5F354h*gB_aR#=ln z#OcpTfC+uSzoQcsO$4oRl-Q}(hyJP>=kc`rY|Qh==?)>muBmTUPbl2|lb(xLC;&-t zOcvx+_e}V_CD|dF=NXnBggM%%Ip~8XSF+id-{Uc#)|pQODV5INOh9e&{Dz18)|%TJ z8FUdzUr4-U3lQF=GIym4;2-ntc%TrLQb6mKVhxOFE&?r<;Z>=Lgs|5(eg9iQO|t-D z{r3_(+DWWunJ9(oDZ=a|im4fM0j2RKNIQOl0%TfIdR@ zv{$_1M-PwJ${}LketR7ZqJ7UXTXH1tr2Tn%nrjEoXY8!QapPpdWJAG2@9p}-2S5Q% zcvBgO;wn&FACN<?T`M=7BF&$U^OfX_jZ<10subVHgkT`4Lw3nzA)j|JdsHcZkIf zeYnk_Hz1G5-)S}5X|DMiW9Z1mDt~7R$JbC$;rS_gMYF0RjyGaA0#q5aCDGoC_!GBT zuuUCcV0Em~%gJk*^DQu;toYfedgPf?bMkN?!eY40UttzO6E4RJ-Mw*=zhARj% z5(DwVUuMVWJ{L8Aii?o)JK}!qnNwTZJmHmTiM{(3L;ur?pM6n-`+Q-%#-b_4zM-gNlTw z);B!8m9X)vwKju_=Zdp{!p9NrD*$})!N&{{r&3_F|98-eIUs>5(UQ4@9?!TX+P_rc+JmaJ+-w;WW6^#dmDSRCkKguf%*=oCi{?R714iM+{cn;YJj} z--t__fm2JovU^2D%~t3Lm30}d{gAyLf%bm_o z(Co%*BDANmDtRQbQR!?jJ3A&}@2Y0}EFFcS*%_fB`he$@`dT-Q;-8v{K2aA*-AUda zj$pgy4zP{lmH`NRt)>4%*IR}~*>3&A_pPXaAT3Cv2r>u=f|PU#$WSwMNU5}RgVGIx zh;+j+lynYAmvo81&@Bwz@t(8ye*VX^_kP}Q9OwtVuJc;wTEAMoRuTc<61gAgv<9}n zKRbBOp-;VRty|F%sgF$aPRvUcsYZS<%2!Ha27`t&H}@Serf5w5Us75_YbmTadu_0iW4TA49d189Y?0>XXw~B z(tA{|9*{V$`AG*3UoMz_jG%V*`UO`Uq80J^9GmiG;e1i~XUSrum}+NAv|&5@_;Icm z#~&UZ4sfpB=G~RHqJ(auH9prxII11$*!*HpKDx_q`&QQZE(DT8h1NXXlkL$gi~fCu zq2_NG;8P}g?v)$(ulq|{0Qde?_oFw_+y_=5MdTaVi;JI_7QFny9RR2KT*f{2h449* z#7B4Ln~yM8TE=I4V3Y>S2Rb}L&NWiJ&kVp|`RDO0+u2?(6~EV;Cv(jtFO#F*(Bvme zzGli_D?bD*w4j|#07aZ;7v$(pE%-@kA|Yz|1XUv-OWPUVgRBuSZ^2m)WzR;!b7G;H zH(sN!UU{7>_+0lz_g<3GfcHRhM=eBlUC+?ZZSK<@JtAbY-yMw4^6Uqth2ONE&zCz= ztMhgC8?+$^V%6hDO+^{SG(-VIL7~e#J_o5&0KeqC^EO93FA>m=+aR|uA?kk&Y&@Zc zS3+a3@ACTaGe_~cLoP#&^{M&-aTGPg(^>u4VXZ-%WRkC~WO9rve|PzEbZ4P+bSLrn#JSjyQU5du7=OriuG7{oKQ$2VBnZZrNpcow zjN(ajCO(DnXo2Pz*{Dc7_S|9oCUYU>Q>m`ft%Kn`58K^f+f?!{6}jzK=Y$iJVq)Lo z%}!fF7Zd5V`D~J5$S<(maL4m4g30K!c zihhgt2t46VE@=@~M!uhL?n%Pn#MM-V^$>Z!GE1w|zqJ9I2#JHj6N23H_g1;UgDrm~ z?4hzd+BlbPch=3PgFsluphx(WKo7-<9A=S(TCcE>(tI422-K1CpBXAR@rXR;nT zQL|z2Sl5>B>E9fA+ZuaOrmNzVa+r=he2k6=(i^vX;q4U{6D=3NvO{|3PGuy zK_H)8PaT%Ke&RniA9xUOTkW<4AISHx{#~I8=CC79dJhyI9qmDsag71uiI5&+Dpg&z zvCNmZ{ow$N7^3Fql%4geRx|Vw8JIe9`i&Cqm^#|o&!g}L!``kqOqHX(G`$7iNXF~| zTmz=5Y90`r)e~O-8>V#{6MA}ScGk(A{9UIyQg{w6W`t?lxxO@~%i4K0em)%_Pyz(x)r;3ncV+M*X$6V-BL)p%FNHJN@S7!=1 zOcj21P`k5lCSh8V{tyw3FWtL2tC9~Bk%h&N zhO{d@x$ZWDe^=v68D7VSV@9~1U2y$uNT36FhLBQ?+&do!ROlE-L1tc@M!ju`OG!Y| z3q=4e{bMG1NMMRb8o;s!yUC;-tpK|-w0I*UMpaPe`!seEZiEArNwqf~GZZD(QDfv< zjZRG8EK!D~iQ>6qP+CR0a>{vLp^^8v!tYVMR(3d8T0$u(&8%+qiQ0;-SfU%c(L$OK zPR?fynD;G|aPMn~&l5%le5&oB=*%Dr!tn>UUvaS`3r>VA$DbZQ*#Qd4+LfxO0}Rlv zvI~56wr*Y7P)-y~Xjj^NQP#!c-dta<@HTE2ySo||l>dx3@fl=Tq2aGnfMjc9FvBmA z*F00#rnX%_j7Py4rb6-x8AVw&NH#k*(Z>a!;PYZH=v0S09#8QdE(DLvd}#WxZ{`f9 zKjcNzs{S%;eX1_Mtx_Mp7pUl@Fgpd?z4V@Z(Gb<;Epezkm=wx-HfH4#ORazL?Obr$ zF@YL9+_R?Qp6clmbuN^KnUFy)52apanSO~{fTgWd>&3l;9Q^0DZ{?~qR3e(>RxjPd zsKut_l?*qKktXGh#+&H?PUXhxig24tzDcq$sl0y+efIJrJ|yPN)nrg;-a#N2f(1&u z>NS)8>86R}xVAh}mtWgyFEI}3as&J~Ezp0((PIdSIFt>B0LG^9_a9pz}6 zTkyNv2D$)YU*-)Nh=$QBrV@7zfZQ(Urp0d`8c6hlIaieALw3C>^9)ci$^K^xVKD1i zXgQmEKz1E6f4@H#3HyBfH}i)p>^#}!pRvl|b5ZJk^e5kDrY_%hwz0U3Uqq@~s+&_n z=E7EnoqswicYW5{d~+cw!up^+aoi|xAQWImGJ{l?1)6#zj;Vj-2Gc^BK7!w8cF9Y~ zR5V;ERo2aORIVko_YGZkZL_QoG~@^kG!@iwHDEIP_c673C%dqw40+LvI|KHjDD@F^4^kE3gEKG4oH^j(-z$lA|f&`MnL)w_0_|h%;KF9BL z%i<8tk=*XOttuR^GxI|43vLZA6nWl=va+*UuI;RAkv3UQP0fzl6OqsQz7RbeMfMw~ z{=vo`EPxOXe0>F!+QpgS-Wt5ddBzcoD5p(X%y|(w`zVO&LpUKdvqvY3eA9JmVM!D8 zFfsv_uZBFt)i^%!#Wkf!Rs0NpEIC$ z53b&oS5qUv|L14CI#5Z5ZKf=R6=V^9EeG;hrDOqv0!`aNgevU8t-!rgiOSIiH&ux4 zi*`OCIh|d{#t%2ZzdqJ?L#6oz2|-qNP~7Z?l$Cv2hciuX+7I0^ZU7%HqW66Eo6l43 z_i}gaM&nn`6y&>9d6=W4uZw5QwBXM_L~Pg#XX(E{ZF{z>Xi;~1raY=h*%CHsJhh+= z`anXU{xJSFb#~}ud4425^1+9S;Zyc+lt)^>AgeqDBzw(|XrY0l4?ItSXxP=r-=F<( z*&~NQ$_>hzzGT*$Cg&WJmgRovys=Thzq#8vvA&YTkASAkKn(QUH;c3|w6_v_)Qj5P zmVB*Ru*93^~zvYa?>&0;o?yl7^a(vWSaLMjk^3vjm+R}_-M3Lm{VN@H0JX7(q{ z3}Eu&(7lZ_*2%Dcka2*AM0p&$ZDw%0@2LL`en`xFtjGcoCkphy(Dq^Y3&Vk)9d_i! z;t!P{`t?=j>%$T5hhs`e4eM$HA1_qky__=ZWB^iOId;<&=il`3lGL@7JPvfD?eU6; zt2V7}r10uD6I(2x9R!*SK+^m#@P3I0Q9PgaV(bOM{Gz$TfdqIf{h#sfZAJ- zaiB;+2G@D1-G!-bd%(Nrhdj~;4=L= zy&eM{6gg(z2&8dd1I{U+&=+yvX_h|3Fq9l_-jpQBC#y?NzK5f(pztO4b;!u$_-v%T znfdNQ5y+h@y3d(h)E#?08qJjDIAxtQ6Wx@TW>AbrmTS2^{e$dAzJJ!dP@HyiS?GSB z+jo3+aHJPi{*zA%*Yg$2jtZrg?gHZBnZfFzEIHxQX3MT8(zks>4PN3nM!q9VRAE7t76@jFROg@XL4U5kwUo~YNA zTOpb5W5rydZg`ayQXw9?*Qh**$BL6!LdlW1Vcl8Ej=)7WG@f&6P_XB&O*_O^czIM* z``8s{TK6X>P(Od0i1zl#+gpj-Zg7YQBuGf#U6iXw!(q*;B%OlGt|N{q8GZyz**f=( z+GB4Eay3xC*1}#bqW?*OJ)->RxwA|*rtnf}AYK|abyDVA^19sX+!>s@rz1FVO=#`% zx3#i!Fe_CNpfmR5U$SirHaeRR5<56q+;_Uz?Y0 zB|2446?`s8ifo%AhK^@DU3M_NK#`$%QlNn^nUxb_qaD{S-^gBvPagpFJ!4r7$ zEzmC~g^5%ZUIiaD7Eesqj(~2-ijextn*Y>~Pk1^_-f30VQ%l*S73L!ii-M9(mq$sP ztbsV$#)R4dbCMchWb4CYk7VE64}gC!!#v{lj6bK*iv=Qz~K~IN6yj1|hGYXQW)Ivx@gie=SQM(<|mwUY> z)Z7con#%#MMo$<=&b_|+20%XyhHaz;4==Re<@G$Fb=s&8t(i^7qX0PsH%klUJ4C;18nP<&qR<8uqUa4NJM1#PF>(Vxlf zZ$xxLqarCJZ1Q7RXTgk{#9wFgX|Z!xa9r%EeZ=!$q$&ECmwr|xJqMyJxeT)mTlE5*w>W@R9+4)QFG3xn&u z0vLe&Z<+2rB|1qwW}yY?i{I4D0&otZ*fK-iNZ>fL4pxHk$?9k#4Sn!NKMMmB)4#K| z*AfW#d4}iq_$1Pm&QAO97?@jM3Qduy*8)6CHWaC;tU87afvB^9 zv6gi$Gc4wylY#qp>Ed831qqz?rkKD=7bJ1^QGtLBZ|5o%G?BMEL_CtMC6whTnKkg^ zL!@tfYw5BcL;m+YXinpW8uhau=g&qJH_#$G<<;ic3+DAB+CKP%|L`HWY4Qe1UU4n;U`=mVjL0>kcFk=3O;fLfUy^NGze@E-ayejO z_v@WD@`uwU@D$|nxEvxYcM5(WvOyamAhILwTjw&H*ULFo7)1YjflfH}XMCvv09W@O z)6YISB>s#m?*DL=_2k=aetkLey;NiRud7^P4*jRB59Tgo#=nbWCfrZW@)aoZ^#s?m zV;k;9xa6$Q=mOQ?wKVn#ME;jx0Jt?iM&E=zVm~sw*&C5yrtT42%}|;^Rr{s7N+MOI>i+NmKjcl{ z4YbEH$B2?KxRjTO|Ff_!ihUHzCgT3+nEpEeEeU%z^pr+k+Mi5u{>G3v5hT!+uK0Hk zQ5%j_NAv?+NRv+QHHr%X^tpwT<(eD0NefObcz_mU+ZbV$z@R7#CJe%aiMbl7@bB(v#@VAkcw)97WGHM$ssRw%gIf+Y>pYKqqu2}ub748EMgbR623SX_ zjQ}Hs`^CNdc0pEwMp*r+)J-oV3fHt;_s!_zRNMxaJmq-&`(L)}da)#)EY##bKZr=X zlffuY$s5lOEEd{AprqWRgp+ebO+Bv|DTW^*Ja+@uBos;wmZ&xfQ|^NljIly(0y^D{ zuN<*@%dd5I4@R`Y!|!H;w#s#?LU*bfqjq_Q2hrw{^*S#;7#A5AX^!@is$kUoC6|qV z{^xE&cndcsoZw(9ytXjACcqP%X-Q6K*57x;fL(oQ`OaBo& z^m*jpt~6JM0?;@e&X&{Op@L7BzyZwqXTT+%i4>wmXkkL+~SkJBN^fHtqo2<2I zP1nCdJlS8jgbR@iU1sy_MQu&zeST!3mxVEEH9k1U-09?MYO~l~c+cu1Y1TdOn|Hj0 zrppn#4u`J~s}*QW@tJnb{w{x4K6N|r<*;T*-el<(fnWCTpCz|5xdnQSx@bCEC^a)S zcm1sO>Ub3<43gGNVXOV}8#4gPoTU#oK#?9yJG_+N?aER6o-X;9S#`RxSM#S2nQEaX z?J8nu_E)Nbw}yYDnkY~(VC1ClV+h;6Ha!iL+Wt+EGn8clPZvjSH#{HHmEF5?IVueh zdpQKI!0P{g^AaaN!Wq-AL9!x~A?|%@okw?hhHW9d#EL(o(t&a%p!x;!o1~w)GiZcm z(C#;tz4j+O5TWdbDj+Oc2Eg$9xl>ylL1=}bd+z+)j-0Tsw4B7UQ&n3dUdOirNzl$y zIvl!H?S|&P|aQJ{6Jb6&+>zXk@(V67dipM@)ZKeIi{ooat zpz_alTD3T|h;wAZGN&g__p|f>hB@a~{rpk0`-8f!7iu~88?KT;@QvM06rR|lLPYYO z8HLj3q=7VJX6Ry%$Cue61C-k4gzYec&mRX0n27tI@rCx%;+E~AYP3D3HTmN)#H$*c zM~pRCZ-Izg+3f!+l9AlRA_SXR3iu(u-!dtQ%h(yWMd&TW zcrHFh-1*xO>giblzMgTu31*OW3DbP=HTfdyeO9+%8Aq1XIf7>)KTux&DXZ+}IES4` z1D41fr?W{-Cb)L}@EWh58WYVJ zSzfU6w@T;@!5e#6s0-_92ayUdkNMw%Mv?8`zA7wyVS?i$z)*GvPkfAsG^9}sm%%_; zkvNC{!t7$V^H~Hxf89>aQ0vF2i-x3s@zBv>U*UZ#N@WL-1h+ zRx#=$r~xIJRNY7iK)I-5)K_oKeC)+)TlZcmjlL-0ZdQen)R(DTkhJC_g#3YMNaW|z zxqH9O{O7kgYC-!zpqpB;-Yf+P$bKf&o_eQf+Zxe#{jY)a0DAKdRISmoyY66^aAWG& zq|$0qD~?^$xii%yDFyE~MF{YJ*8y6h0PC>DdSuC(z=iwS_dj)J0=LDXOrH=|3w~ls zWh5-Txo=$=uKio(Galg?szgw6zZSc0P&-B^Wh-*ZkChLwtmVWzZxq+xUr%80gpm*<*aoWM7R9`ziiL-Yj3y!v8U;cz5#0f0bk({6gjw{Qsa zezO_L^*UQgqDpal8xu;Y?9@63fd!8!@mr79*cG76BfTWzG#E);wx{#Jms21au!J*q zi*kB}uAU6sgsiTZqQH%r`fA~o$y-uqIu-Bd&)!wq&Z4mw4@uB}K9(*cMB?5uImKhx z{?6Zt8FBB!(S#ej;kpiAbuMOeW?oE-qJnEns1QXO5# z;8O>e$IqPxM5^;aCM`IS$Jiw3Iws%AW^ima{K+@77dC-8S>|_`@+^SwY zJQz`jRd2IUu*D!*)226-n$$~6KcO|PqMB&Meqlf<<1em>(#Zw2?gwtoUE{fQy-sj!(!F6uq@Fg*52?i`t4?eOBpru zFA&;2XWhzI+(acwCE)~e2|%tQ!K2Lon(72Ua^>OT;1bqd2F}2kcK=Rf?Pc^yQ0Zht1-1fF}xgIAWz#b`OMcMo069 zifz@euXZ;kxD5Pmx0|mGQfbEjw(L`j-W!>%m9-bkLqYX8PiR#1To&J_zGP>l=YBB?p{*OCa|t@Sz? z6!y*Rwg$>tT#mFtuI6bV=j%U$gVh_J>`#jG)tRL00NF3{^bd{D)+xW!ZwK_mtKh$F zk>g<{o4(!0h>>t%r}>I{duzu;?|7jBlrn{_W#-wKlFIU1AUl?QvA{36ng%KJwYW!Z zzyHevP|6qmlrRv8<@)Yxx+IjO>ro&H&mVIT-U6zFAIs}kL-B?z>#@qDPS=<6NPS_M z2f zTw&Ovbh|dt3JmVB7iv+$)w;Xu!F(<}_T4Bslwxr`$j5vXK>`)9e=6Vu>tv!gA5MuW z(?@x97JM+O71hDR`-J59TKTrm6Lq?xH&#SbA{n5jZw7ypGGD!{MUXwZ(g*Xmv&qy8 zpt`K&Z1OBY`}h>w)aH}`R<87#>!Ia5CgNBIgrPZ2n#I>5B~n9A0SgQG+gz-e%`&l{ z7ke-K=;NQxwLje33>eZsoDQC5;EMQDsOXK7vi5fW1y_?%NESKpu54I!<7=`|am|QT z%o;jWE|@>OY*=G>V5jfHw=t$n>FL_UdC?@gabmt5aaRmwm?G-)ih|1`_l)oVy@Pp8 zrPGMP2hMHLqL74(4CvL4aO3bzyj>n!bi->h7>-ndt0bd+=te#QQG^~~+xX4b&PT`k z{z@q-uX;AOh4sQ-($C8zjp}R7hxr?$iMU*EDo781OUlECr z1FiK8+Wx4<8fe*44SeV%WVj&aUC~q%B^yn=Kcn`$-I_3st~1^qzQr{UK^_QKwQ`D( zHSF^2(O7ODaeK%1b!(R-*tar0s};+lDF2Cac^~?#Pf8|4YS{{UHQ^&Z!Ohh zdr~Y2GnSN;${zV;}zEc=kjY*+MHMW}I)l zJxT;QQsVlku&di?$RS$C(MNkLs$Ok%wN{#LPmh|ZxzQkF-%>1NfAe_a^zdwXV@>N? z8~(&IcZ0vs+808JwMv!!y+jUI4W?})6iGxZh_4J%iNKB+5w3K*zORLJB$hRys&O)A zL~+bb*H`sZk!$bISRRAn3&-o)%UAS-Q8nH$wQ#&B_HZFA1dT}$yRi3rq@}{4^~^Ad zAEQzJwi~Jad{XyqAq`Si8F1?mUHe9Sls$?Lmzg3CMsR$0A`4R8L1~$|ktxRz_C#W=;k;Lf z%g3d1z#-xHD(WH6!+~WBA5Z?2*&V!;+3d7_+Y1-f+R~H)&$&Db^{h_<>uo3^S7%1QuoLWaa&rxB-T(2U4-}&OE+%#hrV#O%Ilab;1 z+i(xU5K@M;HLqv@s~JpHwOfwlx`OzP7@dk&LyJ)ke+x8Z@7)^vGx?m?VdVxPeZvHy zy2nm4vu(llCw>DC46|)gdaPeqT`XSiS5C0xE5`r5fxZ;Pyfhe#yBFmCdDLu62U|GWNR5fOLY^%ai3w_8kc z9b_ns*=fB@j#BUYjIQ&S;(AVbmzlWqO3SR|Cq0E|C973R!!}^lUC=K-o^LhbHIl6G zpq}+Qeq7V2^Ky5m9&>|2Z+^V@bj3@&{IEzjYnmIOFRaa)LSEv2Smj!307qJBrU=!8 z_QfW}^`ZTTfMa^CbUsUs8j~Gf%mT3uljr*VK#D$+8*z+YUqB+UVu4%#wBxjNB~McUUyL5s7t?iP z&8t-m+D7P$OG1A3rSXGt50STAJK@Mk=M&&FQ~&Di8S z+ug6_x;curnnRh~LR%H>V^)9xo{${5=`-^@BvF=cQ92XAPr#93-bEcT;+~XBTN8 zqQRQ-WeF0c@Fku5MQ_hbi2y6N(`5mGloZ}3j2Ub6iV<>w4kx^00qJVwW5~F|ytK%P zNV@q|qI)>^&da>LzMwt0G${4IWPjzz(I{H-YSOQsZB&als8i}>NXw)&)XN^qVzvBT z(Y@^@E(XBTujatH2h~#j)X6POZPZTOVz-z%Q+he)CxE(iPTr(Ax;*CAsd&P87U>ww zH01{~dBrC6%;#kOj=s<3p*z^Lal_|{AP!~%tCR#QoQkJ70bBj5`Jb%YGagoc;7EH! ziFHWvJXj(xyKVGCrEaqPT|Lp-L5!qOj@%2-9TFYkLU71>3HbHjKVvW?xf%Q{n@Z}G zn7V=fX$K258F9M*S);yT`3?keA+8cZn}B#>tp;oK1cF21igaT)*x~mQ5F4=D$pxJF zc*U&h&A$1?63$aq=Ldhvdk;Ejo;$6kSOSHjZ_ihyL0#Leoycz4iL8)NZXD{VZMpA% zc7h_YxU!*7#E@;%t%VU~%}rn|6|Qwy2zf;x$o|&G#17)i0u9%gZQR`$WRfSOVJUDw zGzG$;Ldy~*M-jxDNF3Vfto1K4vYTF58RYohvYk$uX`tx2RU#l`CBQpH%Evle9&YgI ze1DS)Y$J2jMf(N|9>@hmk8YDUO86*P#h;>Ac$Ayq-V&Y2k6dWP(GgR#d=-zG8mv$Y zS%AB^3eb9#b+GdYUC1AkORE8}M{CAo2PCFwMA3^JjH)Z&y3&dyK{)@T^8b^l#ed{r zZ-VuYF#!ebqu3pf8Aj`9b-?#I*uMvs73f1oJ5NE!nm0nCbB_?wq7O)5KZ9zhfEsSo zC7=S=pyc{;shbSYS`Uk~%XRJSLPA4ATu4J-DMd^?A7B=22GmyQd5V}%&CW3e3Ae)v za|ErR@)+_KBoTC$!?sQBb&zZ;Yx0KC6aPrUNRFS|JJt5aoLU6vYru1K;>{Co+V z-g_-2p~hM|fTFRQykU!SK$%%8LVF<6YZ;Ll425L7{!0Lh zQRPQ}12T@qgR65p_xS*U6c>ZJ3w=+!7@GbrphM}EXaDs!`_WXlGZFnutqDDch*W@_ zU;6N20et6wQ5aPJyJjc+1v)rpU!o;~_RQb6%s+mRfCi8W2jMf5#@uc8&juMA3k-a5 zuU;$1^?`+A$;`MZpbsy9r#!6ha=8WZwP_TT!8F7ofNEj19>cOx)*FKb4I!G)v2)Cb z-r1ZPG4S2?x=K7?wfUh`q|H4nyVq79Dnk6TfZ!rd|w4^M^ zkeYDY|LH7skK{oxAEZf(9?su&-v^gQ62fSC!#4nI#8S3FInf~39IUhaNoSC2ZcLgt)uSihw%Hmis=cC-)Ogy@)bJRNddEz zaQOK^TS&*B47m0y)=0dvP$XL$QQtO60!S@8id}Q?JFCs z_T^9e|4~gMZ?t^@g!daQx;TylPsw|XdBl><6tC9_v+w(czOvL}?0q}a#Wdb~wtg#s z5x3glx>CEGWiw7wqd+~+bqv7-M-Wr}25eA1*ii%5ug_icxmI)}9QF%3I^A3haePIR zN)SvSmHcd!K!VR=lol8W*Ko!Zx!TRok3PQN%^BWX9ztqz$#Cc}Z&uGnQ|YGsNM{(# zlnI;O;Y;ksdd+Q4X>URbi8W}|P49LFm0+1Tz6mR}49V`gGYfZfx~K9d3AyVqjT8@? znBoSWAzkl^YN9*J@%qhnTIDPXes?-IGJ+NkFON%$PtWtxI`XB0s3#9Bp1xDZ;h3i!R^_}d5Qc(Iy{WT2~Sbh#)Q1+h0j(y$xHKKK0#43S+~ z>cARcCBbyrp*q*g=KptV!5X1IxmKfRtw)4UYU3}#m~{Nxs$@2TjZY+T8R-#1l9G&! z@1y%MlB>I`i<<}Q>U{184=;CG=~0Iop{ec=c_OjoY}V9Qd*0CDlU=;umF$C2J!$b< zZzc<|d-`lr2v|)G1@FIP`fzoj4FWYM0iB4(L;ihpK~E-<3-Jrx9+m~Gwh7g#RB*Ou zVPpG^N9Mc1Xxa<*)=jItkiOgbNgP&caOPf$Zn`$WSg(5WS=Z@OK)R;3acqU}6%#8c z{4Nl?3brF0n@oKuqsD7*8eWjQm)G6@ri9DEH7rK1Juw`Gb*I5OP)_Dz(iYH&U;_lnZZduk)-D$N216)BFZP#S8q%K{ufFm%dz}W2J=jH4lQrOWJrq+@H>t3l)hN=d z6$hW=@Xv(j`f#%A?_PKmoL3~Uk;*sCwC!zH`(NWvN|wo*rOrJ`v+1%2Aq8qK*v!F zBtCZMbT(H9{yqeP-Z-5%jRHocPVBa+?ggtXdjYwE9_IY;>}=D^{dfikbz$ECHaZRo z%jE+Yks0IKS?5JqT|74@FbJR_&6~It{{DZ-vHq>}Kk!#z3Y=!jIZ2OtmT`wH4&wPrOow|}PE9QnT<`gY-U*rT z0@8b-e3mf)L*)b=xH1S1O#^^S0-4adb~P6wS_C#USPc=`XA&W%Q_pDTgG|Jlov1`UP%iJ&Kq0RCZlyuWwA!@QsRK-JeJR$*B5e8 zHyjEUjn*0{5j@aqOkQjrAvHHh<(8LWtcr;}7E+aUiXks>+t0Na&fbOc1}3e(e|SQ* z_nhZ5KE!{_AI}G2JFAU^TgLA#Njw>1OzA1cA*2iav$+1z}^%JvOatIKe41&Q_Keg^iP&YmL7)_o5*LnJg5wY@qyzl|^{a`&n z31RS}#LV;UFqb$!O})=+M>$^S%$Q#?9+cm9$`*G@HHDr3z*B_eA^JkHH*Xh?4Cg%8 zGaCf!WHZICX;)Ix)`eD+Q!Ivjjl9mM$!@W0Lc0NZC`w{wi$UX|dCckQ5d?MffjG84 zg7ovNO13-XVv*c%C4!;5{7P@{tn#T?S50^jSx+3#+JbyRb|(xem0(|M2cqn%(y+~IapRVc7+ln$!~AQEBLcbQSI`%}hQ{QFsT>-Amf#jGe% zq(+gBTW8s4&YHO!*qyIX{5w)5l_1<$Xx$Rj`MCe1-pC5=cqlQMoro)w*`~hWH*Uk7 zU=3+}ozb%LUP@s&nyD9n>S4#nfeO2#e^^`@_9oUkXyOr0n_JN6(c->4^Eem=r;}y> zz^0S~yN8Hr++zQ|)>%V{_{m>Nl~hVSt=()s?xL=bo+ElDt8Jski>6lHIrz+D3PCM~ zBUIjeqigN#h>UQC!(lW2WOU}Y_WuA@6?EvT`D$SNu^GWM*{Zz$31#hmuhC@G`tWcY zrOjvYXZ&!2YnXpF9F`^P(Qw%5v2VS9#I)lm(_}j`5Ag$u#H)1I;)WclR;yw;(OwDM zezJ@Icn9>EEpIPQc8f#YYRq|rJtw8V`{-70k9S1WH|#g4hf!iiAID3(by}qzqjiyB z^mL5#h27k3%wQ8+O)}}UBM=Sbb$Rz|WqZcspqEQGCEfkRXnmuAVGM9c_Nedn!WT-Y zotC>6(>0~;2WuH1k$CdF9;F%wfdc}5cI&Fx6LiJzQXGfnw|&Au#nJe`n$4h3fQZ1` zW199+V31&7@xOukd{6U}BlG9P#09VDw5$xNJd4S)_kD?#+G1A%?dNWFS7(+HHYmQ1 z^Es{=JGq$5ETV17IE@o~ywAMv`k!v0QIk`>gY>}9>}5kmXN;fmqtSujUS$bbd(OUyvjuVNS%7HgK3mHYnBYAR<1$VelmNSUIADnY@nzsOXj+WK(Uf-v^Hqg$JGE^R z36#JjYI;4~~B_WV%*YRLuBri(XAoxui8mxZaK<1nsZEc>7HCNAi17Z!o)(teE z*aSFitj52g2m&}jk=u6H%Y;}pm<{JfU$dJebsX#iKxjkg z*-;#4-5R~qmV4v*RJVm|$({j-qUAbVNwT#7#SUEBF8UNyQWWLX=xY!w;nMoiPeM{; zw~Y*HfL(?Q-N1&0LG)_APrd#ByB=D8M2j|A;Hyf+5qdd9t9zzP>#3@?sl0IKF{?^n z@Plm5-8>r1keVSDM#6#oO2BJJq6}me!$)%2fNAA|=N@Gk<-o{Xvm#L9dW-^NV~zrn zj59Bpw4^64%W=0|v4Q9q5KfOGWopcDg18WCeam-|9~Kthw=&r9gC=s*40xF95Bm7` zEA18)FrMwMn=^U&>RHodYipF1u6AEsejZu0hpyceL3Bh&{k3h^TKN6~`j`S(CA(NC zOg^zraX3DVF&9|I)kgYX6xci2D8oSDCR_Y|yE0lm*Mui!F`Q##91iE7gFW6uXiYq%_ZXfmQ-l zZD?EXHJ}h(2Y{n7khrjV^|jeHvVS7UrmovVtjbR68tuNx5%*Mk7|S+mNB)y-HZq0U zfJ}E;PHeu7|HtG7zlRJzNXQ5VG6V$}e&pxQyIiPe_XHx7K-{gV{4EOKAS5~D88ORM z9SN*@I_6KRuS+ML=MJIR^*k`{y47tKHv!$EZz8|)n7Z=gHPrFZM&r2#!T^OTZUW&D z`P=!QLPF-_NoZ+rn%?RRG5HGjg@}5m(Y!dB^ZVqj0yI%0-?*QH=qm{3q|MbZYcxyF z*Ta1dh>3z(m$?VZ%v3FAgcuRB5uv;N&wJ1F?vVAq4W@1h-IA*qw4$4A8Gzm5X0|-u z*8g4W5Cw1{A!CUE93YTL1GWZPIw9s-Bgb3#Anv2xWA&lV%{v`6vq4GxT}x>VK%?Kj`Vw3{-JmZyC8z1N(6s?i|&*wQy~wEe3a)bS;vG18FjHIjX^;G z`<{ePX_Kx|lqfLYGaJukRd%Cl8RX@=41eB`yZV}cgbm~W!euC`dD!mpqCnj0g)Lg| z=52xxJi_Ev80=bP(BC6Le)rb(`HO$%;9--6{tX~95%bhKVh=dk`T2~Q`4*hXb8e;b z&4boEDmWByXu@N}$wvi7$rIvhptGnY8iPT({i8(H_g5Yxtf$CRoOO^u|1H7BHXrX) zqg7^_0r;u=Ky6JOrmk8k+jPM*JvE!3D3*46G9UVbfGT=v+^&YnAH>go$#&Pvp}Oa-oSHd|B`M86z~xSAA;llZd1&3A#8U+9>i{#_9ap*Ze5FI zvKQ?Hy%>%PW2G$MMsvp z*I5M?0h>Plr%Li)Av_Oii9x!`JjuluQHVG(DXgC{En2FJ4JegNs9IX@C{r`sx!!)| z=TiZb=QfQ-5{)a}v@j)-TK!cuI_e5$?id>9v3&Ki%EP=5BWHjq|Z9PC7 zNiF=MV`_z}FS&M5*ic>n*4WV&H}pHB2GFa(4>#DLrX*v?oEn8xpq{6>a_N2?0)+*H zQZ|ux#Va*1xDqiEKIo}(B?LF>WUXBzn?eAHPGTal51@JSchoa^bibKTZRt z!}N=@mbp7gY>ZFOu#8#9a$+02*q))AxO8_#^5Dyu@^ z`-KA##3+*Y)-?#MOeS7nOKz|&$G=Kl9Z18t8=Oi}pNf&;UyFfwr@T&jk3TzT;0 zUd*Pv)8W`U5Y^E2rHLt%u|mg@a1TDm_u;qig{uQ|;C6_I*LjEVCuulYCflRoR-(Wh zZbJ)*h(A?k)nE!(VCuXTYA3mjEFg6nl9At-7L-pBdR&`fv6`Bc9rIWnKxC-;7%+ff z<#klHQ>`VPM5cE$Lyqsvg2g{09%)twN7krVK`c3h2TA<(h+NRHs~y}7f=2CbrUr&u zIoZ;HQPu)&{hw@Nk@A>W5={@3*J-b?Xl%pQ;gz7rb;^UU-{8JI3=7roRli_OVYUq< zs}Qi=1!4dMyd?voVTb6TgDIi1JdWGcNX+nDGrL~wfv8vZ^Zpt=FhY(A+oBWpbGt1N zS__yBu(x9n1TPnE&(wQhLi*9O{aM?Gg@H>UO>9@9_ikN#Q0XY7)emAFG~qux{(H6z zb|r=pY#P5}@cqOx3Dkd`8Evs#U%>Ak2#q=h-CfE0efzV-*!_rKBr?GSqJ?`Ie2j<} z9wV&pw|)ai@iWr>Pz}h9{{BpEQ)_<99$#DXkaCQU_s%=2B*VuY&sx5HbDeQDDJw_^ z9-B+43u2&0?>qh~C~w>NAnRx=7NA*W)qUOP1qm@IYvpk=yxPvnHg0FL)oqnAdxe zKAK>`&ZMsM(T`c;I9u!M2$uC34P*<^lxJLeoU2 zpR$=;Gtn`R?=BJThDGHmu&L~l(c*p^E64&+Y|Q27qPn1Ti^o9ObcHE5`~Xo*p|A8bsU&?3sv6Nv%Dx z+b0aI9!%^WAAr%y|6d-te=lIs_^?Km5GlBf4&X;~=cQPELy1zsG65~FF*O*ss=3s4 z?wr%32;z6R-<6WCjzlk5*YUV>{x-O@7q3OT;n%1eye@`>4{;BQ%yb%^xgs;TZ_A$dG1p*d3(!E8X(eob3}Rh zvZz+9$?6W1$GG;p+PqT$fht_2s>5`gH^-H*3*ygDlWZI9lriTU1trr1fJ&MK8A{l= z$N$|GlkDy=k$x8UC*zvG@mU&v8x`0Q#gJwr(e8=E6AFM?2-Q`4*~RSn$is+}XCh7; zJnLR_SJ`x|G6>R@v22s|!*3VB7$3_Ez{PommB0Q<1(fy{PIoDHDNt&v?go=wZ{ zOG@Cs;F0n(`B;%j>)q`JTgIqOAHu@?Bd`so&Yks z42hhvo;X$arO&Rcflv`XxLMCmwuHOlW28z3IjA2on@Md4%`qME3a^ll6*KiHhCjce zWhS{1Xq_M`(qk=myd7dw^YFa+L#^Kg&9D3YT%CdY(6OIY)|Py3d-&i(FaWU4a62W} zFh})I%{*uU>M|6VIbLRbJ~j;pf{QGtz`edyx}r!rGSuPlIi&mWXu;KSJqFPdppG)j zSu1H0=#YEC%k|uMQ9MTv;d;)h2G5JEFjG8&tKMpj-3M++;OeRW34i+babvoq_^j(< zL>&+`&kXq8A<_UF#CT1+h>y#$4BuyUU!=^y#OTIZ_2@=mH40-$B&-B7qV6IGG%AMU zH+42Q39S$MSh!wP4aINjTJ9hO}5jDi?ThPX1QO0swL%mzdQM{7O+@99E3zY!wmZk%M zXD6LJ@G%ZP6LCHJFc0Fok#J}>Q@fVf<`SQwikqBx(u?-Zm84hzj1F5@Ken82`JsNZ zm4jbz$B6cCDP>Mb#rq$kW<93WCA{A6Tq#<2Us!1s-RB^M<;NVfQ}RY#xnLfd67W3H z@?;F48h75R41qfCED0_R%1~~N2Tru2UWoG9(r?mL*P^eU+If93HaT`(ecG?9nb~;3gH96d@Gz9FzoZir)d3nG zOypfZrt|H3ArM3~ z6N>|con^7f#4wH$5B)>onUXT6evNetC3v|$SLGV_lRp*zh<_kfw^g!4(%gZm7Zm4= z&HXeMTKxa$dh4ht!+-6U6loL(6r==+p%g{B1f+Y04k;;V=>`Sq29+*>p`;rGq&o(r zTbiLe&dqP1{jT+%v-cmYHGi^Zp69;5ab2J5DrWPy`f`p$eQ)Qv?xrIF|`RSH!n+T^={8Obw zg-oap7F@*1{3`p$c#R@l$s&uZeJc+Zy$Mw-th%gq+L_xu)#5||RRG={ON%~B{OIgg z=*M*QH{t!G4`K4XsQmN{%V6SyuXIqpv+IoU{-9}~vA5iv$^Y@fFr1@~xgGZ_XLEKb z|KYUE#xJ=iTDmAVzc7{M!-YD3uZf)d}a=#b#pYs+_ zJo-9S#ZDXNx$j52lrr>N z+I&)1YygLgH2xY`S5{@`PgIR1dj3f}wzX~{u#AGYs`RE>@ahUYoSbM^T9=0Og{R@{;kyh_3=o<&{iIlhL<~L7;sb1*Qiup$Vb% zt~QaGLaNaDjDDlC$b&k;XKT#r64@zPmU+0GZ{t7#xSPMx62w33N>RV|^%05A{1==aMxjGAm=8;Rj ze^Ey3=9m^B(XG^Sl56tV=Pj(#B6i#R-{n0*)v*jrq_qV;6zh&5CGh=E(+iU*uaztL zq&LIUC0g#gwJ<6dUBsM+ZT*J<-R(x<$;LrI6<+lc=@yVGpHY@&pndjXzNrYQLJZ3Mx_Yn&Nym~tarnS>{IZ7QW_}-dA{3iQinLhLKspHE?77!g)0KkC%9R~@}>*N+7C1kh86bJY>m}ykEGlHlO=A`o754xFwCZMns}NxW!eKKBv3`Q2ZfMW=&}yn(VXNCt--r~G1e@3BDqt+6U5g&7pt7F~)U ziul<>5ZZ06?%ta8Uc4pW?}b|xCg{m+j9FrTx_$*Ox6U&FoUXuwlwoUMC{Mk-q+))? z+vh#`uIF_K`+rWg%F0e^{epyWRweq%KF6l1)mlxne;b7ex6So4L8iQSsTXr^URb|@ zshEbQKTCy$YV)|+mk%H&q}I6Ibu^pHn)2=C4e ztkI0tL_z8A59r@LRkfotJB!*qO?o!G(rtLhUi-_)Falirpt^RjWyLgt3c(fbm)?;D zayv^EO~lrPRTaPk_ekjGj2~XeeEO+Anx3@Mg_Ce*x$zG#iHZsirvUF?@)*+}arDaO z_riHI&;=myro!}xX46&7JgmkSD9v3VWw9^vl!A}N9v3y=XfIix%Cl>gbxaTLDU7_Lk5Yu$ zt5`@lsn#sG1jX{*ho-&0a~a$wLkh*+GviQJE9lMknE)Hdy5D9|dGK%I@US`CV=(Z2qx=nfBm`lP5H z6><7c?jXV{#ARDe&*ufArRf6Np|ciE7<_~c8Hq7;zC&VjG+#LgT_25JbxUG_lq42@ zhhl?f@3~6`$-9tG0kSItz!SNTb1QOY;L}FP0|pUOyW;QfB31?#PpbtUmMJM?uZ-Q% zCH>ob5taqJ(D^+K2#cB?gwmzk$v-4jji*slukr^(C2_ik@4@k8k9siijYLky=Q(Ip8%QrX5y0t_Oyf#^X|82-yf1b`9bcjjjJua~!6yrwn6@2M8U3jWbd^82S*L7V^!8MRyB&Y;0aPx{?7Hf zR|y%)+2J;wu)pXHHhhO_++Fi6fFxDST3pkrGZ*S7Qg{EiQmXLf_A8C<<2$M|amrmJ zZ*lEE3MS|OI+BP@#z>U}Ga`jAx(G?%DesFMH`C^=3m{iQa^ZFEj`4_bSgyfI9O8Ae z?iZ!v7*?>$%B8+6E2c=(c?+jZ$(~?*Jxp$;v(-4+KX$P+QjFO2Q`+RUfY{ssT-_ts zYw?naef{`C&$^y5%dNmq34BHIAt~Cd6L<(ad-c>}XjE<#^k4oW0#TM@=>7%|D#G!1 zp6$BfU1Y7HkK4ft{8~0K&nv4lr2&se=^qd>09|anq3a#xfjV&LWp;m}=vPW%-)F>m zj7u&AnAw>ujqz&UzdRAik5i?ZlPms7epM;@@`BziXN}b1w=EIyknT)RYy?tPxsfM` zj@tjo0{j(3>lY_lz*f<&VtDXj9^u(5xoNNyg5r1uv|H$tD!uPd(}!Jp+~?yP0XA{?xH; z-vF6G>x)4XfLBEY9N~3mOA@G>8y$l4G8U39L<$h0>&=yhtL4acQ*fE&oTh%?r7C~9 zdmSXYBJ}5lfPvl}95XH<9CKU!9F`v`!nLL6t$4sM|H z3!V@TJ;_{yi_hBnr-41m<6K4{eMc2IS@NN7mV9Dx9woKY&w#Y`!QBLm2Xh@M1hacg zwGpCCM$ovD&x%lG?MhbNJZD1cPjL9+T5ln6+yM_T7Xk52H+FB*5f<*Vcm-)R;cxW* zB$BX71;K^Zu|5?cAjLxySMiuRv7vpe86*^$%LJIiQ zf7r(-ADH+P?0AIKI6OX?J_O@7wb!54U!&IYHz#bPEhpx(VxH#!g>G{8o#J5DNyDG= zq3c=-?p#xc<#I>BL<(sO$lRJ+yAX0hrKUTqn*T~I)UJHkX#Ckkb_l7#Mqv6ulc?q$!-S9}>qn%w`TdS?N*;?H?Ya8b z+L8NnuLzh31ks~b(|{}!j62$?)CNpzcJ31e@+y9RShSWFsaB#dGv^jI%wElQI54da z5Iae8-cE;A`$9J>&oD&X-uBSt37@Z!p1Mp{T4@08EOps?ps}I8M@W$kQqid@)9bb( zrsHz$&e&CT3lL7bMa4KONHPt6F?)5?;LIt(ewLC-Qqr8ri`Q(d^dKa_j8%Ih zz^&h@mlX-GG&JOHOwXP=W!s|2nK0MYq_X9QmnaC*lfRLn$(B{%RO#*shaGO~TRI-% zSbFqI@#m;hLO3;~xCd+7J$Bk&9GW_;M5SjMUbwk&YRn4uCnvvEt?ZxfZlt!k_|N!a zjoTAO2sJ>+GV!R@#t_19&5Kngtrpwtpl9Ujv7BjBDA_>I>upUSt9jt--d^Ud18? z-%IN$O<=l@qhG*&_)PbOm*_9LtULl^Jx^1;ru+0!kI==qr+~m|uJkL2uLxFj5Im!8 zEX7cA6@RC~Em_%*fFJz~83tnHtzf35l!|;6dXO<5zY>Vu(J56QLK7GU*f0IlyYM-| zLjVGJ1c(Xa0^>e)(3JXJ?ZQk+QXqmZqxN9a2ad-)OZY~`DA?WQfI0pcnD~ts=NvFM zyzegF3!)qNPMWip8akoC^Be$jjF9=Gj6w%p9~8bn+%i#G_iTy&R--}6D6!Rw4}csx zX~M4yq)tH2m$~BLOL2YG;Gh)=*!~;?2^{^LPw9x)m;xw+5ieg3&$&f{+1I7f%el+* zq3O1Li7SQV$3?(YV0Yc-+GAL0DFgU&?ziDrsVZ2OJ;=z3wiQ5$M;=PMbe0{PyFBT4hh-R`DC*E|XVQ zc_LZ9?Las{?o|$bvs(!5FHx=>*ANI6i_Clop~$oBEf}k~486-lD3@_hBep8ocY8io zVJssg(hxnDf|Lh%)67F#=B;cbJQCIe7s}(E$Tb*OYVVmC4*n8Si&qO}FaESvCH)-% z)!cp*Upy|)J$>Bc{^UERKOU}=&0O!;3nf-azgjEcc!Ci|S?*~Z%_KChf%lrliT4_p zrvbHnahh)!-y47XClg`6$yaPqcbv|}h54VxG21@``p8{^Hkd>eP@|Lqp>*SG!jq?0 z%gSJ9S|W!`Phw<-3+@jQkse%xLQ#}3RleZo))~4Jcoe`{wZMEUnO`%67CoBd{$u})*w@i%uQ!M}-O@7d5jxkNAFjyk zCzMJNi!{JNJ}cboa#v+%3YUME_X55BYc6I-2lY%Wi;&;v{UC-5dj`TkxYjY!7yE6=wy*2id0ml!Y%jSKY zEaSKf&0ZCuD+$raOLOu^tXUPmbpH}ac|wcjK8ZoHt# z2Gdn2cI{B0vElm;0b_%v=@Io58+LGkl8;@}OgIbWN2%OpATTwTlyvOKt$#(&k?X%?N5eoMg`9FrPuCnUBmLXS}6I(Jf4nCC@MZkUP83_-4W2 z&{Q9YA^)pQ1S(f3t-1d|lrVLc5;iFkCZ){{dCF#d_)o#+@$B}bBZM@TIW>=*D*S%P zXv}lgMyE*NZC|c1?o{MadUxVS4JQ!K#@kl*?@3haVp~a?ClH#sIrq0*?RgILn!^Ay zqkn!VS80WV$Q8cfRYx@(Dm4JOL980=dhfV;-srWfT^MDI)bZG;k{T6R6io{k11|~V z)5G(_+|8|$350RyBPQ{XM=~Q)Gz#N;i_NanLVY?nCa60dN}JjL@s>gl1~=(!)k60UDoGqupe3M0vmq_%QJ{}7hI z7_N_`hJ2o9sEhs1jLm76438g6?c|;2t)EjuLh)~5qS=o-{c_P$KdB;O!;_WVm4QG= zYu@{^fZ|OUe6dvk=%7$rz;As2YFP6AQ3NjIS%B#}3h3#GBtcyJsGaqRts;r;mhG-n z;{jRzw;b^+^B7Prc`NF_UZ?o_E>Sqc;do$F=6O%S#}sEv*WoNn?|xRBi@o0NWB^-7 z=2TbPBJpax)tWwE7+G5h3Dy)Vij;h&NQMh);6efO@~Jw~i=GVN+t0OeIYBB`Wy z_=9IRQT;ac|L*?2CHRDOrQC%c!PAF4LWr*xe~QG2V(g>)jAWk!Z_+CNxrtkMJ^)MW zmgHng#yTwp?*qV^Sk}9CkF5`>d%bw%pf8S0Nb9$Fn9{-CRAHbNjKx?wC|f#o~T_oHMs| z2!g8KRX=^`dJ?~S-b+LTO?hKIwZ@=R?<`Yn3;Tk{VmU9|&Fw4o*c3*BjkA@5z!m$# z%paF-B+an3iDYdNGDHPZKAd^c^5@U*Hy6nIw?AvjVf1Mq_#5Py=Db_R!9) zXsHkDks{vv!jHloQ0NH1yr0HNd^SggQxz;=X#Zm&WfazU>s@%tKP~oU{vE1@7&w7Z z&Zx$>-1Yxf zF~w*QvSzv~6oA|=RL~A?7I4bU4ZW)-L}cJq;vw9#THTkX-Q7K=p19j+(sQG9Wlf>0 z6Y<3AeQ@zBqCaA57Z44Ii$geBu>-<=I(0WV!f>s(uj`$dllPnrA}zNzWQC#{G?;_4G$WX}(eN?Kgrf{ZLBDMO)ug>DkS7yvHLKhNO8G+dN?Wz#GBdYY18OAm zwBpXD$b0dtF!}#9gC-Kn`^3@2Yez>JV*KLSj}JP6j>Dvj+pzsDJ0sWuOF3h=(d7#% zzhL9;cDY7@Tw=2x<|Y6u0Um$;9L_A3?X`5~-P6>?2+jO^0#aB9k2o?1ABkVA`?U{V z?J8fc<0Am9zOJWuI4kQIKpZY^E*}F_QX%LG$B*L<0&+Y1Q{>ckMIl6fe?>L#ycbp0 zv3keII;Z<_zMsVO>HNPjvIGmuUskvud)o4if(SOslrua6mTWDdwSdi3y*W>v2IC5u z6alH)<+HuE0^og$ZM>%0p{N@T#Z~EylKqVC-<7C7JghPW_N_ch4N!G?m2e7{&2jc& zI3DOcGdGcHy4;TsO<@G*Ao={_t$m@E%lhlDOsQo-4Xgn0>$LULl}2-^;rgpHA+j?J z7b{*~Fqrdb!^Z-9%S-SUMiw4QXSi<{$_Px9Q+Uv)bkh zPro`(VacZr7;5I#(-$ua?Up;58&7|epsrBLIkK=(h2uG#<+Ipcu84R9gO7ml)t>k< zYJT1;4@=)?yR>sGTZR<%4K_mMGP4*62kNYBY@uutm~M4s;70&+fWNxTIu1wGc*C$m z(+~%Q0w2zYghk4VWG=uk-qBKR_Dq)>`B7T}PMMtDWSp14I#+U#UIAn zUo!mrgKmsb(K(3M_FOScdmQP&)4fT9Z~e10m%*?%|e8 zhZ@lyx~X*(xU*$Zu1X-#4wuL>Q`$U1N+JG^ z>C-SmLzDAzGpSl{^)8lF&l-p4NG(``N^NC7q4>(R|2HE|*3OUr`eZJ4s=ou9V21!s zDY)_#=8lq5ei`#J`&(Xoa`%;iBiP2w()#yLc&x-^Rgu!-?@fk)TVu1yEU8@X{rOfc z$S+0$BhP6QhQz!@Zpu%q;@WqKx(>kuK>Y<2BZs@TAhe7A@eQY48^%3?#|x=TQp5r` zW{$H$F}IY}^X_UVxQKLf``ImoM~hIgpY!Y?3a=~d#V7!-C#FBUOW3Q<ZB@ zpmnLR#2G!YMgt)Q%+A-&ZP%^+iIC_<|ILMT9!U6bL1j}1;b9-ylkfcynA1V(V(MQF zZd)q&fR0-HiBx+7H;hxsG37hYP$YEb>N~|GZ61j@0m@4CS{n)VX+orqsKXU4mMO9h zC7R2(X4ificg;7i_4obvBqOp77wRnqCsc@dK=G9&u<5~9sBq0_x?qv1$?LPudXH0Q z+jD%d;|`7EDH&_N<+7e@Ku#}daz1Q$|5djm@Z?-f@{xZ??r!wT#bu4#;XS}aktd{h zt!3L(J2E|wg*toMl$*er$IEyVOL%=ho({_>1WJ=l1XuAyX-Hik$c0hxC^|#r4*yBk zTDV=elP|-Xy~{{nCF;hkdvz(n>YXoPb)OmlDpfsKQO0tVaZ(16<{jb`-WJVDFbK3A zniIb%YaLOI1n6OhC#)Sp!BN!m1Q3ZDWfjT2>+Q*MV2^m_v_rAnpOhIy^Uita=wGOV zpJv^Dhp6%1w2kp!vx^*(%TU*(S3x&ArC|8c53c7gKtUDnyp!p^wXpHP1-S6)nQl^i zykn~8yKM|~DrMET#&5BdIQ#O_jV5~&BsT^Gmi1`z2U22_lZ_}b;D&zQa8Kofg5JE} zI^`e20d0IMrQ|!fkhPsQw~@CuJO*+JeV?9zs=y6Q;J%(x>x1#G8Z4)W!t3T-7FA=) ze&6KJ5J-C8fyM)S*}q1!fgy11to?yIk2JVBqOP4w1U_7`zmU(hU+z%!e-;e<|28Py zDn}m?AafCxwsD96)BgE-YGPG@CAlNSK$TdyM#8ig?Da-#9m`xk{{`jUejAWwD!)8> z$u9QgOta#vVT?-gh(&J#v#r5NZHaV_u-6^53^5@1s=J%zpO{HF6=l(#-Ox{ZYh|lZj&YTPp8^B9y!N0^o84j z<-2Y^vG`Zfv;Lc!!O3K>N$N?qyom3hfQvu;KMw$dKY<~BJV!%EkUhfc?6<=4BM8qG zk=U0ZL(RwGS+CTTu>&WqN5}F>8;Gx(Hvk8aw^X&p4xz}~a@p1&yBXMgycxfJq4ctW zlm0sQz63%1dzkY6VVSAiZ0vm|#pK+7{g`d5n)9oPxdzH~u6gO1oEqBxA%@m>g4}$^ zc1SRI4|p3-^^*aDZH}8x%Y3in-sq_1a3xSLJ2gcvQo0XB7O?5s?N6By%XdIM&H~Y3z->BivE+ z0aJjKl1QT-Q(AH+*hIeuc@;^gc8z&?HgA*Z7R;UJ3N$h=PUjJ!98-%7z}Lv4s83f) zR9%=w7||k5++L6OvMMMe^r@qP;`sv`{nnBugr2}8ym6Q~^m_Ta{(XxlSn~~T{@woo zfmX~3koy3Y+I&!X;8Z$ z&ImWXYTOgaQc4pBGFIeGK3MJCwA>oJQv?7IZ-!fOh5tk_|6f*xHu_{Bs^YNY-#Tz`sBn48-TgVBi$}UJL@M zoVe6`|5e}a>O?2(j(!Ft?s+ENk08*v%?l{V)64`dv~GYC)Wd)E{`{pldT>n*GuYFL zzWN}K{J~w&FoDsSG+rF;kO?4!MqmJLPr)DO4IiFs!bY^}yVA`GRQs8p^%4;_+w)G( zWr)X+>JJKvbF%#iyhA=aR@yD{+R~74I$S3LcZGUQNQ#h$bG@;cvgx0)tJSh=SSW}a zc&zAhI=eq7{KtH*A%5=i<4>D<>lC2zrEL28SDMS6X(@T_$+5s3`g#c!O)Grs>+b)J1ArK5FU~|DIb}_#-&i{ihTk=%4sMC%b)ooeDde<^vjePqrlDoui+I>v{{g7FHlp z76XNO^T<{wW@e~~kaDTg+1meIxa<&^$_85^JW!@JhW@85yjKU}0mr2xuU&H~%nw$A zfwsSP>-5j)IJJ+U*6B(5#T4 z8#VOqKL>!&^9iYC;6t_bQ>K8ci}+99;b3RJ{F6Y!pYE3r*VtVm#|tMBJad16IvQW_ z$uCQ!CLu*d7a=Dlk4>S60qt9$yj!CRvYvZ+m8&IKR%qPG5D$%CN||S7ngXn$O$8B| zG@mA2kc$-0p0wGB6L2YNYD3$^%Sf&pwO*CRdprhn8Hf(_j)TS{ysy`t$eMB?oe${# zHhN@IY6oj(^N@%4Jb?M!@QrX+iX6i>TJa~$*N5!qwnCH91{8TY!4kF!#8Pa{l}uuz zWDuN}$6ISZG7fM51T{M%>hcP?fPskUe;~}Xf!(nPKLb8o^*+-IAyS24c^T8%{0n4M z&9Hs6Hi|(C@OxId7($w)VBEtR{={dnc{W{M15&1l{VJ|u{M0E@^DvHfHyUSBSg8>L zXi$;<*nU4ZGJ1)_m7^c6=T~Y^S9bzZs!4U>{xa#&za$xi>ht5TSP+`7eBry^66nUp z;k3gH_4&0seh$%RCww93&n~Vm~c|Ps{v>bRikh+Fe;0B zI`l{2#b*+hb_Z28Oo%SMR0p_%Vcga~R#ICN=NOmnXJX@Lf=WEICoW@9walu_doAU) zNn*i!R;?I3gv9myt@S6Qc`TFCLBt|f|F@UJ0xso1IJ5->SUV*7(8n)j%3E|J7wtKNz_A`s_15_s1yK)6PU*M3|+5}g8_1J7+#6p zDM|hu02-Hw+!}c2j)**tk-SW`scwvl0?xFv&gMvHphN_$M1Pq7=1g06<%#X4 zN5xQIitO_XF+yrV*V_Uu%%$+=p!a!s2dsDji$3^53+rZg0~D(N!TsIgO~Q86p5Y(N zh@z)+L_D@VdBhZX5lzxWAFVSme;tMlTKXRs)` zaP?w&J0YMn{~^d{Ew*Y)sp@A*pK#trEMmUDBsu5VO(7yY3?C9aBI$2hKHjaD0Sp#n z|IJ&Fr?JP3;2^-s9}&wwoh$ehr;mjWfk>b~z$YWA|0^nqO?eWC_ZJe)n8_@Qk_1o$ zJch}NS)p4m)W4pxTdarDZwybp2OU!x>R*+=a$!tL;h8_@h$32>JlbeBUwC7q8<(nA z;^|f*xG&M*E|e&rDnFef2M%8K4}%foBrGKE4Db$_&#b_o$y_-&d~kg_y_#gJUXbR} zo3A#!`BOyIOl(nj=~Bq!nbY(nmCGNQlkH5MDOa}q1nyi62lP<@SW&EQ)pN_y5J>*U zy*Fv*rKJsFbsyXRv+mgGO)cyxnlehrW%S2AUwy}SWsruR&#^-~`q{80WvOa|tN*8s zCDlt$38c(Um3EV7>C|Zvm;0xNdoW;ks*HBF_I}j)4-5A?bP_ifPj5}Ly9bn|s)Kc1 znx2=|VcSzq@?>m(nQ6Q#U?S8H_Jafu`wTYu`4Cl41iybjxKHunUqQtqT0GHS6d8v4 za7R#EXhExim+OiR?%G@jXp z%%ZvFx{Yw<+?UigtPry+lM3|~4I*j=$hlM8k^_(kymG>(5o)X#ZL2F-dCuEkW9xS_ z`<2Gtc(QxT_%T!k1z@^7He9c%0Kn^Fo8*bSB+F`Ntl^F-@5aFtZud|AaV)V8qU{Q2 z*)p+X!X;4-tC=~n@sH0pS*ND$KBvG|lT3+3<5bN^iV-OS^xQQq-4bVRV40F z>X#*y482en^dNXg!U)l6|Uth=o z>CY-KMJIsVS@^{%iW3uuXzbNxsV#^=t&mUVRpGQXHHEk@dEscLjH5)lUjwft( zQUPmFOfh-S`VvS6?&bXz@tsOwr(C^6<#r8aP`-Y&IckHPOKkvf&Ir#4Vk4VRy6nQ0E732gFv>;Bg- z_pBi1^b3g8We|>OD)WU3_FzD9zLAEQqQ}GUgU_t2&u{8wSV`DDe6v=38GQ-1hUTP* z;0|bz!E)+z)fJt^ADNY*Y5j*cy=9zM6UB)@bEmhHLo9j@I;uZbysG6trL5O((IL|d zgvrjCwT^3>MhSesNe0>V1?50|EPM6LGu6gADKFm-x1Q79m)11tT;*r`HMdpQp%Z9| z(Ra*;<}R1{YiOf;N=z>&`4jHVZwN0-nn(oY!RSbOi$(42jS!b{2VUhBv{B@#$a3eK z7=QKOi6u=A8`~fYR**lMV(typ+Li9)1N zBU9>LKci-?ga5~AXX(LvwXQGE)qL`5?8k?5 zV!cK2@yHx@Mm0mO&!B@FsdgLiwb^Nd1wXGfi!I&3moGyT zysBT)!9iOl`zQ*}e3>|S0Pf59^x@1WjHCgaJXd;b-r zm>E2wslq8dz)|;238D^(^*nDFY4%Ab+F`u%>`kb&Y?QRTJ(XT;&74*3Tj)Ia=_^M* zGA@N)(N&HS;kExoE?lc(YM%N0=F)uU2UK&2)nyjKB;ZWqysLxb`^C7xL?&{^^VQ;s z8Aivs_n93{`m+VXGe_)Sq4_zyG=IZq&jG|Jb zM5m!fr?_dVphG>|TBoGKW=CZ;VcsV)!MHPKLb&_eS)|(g=Pp0RZx5_r$u{1gJLxCn z&qsdQ?68gn6h<+9uVeVJ?i4-_Ks}a;cyAMi4XC+`e^K;{74O4^{~DiSi~m)i0Y~ z96eX9KOyS=u4b~gfCa8P*C*UieNa8?SD!$b7vMBCLI>kIY?Mr$5$M^Tmml!Z%$>8T#y~vGwD(`$+D%M6 z>sl{hsF_}0a9axKyjC`DMpu6pJyP%DzcPr5e?b42ctIxnMZglUTCOZdy2u@GW~R;8 z?8}qEXhmf#(A~R|>aI_CQ(ieOhwnR#D*SH20=Iou^X(Dqf#?Ptu|YPFX#v0#hX7`KInpf?3*emuaHgY7os0yt7PP=roc|`?Uek zMALIN{1U6aops-$JKrv+veb3y1(ib#z~|68uD$ENxgOqhLX)V#;;&ZX;;3N8m}|Nm zcy8Q@F>iGA(d%bNSOzw?s@^gSjZ}Ij?oF;&{8m6x1<#zTIF-*7zB15VTM=fPmt<8& zSPo}?xd+!Ru)xj>a||ibO~NW)_FZ|oa_fh8%5bi>9r>SA!+Q`P#2PTkikb)lQjuP& z>k{tutDUPHxy11u@Bih@82f?-{f%2BaJ6=%>l>2fd}AdmNAm(dFE-Aqr`dc? z;PJ0_W(w%}zSN);nc zG`k6V6d>ad4?qpfuf1!J z)f1hSg-)OPZpH<^*qSI&^9MpZPNXou-~ zIr*o?lbAi>)9~e@CB9s0Ci;ai5CP{v7T$Z)R7a3Z{~32!wb|RBLZhVNKsJouK*jRu zO*Eim_QtW)=U8b(3P(6?p(&_r{>?(VzM@}z>2^9BaFI;#v`h1v4tTQTkOkzup#NQ2 zXqk05S~qms?mVjOcEeG{_J=0xn;1P{9^}j9GCN6egvyc48>e}1)}?D^5*>xw*Qg!aUZ|T`01X9puT1Hk}!T zdmbt@$bTc@6uDV1uozuh9T1qES{%)bmIBU_#OHE$rYZKyi%~7-V!oF**O$kC_b)C_ zQBn5G-q94?p5^#NR1^h{!B>ZAU7uUOf5F6ES1V4|ark#6JDhp7oe(J!5u=cF(i=!< z#eH?TnrN_6(#+UvU2PpJ=r%LbC}3^hdHl^TtJ!AO=$PMw+XQ_(j0QD=#@L9_h7JSa z0CK?(soieQ%h4g$js+T0Jyb4d{v-A#RKf)5+Buf_;8-^Zua_=R$K{vn`ArwGI+QVb z@fQvDyA;A2f!MMw;OFT*@>$K5|DI_`x077s$jiyQ(Px!pr1KF_H8Y=kI7f5nJzNRr zG5Um0uN-3Z^AERGCP$Z8J9nqm@(gAa#s^&N+r2hoQlIHo@vGuxiUN2+0)JZ3(fOk1 z)zPRDNRz{K@`@Wm-!O>+?UUXV##@bIT`e=^eID}DF;Hk_-V)K@OsW=YzPzSRN&TOS zy#k4xN6P!?aY^=5>ow@7I51y|CJcL5BH#8Pn2qLjwS80TmlhvmpUWMM;4s0Q#VDWe z3l92;*STM!5`?*O?S7%*VN_e-opGBD=xSA%e|aiE#-6Kqj`nDE|4)xq$#dzA5&exR zWKqX+^&NqdWdV*4=veZ@#nK%Se9koKQ!LA!x6d>Ts7lZY%Il(oSBOLHsJK6_JA#y* zwP7286uH=F!yy*ek&_B?;QF@GD! zPqQLy9RnN<8(Y4&V~($|ey#AYx|`^fmJVC$W|=HWBs$9r9Y`k^>QO%LQ|yxWe0140 z_@Q|a9+rt6f#dffUbyVqv!|6>HvbK zor^}ld6m%{GgDKi4ZNVSg_eNlAjRRrw|W;=}Aq4q1NA;I-Pb`u6hMbbr(3Z*pr-dMQTpzoW$|wqzoa zOky;g8G8*xAOk$&>F_jHNn6wQbhTR3`9JcFX`Xihapd}FW9Mk0e&QySi|H8e7H6~? z`DNeOMvO(R)T1(y3Jta&8U{to47{vua+T6x)4~Yhm9&ISBq@X1Pi7X4-EXjBPgU7? z{-SZdXYc~*QM;|3>d5Z(VHJnl>n64$5JRraLu5bo66;9DqE}mXOZN$^;bg8c`nG>X zqY_bK`pb2nmj%YGPQeoqTRj&t9M$EM6h^Zw{Q**rxzqr5{Jbf>@Md9lzt1`KrY-(H z$a9fAn*?Wz)0FR~F-gw`C~sE{DlKa%wu%mr8lBJO$jW|uuB{WiKj>HgmP{rSVRm~X zy|Z3-GKEDoHo!d!{KNbSu_F!V78WBbz9xghI(lmL$f`CaUM5_*0ic26UwfchZKJni zS5a1mN3~y_=7ZG$(0muC*PySv7!+!s9{}6TZUIz7mI!h7|L+F{KW-0Hg&ZRfjfpqe zP8u!v@4Z)-_TnL--JfI2KgupX!cK36w4Y2CH@AgatF9=Z(-FMRY>(uz#te^vo1{fVU^c`o!>`kVi-F?yT^Ow_B8h=q}Xm%<3Yb_}Pv{*Aac~@GbM`!t0Vz zx-&w*_TLZYjS1Tl!J9MZjhXM0D2d|6lWbW2LwX2nYJw$FF+W6QHJVXD@Sp{UqcW0r zG8yxaT2>#Mcf}d@z*6TvNgY@YH3V9q8kR?POZ6Lxx_!|vO|QPLQCj{n%A ztLLV&J9;P7&wpd?AB?H(a~Vk_5B!{}Cub(hud?5@$57sxCeI%ahc9m>@H@>=d;&+`&PTlL-Ome!Yu zv-Qr*KnaYm~k@fmcn&fQUT^(z6yZ@Sip zb+T>euVdJClH)jb)R)`$Q^)qc7-#ft2|!9B1A@JP}TZ)XvuR*8@@6MQMYn7|%7X3rM0+-MNaYUifP=-L0vliAzF&7kIF$_iPjse{Qf6IAhstAHH90fBRRN~RrE_Mfu3Y^CXJ2|oq> zYru~L%Q>D`KmNam@c(&dYfAsLia2e;t_|ZvZE1;$;DSCG?|U9y0`(73qXO-N60@rA zEq46|=`}R~$CMLEe3(c!-JwwaagvK_X1k)~BdHV_ZRndeu(?hB$=V zqC$ga%|POL*7nd>lo?Xu2o^O(gx6$F>aA1)+qs>h=KWgoWyK>SnS z7?sP52sWCW={;WZtDuymAHS_r3;y1_Y}=BQaIM7bL4)SDK9RhKr17cA2LIBNmrHq` z2i@u*oik;sgNXYCs|J${7crkTuo`_B5sUIXgK}_ahxaR<9}YH~&eUMWu#&H>GM%4H z!-JcxGgx1R(lq0Py?)uJ-5D+Kt9_~KZ_VByFCrg+By=77xu+~Fz&&aa>LWkg+Pi{t zH79b|_X@PK64!3OPpLP0(_f<=Y&`#;2P6?eacIvm)4O8dQ&?!boei=qbXlC^v(P?7 z3nOm&IPsg#As;YL6TntU%4<(3^CTt8Va=KW-cVZJ%$w&+eA8(YVLef-=kCn=b`ITy z*Y2wNf#x=+FW!5czx)m>-#m|0vg>e_h71>0sG`}qfq*QPj_BJWF`pe4K#=5MjN4inJXy<2J3!sa(WE;;*^5q^Wp0yXEpM`6jw@&yOaF8n&#BOdiZ)!&L{`R}w zTni-BdG2%d5)NvkIvdA7kb93Yny~o4xb|0C(XRqBbGA;kD~rKgQF-ls7t{=&K8k*? z+ed^&l(NVb0QD1*K`49SGGz?sw`}Bkh6>ye5*%ivS`K?r7%nQD5ds&bV!ji;u`Tna#4^nNavc+YDmr!lGQ`PNOi`P%2SBA7b10Q30M}Yg?F_#y;F?F|PfQh^Mjr+phQE<{|J1k))C_<2BVfx9*xQ0S z~iaaTtmRRCd;?HR2eVTLll#HSKb$+% zdP<63c>!X0~3b zr*fi04y-`j)>7yl_7)fkRqRHFZDjX_taI*Z4!@PQ4|gbf%KJ+JoR79FS~0(_V@D9k zSj(95fw!_?`(=fc8=@w%ezTn#K3FKYIaUm62^ z#G9SW)}dIaFR`@~?>9SRw9xx3o!qZK^fjv%K6aDKr0cG8V!?q`ke#C;?VZFGzVY~O z=7AN9{LT5A?J!lpIsQrR{Ge2|)F^w# z#UQ&cjfNd7%4IVyT?gOZ2))Rh_2bN6_St%-F570w-u>m_w*#ZWq7(MW0k7%OZl@Sruj%i2{_XKAG+Q$ zD(ddtAD*EF0TBslQAAomU}#Vp3F!ue^L)e6Dl93}jf|T=Fca#>nLqAL#v_HSTU45VKXPLeL;qfo^|CKZ>T!FA=p#D%;N%_zS z{DBZc$T%Mrz&Z1OGwb0wm+tAoKD+PCJ}J@8Ogi7C=*yU6Gw=7zdm|lPYMkjJfo-N! z`KEFD#MI@&Gi~q_NuEO?ao=KR$;yvlCkxR@Encmox$a$X00Iz^fTWiQ5@%{5E@BrSItZv0jmNOjE6I+xg-otpU)>3DuDH}@C@jAl(c z*<=1P>soyvKzMbh*-zpYYU=E0u;26eOnG&s`v6+}3kWwIeR5=b3NKNxp$JmdSS_i* zdnWz34^eA}Klp&T!zM+{ugq<`$`qX_(XML&#>s~JMaCp`j@Xv>%0`LkXf-p3EnWJg+XEuj=^3Df{jJ^nR_fF+3o6WS1+zZjF-de+cbqBqUqMD;ce_`J zYAMjxPMpHGDWKcpnNg}+{am!|)wmIQqeQ(4yWDOHE9F_JT`sB1+289B7fcN&CbS8$ zu}aBdrJfeT=t;AO<0Qm#3OzNQk`4)i)*>>s{; z=-S}#;x;un_@%-*7dKfjEZLVXtk<7RaS4;?4=^-PYv}((Rw0R5@4!bR0#goVnSDJ~ z<#jX|$ZndZb+yT?*>>E<+qH?&3z`-tkdyP3?=FE@Bc6_LoRv-`#IcnBWNvTQE)|4P?=icb3E` zfq3Crj)Umz>+`Ap{*6)c0P4Uc@L-KAvzUOav68=*;Ssslka7E*ypghve6jB(4$&Sf z<0+GU9?hErryP;ra;f_2o2leZ3PKd-^yiN{Q|}-?_${cR&f(U{c?C7m4X-q#pRz<9 zS*)v~bk~2HG!U!3cY@pO#U%^|)M9Hq;QZ&2fpbc9{dg>}1bd42qUV;B!)<|_8bW8@ zm*K`X7_4=Gq0rjSc++$S3OB44t1Xn$p4o3CQ9C#dEClq(fBQf|R0u9O&TtB@U)2rL zQVn@#^2s7-vjpD*`O=oX=Wplvp_2mhYe@mu*d1xDgvTTjOT^|EoziAzHEBZ9jc%9c zjtN`yeiA?o&^mqMwb78@=Q_+Ww)3UMYwln**-VR8Yk+oqk@#l^jF~T(2GJ9c>s+}# z*KBya<-C5IwWB{X;0&V|`m`Z5kh7s2&xT$+<5D=_V*`t;-PSM|#rS?BEzJH5X6D(? z%&AqFwA`k0Gf+azWO$)no4%BXJ_HS2}qZ(PQYEjL7t*Log%Lu&137|l=j zDE+R&$&VDGXpH9WhLVOGuz#JsbZw4f&0waa{bXrDp6eM!^OQ!{uG#+*3yqAn+J+F! zeo#jb1>_3R(Kr{3^GO$8xb!7$Ldz`g=k^@Ms zGf%YW1CKBy?&wl^Hn1LK1;^Xt+^l0d=P+biT!FH6`iZi@b;;qYS0AEyHEmhMujZ@2BIf7xiz2IRe-wT=?^M#@Vs(9#F1hjJMrw>D6)#^o@| zz9bH#R`=4D(`B3#9`g@Ee~fFt+5F0{wR!2Wnd*`D^4o{q#c}U&nxDW|!PVY4$OjmisPFE-X07l?(!M?Nka84Q@p8wKn@|mCG#m z1zqwkuFjtk-m@mg`ru?=5gpN&QYnO4^fERC)a)7`d?!uK)nGwn9Q_mz2!BJ%FxnX$ z+oW`-p~lNKmne!kH9W=*Gi6A-0|ELT-S~m`y{n`$T5~d9QS9=W42Z>6u4f;eKj~xU z-Z=6_qH1UhFmlWEJ@kMIO3MXB?uCrZ8LhLoY+OyEa{ye=Hu`wMUd?*tDa&=IqS{hA#kcJK_UkweU%!T$zF+#&mOCulRY&+s zO`1Yxtx5_~8cCnwwmK4IU;l*>f&#;KgQT&nhSP<$DYpG)?G9jFsSnHi1|cDASa+Pi z1oqpJQg(;v>%Bc?7zi+;;4;c>TVM^irp@6kp!0ny>GaJ0x6|vuFjLG=?E3t&a}r!J zteR1Z>!aDA?W9}>9#31+Ugejjw}NLsgHw74WksefcYoeK{l?M{Dc)_5PV5Xu=BZ?2 z9d*uXd(5XW5Q|8BZ1l!d+m+}j*+fLg(+7w=22J*|z={EI*7jGRf-wE8>G{>!rpjFY zd!NXs))Hh+_^MK6hRs^vzP$*k>&p!2mfK%m(P`xS&dvE~RzWs8Luxns@#b1$Vb;EPUBeg>wXolx z$X3$sLDF=3siH#If^_K9D#&s(644Rj@_f3o+4TNIadp=?Av?33lc~FnoLtjDv(j-P z<`3DKe9y<7QO@P=7{oD&fZPnbfc(_3QIZ;|QT*saBz`#S5zgirrwRjZ&a1&at|cO z3W;C(W;;6HV>$7nNL6e$YQG9=}BTutdAyw(5O6nG!1dFz~=7bDJi~ zr}7s(k|l~ow1p)H|7a$}M?eBIc3!5`67X`BtEMZS675Tq6;+)ePQ z2Pqr171Mqdmc(Q33qn!?7{>)g^Poh!nl)7*3!v1$Roh6uk+=8W=2PB<#Uk##h{o(69X86*yi11x$7$j z>)BIdmRF2=K)+{%+lg78W+cvszxFO?YqvcI;GS>X9#7;%cdzT-lZ}dqm5NYaQu<%zvLRx`UIon^w&LQ=?eWDyt1_K}Irc%aiHTnN#q#qX~@q>r;j z-Nr|6@a#$!9e*QGv)Phk;k_EKC{kO9j{B3it19qU02X2!q7&OQ#(&PCwv$7mc9yy) zIGNW@$FeHsr#rp!$fqO(`TLz3{Nox66C%sOf8{toIX6iV z&k|Z-_ury$3N`)W7vp!ei4Zs2nbka44ri{(DRQ@uzD=LOKp#NE#nqa?kS>f_AOiHR zu=3v@h?Sv?zflqlWI8uYT158vZ*)I$|5C)&B4QrJhWJ~Y3a_uRTaV3G@*6Krk^L-i zQjEOYh$M=hMycC~9OISwTKGadYmQRx-*aXG<;v1Y?1GO)y-zJNeU7In!H5iWTYCJC zVjQ5wWV1BC#Y+^-yCWqx)`jZq~OVW^TwdcRJLaD zlk@ZLh$)U7nKDh3LlD)BAdk3iS#u55ATfne=# ztwph1t;%*WPY;U^lInr21W>a;M61t3ps$>Zq`%5dw-bSgEP2fL%iA7K;lm}bg0xrHVcWP~Mw z%s;^N&z_xH+3MFv9k;Jw2c#anQ^m20p>e^CD;cx@geZaX2>Up>sjn=Om- ziS5y-$}OZ$f}!`m&GESYaY3JLzPItE=-)&j0HgK$K_nHey6N-sdmIrf2I(TU&G%ca zXw784Qt?UkCy5q!+oVucBW-TmtXDWy>5jfw#|98n=~r1_2dk;7JTNhz^2eW{M#xp3R)2r4qJ1%GOU<$j4KL2|sT*yX*aD+5MvGN)9S zA@DW~)ys1(<;TWQKpTSOHNW|<#>;IUDoXx z`gpaxWxj-_3zt_t%Sm5e_S=Eon?u(ScPU+WYcJy6tMh z$h(1J#bm6$q#tZQJ~yteCY!DTZ9{IrwP#=QFdLI%(`s40!3NDm$GB+`b+Y{zX7Ld*f1htd*zK#`o+u0jkg zyeTr9u_y!IWc0c3hecjVnc;-$f2FJ_=UcVl=(FDYcMOLxz*3wjNqqgqVA5zBLuSoAQ1r0)p~DQ|kW?pK-iyaMou;q=Hmf&w`b#67k@Yovnct-eH=C|DEy^6)9gFS|Cd}KGEEU3=MGr@HOcy(F^ z2||G^nux3b&T)r|sXkDhSw&_55FLNp+_yKhOlfnXM!e7+%Q{YrKPPY{u;v$^@|1H% zg&)cYFMj5u20}SapbMqBFM39dUdw6Zu^zS2t$b5pgahjesMify_91Pe!!vv`J>Pc8 zulIuUpsYIzUIh1DT50(3q^js;>O!({Q!dalxb37dgp*6J-#yK^*H`w_=05jC1h5eD zs)WCxuXBi0Lme5TFHdSWHVrP%Uu8?(@&dJ$e|b9Z2!t8%zPsn@0CXz(L(-N&f}3&G ziWI4i0CSM0-p3PrgPa#3M2XtQewvq}pEFKho(=p9=BD*9jREa(iE6?>gUZ)e-+w%d zqQQ*4cpa-=Nfa1-2hLN{07Y>Zn6u2kgc=Xx;JGo(&R{*cQ6s_%kGBL_O}p>J_-mXU zE5(TUi2$y`VSRF#!?1bz&%Vu~q|w+%l3P`F_XAm#cX0)l!KdS3AnUO_j-H%pbS-PS zIO=-g9OPS#zbU=1s4gK8whF>uIeVhEwNWn3Lf!Bx>tFb~t=_Th zMk3nL*fHTEcZM)K`Ummx2A@`h^*otAxdCb?%19pt_#^(Vk#GfEkmS{o(I5n<>ul$LVvL%a)YEJm&iZEzmXJ!KZP z?;Q0D%KC2+qCZh|;ef`-G5h{er!PU?Z0omLG27f5k@iPLld$nc8e5K-2hqO zhm{8^bsfTW2<`C3b@#C$q)N<*iYJA>1Tpvg&@Pye0Ms%Su9gWm>B9SWx#}&C zl?5z9&3vJOAoO;W_cwOQ(RN){xJo7Lk&_V#A^}D<==Hw-v`drPr z4$A`&_E|)B8M=B{EPf^u3$Y zy)b!PZz`MXdoqVnJLHhQFqTKaD|}Xm{Sw2(eqkrBAj8F0ZlcbwAeJG?WBX6X`Q|>l+X@SExqX1M&++FKFi@Kt;vU|)t8rO%NqZ*B&n>= z*7`3UpoO0H-FKY^V1IgJUkvL24VgJv@z*8&FVC^PfP>^{K|5goXATnF}{!|X>G zyM$)9Ftu<0#855MC1nN*eK%F1a#lLb<)1jXZk^IyRmyiF?04syC5^nyQ*mhW6uivQ zE({t8LXYY1r3j4+oYSVfvYj~1iWykF9Rw^01+>Bj8^O(pR>!|9^sx$KKU@xlYp0k-3%@gJT`g!7Bm zjd3G;tn4FrS4ZAsu+FMOp0o)B_DekIz^_I@Z#(_@bX>iX0M!V*J1f>fR&A)apPgg+ zKIJ@$CX_UpP1*11d{*Z@d%GICjLq8L$3y9)oASQUGi95x zJ-IqNM!&O1R-SHO(CV-}a*i+{+L{$gweG@CKI_=QR0EQCHV_V<=kywJkM(=02OY>C zO{N--JV0f(TlaQW!t4cV``gKgK4DTK(=bZ&l~VJ&tv;|DG)T;ulqLiG1iu@@VntJ8ZL+dCXm(wf=R|a~Wz{8tcC*i% zZ^M4V@-NRlN|5K;FP;yfe+V$I@hbes0H`0zzTmSpL#B|dA0_Y zi2@mw3!Er<<32@oQTW;!3@b1(E6o^~w|lF>V2jO3Vgg{@M( zTdI3Qd%X=Ko0$ig*N_Rw^hrOMb)^&4H!e~W3w-_ogyzzDoxbKl7~Ca;zusY)fa>{P z`$en(4;jDxtPOjwR#@E9H)O8VE#vTI`+5?{Zc-STw7C1Q==oS%8IW5=uiHO$SDrcH zO3IPG!ywZ(K9%^)_)A zP7x+r;qC!LkFLJayc(7>i~X4@CN+@QtZeC2>qw8NBV%g6xXbaYK(^m!mm>yrD3;cB zUZoKRLNvJAX-ZyCEOt6m(cHI;pYhXcl}P}Ly{oO#=ao1RVpwzu%h%uj9dJyl`kn8Y z&oMoZdA}XpsD8t)lEU~xdjcWO5klN-cI}@$J!#e24pJ{#iIvS=ij%maM}zjt^qAz3 z-E@t|(@%hWJDtaIUZK{JfXQQL)_V}lDy7dgFShM}>ZT8#{eNq$}A#wNjllz{k4)=Zy5B z5pRg_q1M)N1vaMsNQN1c5O1Rt3N2TiI6DMk4RtZ`8KGUbyaFGmzZ$O&8?V zD{tjDdeJ2lc{f)&9E(MpQ!-sUdDia)n!;^bcMllzjWhrG-LcLSgOC0McWtHiHWrWc>6wbSbcv~pVqY8`t)2hM$K6KYZeAVN9tNSY zqS0u}H_%TF{4nxv*WGz_n?`jTFN{qO`Rl6z;M`h0I2hPm?V%SK*chiC*WoEyGMdZo z=j%Pg(R$^4L*|j@O&rzYA8%80iC^3)bLhx6|6LI<-XJSK-{Js%ljdqQRxG*|41U2u zH`~M82MlDqANg&@rSi;bC*rQXbdtMaym0;8V#mS64|;g>uW zK`G!?oynIZ+qN|$k8PDwq>Os-cEb9;nfUD$xm$%8s*&{cZ19bSqb)z!Cx}|jS>ltW zL?EcqoDt164(N_p7~8Z@j23f|-(SwwCyJ(Zop&(tY8|QaZBzuOSTdwr{6Mb77j0hZ zzyaiaG-5`CW^HF-v*ih^*3fDzXA$4usGrjn4`lcqA`;Vukt?UpZ&#xUc8Y*hyyS2{ zWUS2RCH}pSv6h*0Ipv}7uch7~%1RaKshY1O{H2>wa(JcQYJ`QayrVpaN_8m^IZIh# zhF?F{9T|HgM4FbCW}8E0Hk+OOwemV!|3~=YS_+YVDD%D0Pmo#_O|czZL(Q|JTlLYv zD-RHf+11@HsAIWcoRC#C3N|f=R@AIpKpA zol1h9dr8aRQSx*Ft?EC}+LwHAf7BtH#m$)Mvb-y@i+p*f)8~+>uF0%?K~oJl+zc zls0sJuMM!yYDD$Apm_;sJff?IhqmP-ye5;g z3j_-QThMx+;KBuN#(qI<)H3%X@zR>GhfIHq*DD_PQpL8@uX%?Bg-}f2z>N@kJb7_a zlN}Aakc*P-nTM;3oxhq4W2RdK)rOaQ+?za@10w3a=TK!i=e#c&GQbTdS4wX4t5=Mo z$&TtumR}Vfu8DF;8WuNZBBZV=!(!1rAOdHa<3HO1)0Nh> z#%>?x3W#x-=v;%a(e$+-kmU;h<>vbz96`Sd0H(XY61Q-xqmd8{R;K##9pA1xh)hlN zK2`h0LA2XegYn#+Y*A`J^xq98z;Qafz!eRIK(7DFOG#=p9_!U-Kppqr@WfpO2;5DH*|nS&YX_@nG<%Mu<+2;@Dd5_4BXR*Nm3oDNCMqz+d7mGU(5YUFtg%6-V;$ zSn8GZ%v%S7v*9bu>jov&7qM zzGVfIHSN`I8#QxRxDULJY|r}8V*0}O_11W?*R#_%;?UQnuFFz24oq9KDY_)mt>xNb zWZ~L9)QI63MXvOiMpuD8@P}I7C|le$%)+nf?@Ib5o?i?3o#hI(sc*ni;9P1Hg0F$^ zZ#{xz_T^2;Axi0EqxwD#gg~2o;6OBrEgF^3^dUPJ$S#Kn^xI|yFikZya{c|LMbW$B zPA;gSL?Cyh>&p<8oUVOXc*v`clX^)DRwH&G>rChKYgFY}gJ zm+;fysAE}Uy4_SI%S6Zl59Zx5 zZ<=~X8DR;UMx7OV=dEyZj?wDreaTsY)O4HsF z8aKsEk#87OR4=_gtED0Hj&( zkmrLohnOT7B`<>}mJ`7eyj!?`D(H|sV!s@E?cNfLr!kz&B{K|*rDpXckhkQ>o>=t? z0KWwxno>Nh@=tfeEMAMaWVddhc{E;{F_-1;Xia1;zI1JZ+b4#WR z7zcVWBq%6%^_&}M)X!5Ppi9O9B>e#jm}MQd%$z$7zDMKjO_5vyS5M#%Wpu9J)b0+9 z)i!7tmI(jl=P_S4Xk*-**aqvNHAi_RsPvWuvDgm)91&0L&>%USH4Z|`_>RRNbAngW zJW=RIcjC5RPqHsiFBN)sT2$J{Unc&4eH~YhyvHLwMgehgx+hOgdd-5Z480qO2VvEl z_x$FjCy7&K+V;=sdL2Y4#^8UJGW;UQoI@$S-<) z_=h&BhyLV8gLYLeA`byTO)$v8w(;52zL(G{_Sx?futP|z&tnD0fyv%M_Wa^d$B!MB zlQyx>HGwNNUc8!0ct<;-{V;5I!JADPpZu>4F6j`Rz zYEGerR;cf~T_-i;pkm=wo!wMXn|N;dd(w^1!kqf`8LFmUYJm4pn68hWY{GU{z|st1jg^TM;c;6dIXcu@A2qK};PiWb zu$t2~T-GM`5WGGqO%B&1fjk^tpSW9nK>p8Pc&`%jCJ~5F%P>;ihcM}1oO{y9r)<`J z#y(UWEj2#|X!_(+_3pU;^_#hgNnm?+h)Vszs@{MaL*zTKpXaKRszQVC^o<*w>U__) z9Y+g>ap2X&@8n{TJ-7Zuji;#N5`FIvB^7(x_0r$sQUYe)(TB#baM?4dIqOKzdVqaj zk&w!BrI&<03*R7P@{6aE5itZ})k*gK6J%%5&Z*;5Up(_>om_{OZ;N6uqmkoCHf8N? zj`y#4{JbNG7?_Q14=omMdv|1aOo|yVWe7Pj=o$HPD+5cpU_@^`hMph?y$+5lH|B%w zAY}KYcoryB1|G_fQyU`QKoU_Jj5%EzFRN6ZqM}SG;ktIvhqym0i~F$MeLidH$agnI$F!d$!bB_V(j;WO6) zZ21Ch^jawJT&TIP(aAwj2ngKO|E3EQGvjRbk=zm(uU7}{WS=+{RSr$IKQS|aRSR-}Az)L)sGI{qcxuE!s-<3I`PLE~E zzKNaF$buw@0$NERDoImT#(oqQ8XBi<6#DD+7I34&9W-1lu%30f)SSY%!w7myzRgUD z(-oXvwVnGqJgh48Kj-+ff8JDese=vUVeX^r{-dNOXOw zy5MT+-j1@xbEW@dY=ISGU=Ar!3dO8~M{#n8VR{_Ql)wm;dqE;r>$wBbn%^@#={@EY(-tB;R$GvA){lBtR8jMXcU{I`b_Hk0T-{n9@4)9;ZXG;gE)6KD3We^x#8DKMly_k3=kF;9q-=7ZwN)jQ9FE0mJD9L z$^9JjmmL7S@ZXn#+D7ZKTT^{kF4lW)ZSInR6a!BO7#25I`Y5L=P2aa%UpoC6zuyh91y^|e~GasA3`6kNS=6kFAgPoe*6FFzr5 zuP))dPusjbK)v7hev2O-dDceEI{EsiqGY#zw-STYqWY`;;q7sx@<4%R9z_=G`H`*K zPDuTg@Nn8;)j}IxjVtX-iSY;O!;glJenbrv05t{~-Hi;UU*#$gU+4Qt&EHr7Y%{2` zv$}&xT#(}I1!keu5doG#AIrpRA6NxmKR;r1U^6t)d_Y;g^X;qYDU2*CK_1Duz@ENx zL+9zruY*orhhe@4x61s@l$gXF`|klO_lfA4Rf`5j zI>FkfWlAT}$or&hd5S)6o3v$K5*Z0!S;F8z$PGZWE?YUxmIhfNcefJiJM}nTkmp0U z?rW+SWDph(!kqjT(|CZ$@;f)Wedvarxw}44$%uFPDNUqN#n5%e3}A8I=LQ84N5E3` z7jA~J2`#|U>FT;<1Fki_>H3q-+G5SQGn#nbZcjsaqCUtZdo|fQ+zKzpP+n>|@D@~Q zOAI0{6qA5?(7#P&uOq(7W0wdd6onD9!6>cv0kLkmoFlg6n@ffwx1g-3 z0wGgOvCO?b!i_=se}>XauA|PyW%i?~uIO?L-a-q@^)D)FC};E)+{+(#O>12t!s<)d znS`_wUFyf;g@!#6qN?d)h7NIr9Uj)Q76xu8Y8pqzsnLsZY0rA)tXR)Tx$q&F8}$X^ z*mm0F75w)Mes)Mdg^-zK%jX}fab&k8L=`^jdf2bf_Yq%>DZ9Mpd&h zb29h)X0&{eezV+;79o?z;aOaPpE}FM8m-#KFN(J=<-xdx-o2V6WqB@Av@Cf;&fM^F zLDcxe@;EJ$Ioc$Z)SF}tGmxH?mp^nx~*_43n)h%parKfVvh47gao%s_k)rs;kWmBJS((u zbL$=a6g~@xKS^YxPx|^kC(p1s2dGJ8rrIv)`N%CTsxX_lE#x~{e-ZUY_-%RQDbS%8 zG3nD76~jPdGp?h;Ko%LdEWOMvm{WfgRO(I8iP)xL6+YR#tI^7nW9C>+3FYQ z*yfsh`hN)Y^goRL-Wx401MF@EW43911um>K>abELw~^_37xM&AdBswqB|T8_I%K@@ zLff#qXUvl@o;kx)FXR&U#pT&iJP3xj>Dc6g@I>Kc)ZZL1!u^z`=Wbs%^FDw}CrO0y zhuAiYHb2Wl?9HT~aOu{#Gut{}!77vQ|8)^@H;Cqxqfz`_N3jQJ$Bid~ zHkwH^ngfbaHmgiN<1TUawrng{-N*;ng-&A{5rs_JQuh+ylDyB8_@207OVUUEO0{1~ zBQ~Few&q*E_1svw31)HCLJOyKh&Y26qE#;)iQndGXy~VjYBp)#Q9Ee0d{zpEbv`!A zK>RyX{OZJn^uuYk#;eg!9wjo`4DgW{1~2FSzvFZ}izo1XpbZ2(5I%6*xFCfqY=*@T zv=Z1qd7`m7Q5x*HRwkk*81{ZRE-Cg;^%sgYW7SH3%O8CiVxtd8V|bU{JXfmnv{8|q zD+(L~Ki)+SqpeXQM&`m;4QPhbEke2o#opvRsqopcqDmGF4H^Q|N}m&lOL}vLTjf;} zSy&yfeHzj}l_9h^5E?~{qs58ZAal8f>ZAo zJts$jGkE-+T{g@1)tW%~CmS=5Ne4Mx$z2V#lH%OwJt0u)^O%JV+^Kl~4kT_bE zV>#+`>VyB#jLSPsRIAuMQajlGa%MmgaP-P$J~;yTFXWsJ)K}?$6Ic@M4@NICo(jn;K>~ z(0r@+$XblaE+!Cuo!hi7Rtg>0HfEC3xQ7Uj$78scmk~YsdOcmJ(R{vza+}=cEu}K{ zOXw%SNktH01e~qIj)Hy*q|PU>^xVRyBRgW=68eyq&rTTf*NjY8PywL!}a25>>KPK#Z3GKzGPbD|+CW-Ob zTa?^p#W?2v5W!@sK!c-XgS~S1rJ_)kOLe4_?3U==?z7%*-e&- z{pnWB$xmQQQUv{{M_YmK$`0FpSVj}P@_wI-EVkL(fU9LGJF|>c)C?6ixtAYS73p?j zXn#x5 zVys7_ma70fF56!ttajT{8DygmngTPQpidTl@i-dQ0AfMg`>$Ng>VW#6tTz*4(~}PE zw5V{|354af-%ghGuQ%5NSziU;h4)%hO9*CRGu(nn;|ZC3v(~-laT;vDeZf5d$$}9_ z5bv;5l~M}mJqY>;Gk3zK6$C+g?h4xoZ&USMyD{_RPzS%25*E5lv2U=v5j`Et2}N12b8fFUMq_s8(#f&Rea z99fltq4d})xNUBe^22d-D2VXWd3=cCN{pnQ1XQ3B2w7N|SY)GobKUoa)u^RG>~Y5| zXwqa|?N5x~nQPwM>;n@jYfkuy{bP4efV_hoynu=8@1Uc2_{fK~LbM`Muu*|aVjSL; zqxNN3Gp<_0zep@M&y8qw6PG|WG4-h){@*s`D(xp_Gi@%&auz+gsw7rcFNAHLAs>|) ztTdIJj9p??qrsLlx;Kwjz?a@7aHCn^kw-X=i&r}9nE}eqK)=dBSG7u$odesUSy`d& zyvC^epSKL7xH{dbZP!_Sq)J~jJE*45Eg0MsMU4N@HCV9a4-;74hlgmBc&LU1#{}{` zcVfvo36-CD$CiPBo~Ln1Wuvym=I zftjo2;?y+Fp>L*j26rjVh^qqff||bpq{kpvR+Eu{D5FgO0V2VjN3(+T0X)@@jP5Bh z#ep^cZHuI?SBpf+AEg1F$H%M#Qa|e~`?lPu_3lbImoG2$^S-!om!Ffw#n@hcdMe3dImEu;carYUrBZ}5@2mpGT*Kc%GV6b&t4N+`pe+!C(JG`kUtY$f|6VPwM4pyKY$G%1M~P5<9+OIQzR(j?N`wH)I%4!Z!gP&;{+`J-`(R%@M!skv04mE z1#q0Og$~lwP$_R#(abQ#hf)WSwhSG#^x<$i_HQf4fB;kw|HwQ3!@bnN`TbgH;Z${w z`AD`Eh|<(y(Vuv@+V3@ntBqWC{%|h{J68~(T%W!WS-v_yc37X8#8m?ZLdolh%>va2 zh*@vjw-I!-%+vx+&sumYg87BTq<(RJc!(R==`Y-BWRu`!2=?i~yO915gw$YQ(JJ)Q zDQr6#PvTG3@;5CzaOQxIBU#fHslNx{;>jGHZo%B5pP$Po_s){DbG zgOAomBoDV1s?2MAA3HYF_!!U#2(N<9Q`OgQq;Zb)JRd3V<0H6@g&!A$ z;k0ZKdf=huET%(eBLCC}I7P-hBYq8ID;iL&WO84gn?{R1ekdC?Hu*%r_hzI?uIlyt zUMW#dmno`NjEi$Sp|lnsi%gZf#CZG5Hj`u5bG2ey!u2Yvk4P8mSCW_-qzfa;EJI3l zJY9LczfKg2qC>0&lro5ZjG1;ly5>mF-I_!QmS;|r*#X0dRQgAYrq*bQZHp8kn;4P? zJ3U~V(`=-lPuS8N*mO-lEdH#vwfaCPzWCHOx^ER;qHxS=p7#^89;b^tnJ%!{G~cKx z9pNHnNXW=0HI|T+(gX|^i8lZ4^cNvr5g?X-%SdH!{5GwgCN?#+YE0_nd`$%Ei*BVnH#&VAioP& zgwuuwEqYEMywkjqIpG+eVx~ILOEdsIS6ym>^6GH0=wNR>2q2hL&lD3w-Q`8kx$D36 zJ+m&Q5m_796mnY3@>jy>vmE@*OvcuT(#{}+^I}?gBJD4UTxleQ>(vO zf_MXu5mn}#b)*$)0C{xu4I-XT1z#VxF%sJXJ>L98uDml3+@@tPH94zXDQnYusy!eR zWIZjl+mhpj9ugOj?L46gj$kf-F^vaeitfu zbX6kT1FzoN-YE|cQeaj}vkUU;}(~*yA)AD2y{K_{uF){xPwhA_NZMh(8 zN(>Lf+A0|Tow5YHnh&KZPhrI}H;c{AK>96A_ z$KO_z(G*A}VJ6d{Z)*V2{N=z|RaC5Xq!3Q-kix?x^MX}xuQxqgbJ&h|4?d*HPVC7e zsN<^wZ6jy@IKq3&rM%AG)w%;!L#Uo6t|u9A$tN>>p#y@o=k`3A9E}h-8KB(HwV(UN zBVK8qj0h4G8t7q0D#Co3O=x-|)x%JbBmNawE zL=eyM3hlQ0b(>LSj>d*Y4;z{7PPKJ*1PgzU6LD0(vfG6jd>hn1`ZbaFV+Z%H?PsI7 zkM0teHQ&#Q9)E)Cv{gQx)%o>u5m7j$Kp+mFn&~u35NuPK#Zx zB)}e!2Q_9|ap`OA)w+~dyheLzw~R1{;)y4+WuwR<*jdY%%ZUwGoiMJFe_nkv2_QCz z@hCQ`xV}Asb2@K9@1kg(C;en40DNr&uK5E;J+C*upeKq7nS`hYjlU=L;>eCpK=aCj z;X{P{ssZRyun?jo?)QVi9E;@nB~%4ckY1lQ1s3@;P%BhvRp;tg+fw))8QQQnH*UVO zxQ1xdX3m*>o`#z|Syn_LRjcFWZB0U95^KKR+5T|Yi|@k zi@>7>*9nj7GD&tR{qN0MM`;s4)^IMi)E9oP9AlKoq8~rm)=h_5(4c8cIjp^|RQqkP zNcz@kp5I?=oB8fWqCA*F;nxI^QW^G>C11!$o}_D;6?eD$J;pm&WBoZM65D?N8OO1u zokU>|1+zZ4{$KPQ)S6mGN}gUOygdKog#xQ`u;@gYalw<@F7KpcHn1pe@_#{d)u_*U zr{ihB2xb*nel00119OUXe-dG}1-67i0)S{-D*n4TR)PxzURmQ>99O2H#}b18kMTuSts7U#kjL?DjeyLhnB3zhCh z;T?xzQtKhO^DNq|!)1|;<&fCMV!(YF9^{*pyguVnYN-`xt&yEbXez(xj`NnwB3n2? z!--ezhbsVm+G|rM?vd>2UAsGf_bxuLUO~H}f&iRkG;7^r?*QkB*z|$_%4z7*Gt z^Xc2}ikERCt7RJ?g$4RvB`Qb-EHchicdO)k@MkK$#P1X5nHk=uAC{<&$0_E1gMSPh z46m{*0mg;L4{bcz?KS}XNAbgVn$IYs60n zGikxb*I4^`+7_T2AuFcgE?Smjy9GDbkZ{l&5jsme^KCTka=X+iH-~U`R98f?mk(`Q z-%4I1mn%jU2i($saQcdRggZMPi#ZZB?IWm9srp8~1^A71y`DKH^lw!&9c?zHJjyFZQE~O{j)Gnf|8J|J1oD_Y3pNQU zyz#p0VWQkvL3jeyk-2)+Z`IMLvmgU~31p0cdqB~d+hzV2!^v5S~P3B5rZIA(! z)v`#T$0wG_i}{6{C#W+3TB{T9|>{eYvH8dMg=yT5!Yn2_>t=OxYU+yK8fMDRy9 zvtYU3ejg18?P|5R89R*a6}qctNIFmFKd2w6eBvg#5V1t)d6QOkkYU#4aD9xrkv|Dd z5tAmWI>=^8h@SUDW?cK_h_fzCS+uj0UJ&u;Y`T2QN|5?Okkt`_9Xp>ALAI+>WkK(^ zBR}&rZpFj8iotQBd2vvRRQLt85Dr@kvBR*<&4dp5&eGTcJl}E?p%|)TGLknYoL$S9 z`{DH(aM(0>>%7%H{DjM}>V|pQ)hh2#xI3OXzM+@NeV8T+D8^gX?1s^}0Y9Bi9usHrzKLG)brbJx zO)Uo{;c=nUyu+sY{|r}jQM_at(9jePB1Wvc9rl0W8>&3kzA}&4z6?P@C0QBI)#l>% zHt|UuFqBQRAGjT3ZhFuPSNB_`csl6Ajw8Q^+P3XU+qNbKSP}6bIz>+7D~;4FZ~U*)U-0c*v(*RwYNx_QTErj7N{WWqkem- zD|E!}lUDHp6la(DWUYO3kBNl|D%46}V5@XnI4K{Z2Lz*R*!cc+a8BT}-zWZSvWnU0 z>a?2$d{`!i;+@XT9P&0_-PgePD$f~1(>?j4s!FRcRv&Nz`y=@T^{O16&gX(?0Y+L8 zPW6-zu^Zrb(lSrL7~lNi=z7I}07k@A=N^EFlRfax>vq8gDu(ft<>Sy4awlbsYfmwSx?;y`*4ct zhIA?g3ni`BSte}OHx0x~pF;}@v!n{y2M>K*RtY#~n&>w}W6iMF4t24GLW{NYjp>Gj z#suR1FB#Jv0XG#K9U#VQ)dG^3%0bjkD7M|}^-wc_QpCKkS445n0fwB-zQgxw{u%v( zL`{0gU9uQP*&f(e*V@WpGV8RXpiLV3T< z{WDsBN>{qucQ|fM_&phN>-!_iJYP59i~gf`o+5~}j;LBlVefhs>k0c2^02YT^{GUe zZN!SbLcnsWIEMzBV%|9-%S*7vE?Jslb>3*U{T$7a9xC1kP%Z_w(erLx;HLziBKSTY zg7YpIgX;#jRB!w|DFonV)qJ(o(end*ibXX-(JYm1Jjtv+M$n8))Z}~&(Zwk=hZRSG zxU8^njII`sgqN5Nm7oLtaKfl8E_^+Ig5+@y!a5i1A@X*o2dwFVo|>UTszd{~`Sr_2+ID>1H^JjaILg4QAj9149z$+Fj_RE1`ff z%{h7FW+aFC*S1HZ;P035&z+KA(&-92 zI$3aA?^3h^QR*@{E6-IyN(gHuK?BvD4Tiuk+(Ek|ja9}#X%0L=oh)m@3;y%(<_w_8 zCaO55EsoT8Nt zKXY&dXhrxKfyN%i)Q-VSU)38X1d zg10h?<+m&Dqx{?n7KA8bFpPpr`G^cOTJ1t#g5f!@Pa%ID!GVby?O8=dwLH2)DC zHdK(iBm=mg7U9^lt8)Xo8}GT2_4yVKb`h^Yuq>b`y~{W5o$Mn3ular zSWk)NU-EK@D^i6M%;MPbTjOnIvg^ubq4yi3ePhfW8>Fl?!UN}gp(H5wS$JiL<%bNy zldj`hXSXQvV%J+xE`x4V{XR4a-NB*TQg1x;pnLb*dt6s4aT46I)L75!r)3G`3+NMW z!+HEiwnwc14m^hD;|RZHZB9V4JS2BL8gr-Pn}m;ge;N!=fZk_n4rucf--)p~avDw6 z%W#gBFuXNsa*a|1c;;VTuTGM9+IFpeiB}Eyz7r7ng-s6~10rB0o3DGHdYyimG&{Mi zX(bcTkXJxi7MiS8M?Ni+SxP>+SDL`N^wxXxcI~e|ZsGwU=(@?Hiq_i&P`!K4RYTP# zPa!D*834&kq~HQPcHWwp)5_2~;9q$)$Q8KGR1ZAYb&Q07&;(*Mhst)s zx?N5clT7vEM1_kYRZTo%fWUiM{Nj^}dFmxBTc>FLt^c8P^Vrkn0m{?E2}*z!r<7@- z@pLV7vaT<1%{h&#BJaGQcJFJ0;8bC#8155{D2ynq#l2Ct2{P ziF*He^3U$qq?TUzHaSZ4fT+y{>?flY+leh(5%Kolk9zwOORNTIyY?rprwpUmT;nHT zdQ^73R}dzUhO75Z#6$J-OXsHjLEk_%5Gh&k*v`rbgrFo2$;tUVW9U4TH#VE=Fr_OE znnkBF`a$k&2BXx=>l%LVeO+8}&loxH_JwfXl-~w=Bj?S}ieM~&0iFM=E*vUJku{GK z3Pn!OULtkm3= zoPJcA9&Jh$PCu0L>FUpaSmhuwWi7D&EZOJ{dFx{IEoTl|Kfcyk59M1guP*l8B>SUw zT_y(0Y6egGHlF^h*2B>7=_!n)jq4opGPKdcLHebog4HJaHk<&$-I)zyoAN2I* zz%=n+_SGH+4QS>Gy$?wiDu8wJ-1X;9ItPTH%ym%&3dHINekPXkA~ei-Hak^ZK#5w+ zdn5JS_3Jkr5HWFoNL|d?q;!y2QvZsiv#qin!fJlI8;eXih~`&n`rVi?8l+GtU1dZo zD+&SHx#TuCzv$w80|VnXJw24uk~jt*6o<*qXWE0Glkuz%J4J#xBiOuloPxE{7(g)F zdyeVSXEZIs6$Uq4)lf|4bPLcZ0n}If)6r7&?xM8o$2pz()=Ch-5bLExa%}?A780pj zHXT0ZtUKgg@~J zQ}Bi%L6aVa$x&q7P~J?$wtiVx-PhCc`Uc7!rX3R25^*fZRicsnZ_;y^6$F7T%?~nf zb5Pn4a(JbhTMbbzw6C%isv|up3el#RPtS(%ioen;b61{eyD139mY}l8F>b@$QOp8v zl|bQs_ZG$HXx%4?I@>g??hotXXuxG>$t;31#wGNr8ZEk@H#)`n_vtp>kTP9FjZnL} zx&R>u-a)c7gRb1oBgD0l854?YocYD+s9O@s)80B|g?`S3PXI%{rJuLQT}T z|1pX9wMr_|m3uuwdUNb<;}1OAN98nwh@CfSP5@M*M)e5EcphO)nAIte;`P;Q z<$#)Of()B+@$AXC?7b6Q;uw(Mv|#;Py1J#laASebE5qsArY5oMN!Dngrh@GCg~#~^ zH0NjhAK=yLU*cw+yibxX2b5_DM=;VlsVcS)!)*N*@&)d0KCW6vCoCmftv4B$A35Tz1r;S3B>JPZ$X}=*9wja*#nWcJp=Dhh%aJCDX%Ap+~zveUZ_33k7!a7)k0`zAi z2eSuW0qGkSY(A|V0>&p%l(?p(H~ylT1{ogN$Dz9{-y?UW{NhY1zE5I%x9N4FEz&8| z${*i6A*W-#`cvb!2O98uU~YGfXwt_TXqg6(FB-b@d;F3b;g?>MSDgv=dYLqFc;L zCqeRH$7=K?7DeBLfWe2qY|}5du~mLVopASs9@Le}#P)E%3#%2wK2y zPPQ3ng#`>S<1@Ih;>@6kv5+-5&X@k>uh+ytOzAx7>I>yZ$y2nL0dH156-Bx>PC4W# zUp?c@ERv4#E1C(d?9NDSDHt3SrBk-9DVDnab(2=aQmM6Zy4xDaNc@$Q2B|7mYNsPP zP;tG)pLdscPukpr(oLLKxaz|<)&`Cse(Bv0Ht&6Y`!YU46d8lR^YVK{H<0`ncFPnA zEWN{aaEp$;(?#e%(-v6f(YGG=y_+GpG-xQW^7VFn4mwE=zX&iB{_6-Vdxtp!eLpkr z&>p`E(FU~}S=kdLlvZNuVm~DWdd0|?7bf%yxp8(V0}_@^GlUeS$lt)NA&giN_y6+6 zbXE?YItOQ3_NJU!p0Jl)`%nGizo{c2!{U+Pa@6?k%s4}Bw2W3k3^*Ph#Qf+8vvQHBpfTw33ftfKcsvilvc59?reELDQ;GA-C-FwV zJ@EyImkpAWaAU$9rAT@-UITSEKFOk6(j2w5TCYw{2Ew;D1%3mN&rFMq>WZW2WsEY} z0F_=5Aga%Zh36J~oe8 zn@Nk^58Q$Yp8ULqC37qJRkMUN_!6Hi$GTjtkhLmAU_JKk(eagm0ZOj8_+np z;&7(KkYx|#D!mLEEEDq9FFg7fwDLRyxk{n{UH!jH0EwQ3-P{}MP>{FMsL{BVDP%hs z3_%an;*@sqyy))?{)nW!Wq{7_>9MP$OEw7(eEyQO&cV-fTq{k~8I<457-~is#k?1z zxctJ?Z3g^9`pup~BSE)6(f@Ett1A1xQcTo$68qoM$!cKyYuWXsct0)x7lDd! zGzSu*z(HIvq66s4WDFQ8JIvUnpQDD$Rm~o5HGnFJVz%r%4B6iX;4F|t=CAtrK3`#A zIeRD$;Rw}Z@TI)FFuv$XR(XwD42g|6rUA$M+kZRWX}@p>;2<+f2&N#Olx#$A+$^^l zOdmc&lq1qP8x2vMjVN$&#Oh< zCiBx{=VSm@9VRNvh3J|KkkblP0+_X~_?2rBAtgVu+U9GqOOO2~jhJ&fy$4uE53la5 zQNU6R+JPtix-3(x2Utzs{`)bVL9F2x?}Em4jld(I6LWpB;`-=q#a}nXh037HC>F4) zRA!r%6Xga3klnZ*0Yf!rs1yX3-YV*8x^ZjU&xt4lB=s!5-$Sww+W}tB#jY$>T9`

^9bR%npFo{w;alG;Yh-@}LRS9=Ps7~@02l4SS%t0c-Bx;ouwT)cz0u8w z<5<({;rM;_G%H*}&J=isjp|_Pge4xf@bt^8ThwZV;QTZT1)cGNsCYoXX8)~y2_-pcE{+SWHFS$xk57*lQT#xhBvOl-?j%Xb}A8GP<&TTMzR$)C{ zbkUtq5rr7!m~yCv&6itnyWUfKC#>9n6(@x~VBsfJUL>J}qjBr`d$9_AF!6g(W8)ks zJy3r?hzwG$&DJoG^66>R;4cX?FmIYq2}!}q_Q-%1rvP@pCkQ~Dvc3zYfZ~JX+6&jthe46<`0O2` zP@Jc@2G}~Eb`{d7Q;}qTs#Ce}OcbAu4BVRCdDM_X9IRIL)RlBosT?x?aPI~HKzu8; z9hD3Wg7KO;^NSH&ts{A=&n1&UY7;DS!P00u1hM~;*9DXdZ&G6$|`E2@<9S{fBYj^j)rOPG#qU*5Y!P) znB!|svBrmv;Efs?u(Til>toPh%o3c1rM&RKV_(FA^5()m;ic*f;PebUo61@!hOxU^ zTHC;}^yVl#UUya_DewJo>o(!Gx@nX8r|5@)+0Jni53iTPYGMrCGzX3jS@+i_fwCBJ zMu@kx_beJZt#tECPF-}B2qacu(XTp1yn8u4*OQhoi;3~^^pHwCt8=O-VS!G9EP0!H z3Q5lA?-k8`IY+p$tF2`ox`21l{izDNs%CHvMy7e1IZB{$lJp%6JAI6vK;g9Tm6xas z@XR+Z^P|W5)=lsR;#4?t_k5n@jfQ|->(;;TTBq>h+85gD*0ZnN^r7k~b_k*a9m0iG z>hYb(fboQe5{_z2m`&g?kF-BNUtvg2;PL^T`i14G+1s!mnpfz-s1i9WD_SUh6xTw3 zG#4_m3Ja2d5I}?Myec_2Y`Q$c`q@M}k+$vE<~+SV!V`qx^tIZvI$F7I@Mu%v$m(Fh zS%rGf*QW_uM_`iiK>veulc{BN6xh@4rAZ%N_)#3pb*42v8m?@Q)cs)Q3oRZHb42Q{ zr)g^WPnA5;%xk0f63yyR1fnCTZX-r}U*gfT>AjH_02cyVL1iEM))96O5-lP>jKX!U zhHyb{&gMi`y`dNoS>^NT`mDNn`5b3034`hon$Bw`$AXnu$4dhjp#P7^lo2 z3R$n}j9BP%&!>x@I1DC?39T5Cq2QD7h4y>Jcn56kMY zc$AxWuc#mig8e}tY~%G1_(Uexe;v!(Pu;zD^rwVP%A0MxoP@{gDemz^0=?z?*xR3f zC-yP_CP*tj88IGN{~n&VfATEvd*k`N_O4>;;LO_8)5_Y@{ag0?;b+uDhYiJRZyucc zKY9?Z@<=7z;ltyMVmW3O50SU;>PkQ#`JZ?Uq>>z%d)+9wMP?p*q+(9)y7AN;!P9p}eA6SGE9R3ciW{K+IG za4zvqm<4~lI91ah2mkpAeichGC>UN-Odn=4<-vUxR3#CjXg>TF? zuVb%`y1q!@c8wk_)g6Nsll+|{0fni@8g`5x$@m|{f?O3d(`TC^)9MX2U#STz?V!0L zc>Hf~lepipt_K(ibZi+G?GrgdG6B?!rz}}`Ji$0tyNNUb^q;MYsGQWAXE1B>=9jGt&H zzNg|TE0E=VOfEf3N$gwr9*f zb0QHj2oq%#JzJye2W-j@fyCh?BrnM2_I+oo)jQqE_#>s6suRiiOiTAmTt^_~fXG)N zPmad*F>qi|aOY$Jr|c6msQh9##2n(Jj^-`v3pRlU{K4N$fH2zhFc$`1o*_VHtnAvf zloAMpjP2+1DY4w|iM_L1?O#{HGGzZt8PN}~!=pF??9}Xyz5c*NnU)7h+zI+!G1J^a zwpSm@`=igAHzz7G$@ol7q}uE>KPG$$C&uf}<~TT6c;kpX{QYcuEwilS5qbh_YGi8J zx-^tai>31t*PAI$KL?6xZ=w3`6(0a}+8lr^hbd$79-d=F@tjoiUG$95S4`=r-yg{( z0@LWmx8+v9{q-Y`Nk|jaLc%#WI;OkObnHi&^3 zj5UC-{i(Y#wh%t*XGLekrM;|Y8Ks^Pkra4t0-WJL>s-vDBr`SFMMOcoYP&-+UAdq~ zblgxF{Ersthpmh9w*oi;vy-_pC9ZQFlB;x$4v3)nz$86cctX!BKxP)~m%uJC#@5Ht zFno|0^p6;zn4bf~nFb09L&L`?h%TCWb}&W260~yUM?NUv0#il-isU{IRs09pjg(FFZU{GCoJIPkjaO-vHP3 zBxC1c^;l4DX_`Gt&c>wSZ5ow6ZmHKPGtvc7F>DXt3BL`2NymK1a+zu!x@i@{ls?mEjKvT}Xev5>=K(7HLI`%Zw~x zp6_FGL{{iHKO!pr9qz4Xiz?8$AXiHoc94ZBZgI{DQRx<0$kZEls}wuXsH~ z^l9eiTsqT(i_gSzqg(B&y2wcoa@*VUmie&sUBD&A8Rs@h%sR&wEJo(VD~=E^F;jB* zDHO|^Ua}c0@$dhC5%**X*wkqJ0FFfXZWcsW?JY6?3Gn6#+kLu(OK17UcP_#1&gavH zlf*_?L&()#nd_{V;7_Eb>$n7FheuGv|>n zJgI`O7kd{HUlLA_88LFNJ{n_3GsN*ba1{Eu<_Y8$NR z=t{2`d>=~2$f^}eL%E`=B>@yU@-4RjifIMkM-v#_fq`u}`GCPCUd-A$%vbIpT)Wnv zH4ki-%~Z-bBf>5F-IZnDZw4A-IWw&O8DKoo!jPe}F!xFo1t^QnK+10r{I`R%vSuU)m)3YEIY6vgk}@Dso1m&oHZ*>)2J(SsdsW$ z8gp|4%3jMB{79Aj)$~#!W}h(E!UWJnOI&ZdIf+N}U{fxplVrUi#Sj1OF(gv}AC@~? zWxk{al{8aUqy468A7STqMDVPi(hKb#2q)vi#^f-aPyZQo{m=dG!F)6s<6#$#%c{oA zRFj8eit8_@29VSmimBN5R&(As_z>>gX^Lh0fi3>4J-FQ)0==J7M`WkCLA}KQ(Na0i z;4Vf-#N==8>Y?2-+>=R11^Em(i0PL`~&tymDQuhqnf;St)SDxmwROE#efd>w~TKL0ds zP`7t}BlA71)lMT1VRK|?&KzK=sli8%tsc2uLyIrNAF;$F=os~zkO2wZ2^d0^9v}1X*zYZ!gee`el&e@80yR)r zzBupXXpeq2f}{aZe3GQhZio*t$oT8Hgc=g`>RNnM1<120@68B3v58%+rfTE$F^~Re zG1eYH@GeYqX*l?k$|z(z5|li8X8VQ*{kp2tu-gD~4dy7tw(K!*gWSsC6|%am0U=DY_|`ZS&6GXwnf)R5&O;%^KVQXkN-J z__EIVaAP|M@uM~KUp_(1#P?#QX9vbcab>CcGnl#Kiz_~x-yiA*2@CXV-|on_opI_# z%SQoWPaLMf2;m5=ZZm z^zMrQkXJUT9GZiZ1=x!g5MM!9DJ+iC98in=l#y)iiFUehqv81nJz`Ezr=mvAdh{8| z`)}QLuBSMmrm8||zkNGg9rRH({9}W|QW7>kN>TGQw$4Fdbfl1k@d}^E1)8JdoBgh3 zeh(PcA^|1uL&fWE`e$lV+b@Y3%DV+*I!~9It;|BAObT#>jSR8-?(DIZ{BJ|t0q4ODdtDcJwYFm$t(t(X_ zZ`|1JYz5RZ>px7+z^j*i<<(Z{d2K;(Bd~=Wtd+9wCJzMCE2D z<5*M=)B?dhVuXk*CgkXT(uyx1wuWLqJxN4qQM$y(b(1fBK(y|@lxRv=3Unwc?LoW1 zsY&8hfce0p|a#qj#>_ogqc;g8`Qh7cQ!Z2%v+sJ7G_Ob1Uf7}1oy{ej;G@1;H71}>IhDq6Jia`6%WPJ zp~)3brya_&08ADiNo0q_yEr{$Nfa~q`po|O>mwoIhr}w_jH&W{%vBh>X}Gey4GyT$ zO1%4#a-$G>$g^~aGJnKGzE|{)r+Z@KoK?N7iSCQSw4V?+{ae-QZc()GCSwTgEbgz~ z8{Y_#b(Z0(n7zAO=PeUG6$oXJ$_Gxr-cmyW%y8**3F!HHs%B}IWJ-|Fl5H_};jZrg zw9QlovZg%X`=|!`rY|)qRe&Bz+I}9aTco9k+vKrTdQZQWDNT(7J#vcmR1k%Cqw^f1 z;c#}aW-CKs(ApJNi)EBo0j^-d|ETo-=U#YEV<77o7zuhqiDlV=h%6o?2lw+dOl4V#UDS%_V=ZmC6k}5t#~VEZFG9#fF?~i}Qy6<)G<>>pr|)rQ;-(i5i~pOSR;0}qSk5aVaBdC)PR9fq zMoumC=GcClx!z~NF1xT@z~@faS~P-I{~&57?)sp}(w%3W=w>yi^M69L{>KT7P=Xym z;taX*7)($V~e20ZPmhilh(rjic@qwVz`)b~yU;vpJD3&rb_S zGL?YW2oijGsK1E^$CzcTfHYk4Va%N(rjvjn`s7;26+N-RHp7dn%VfPSArwIU*%j%1 zWd=kR>*awO&?#4RBt>t6!*XH;7EYCF@gnN`x=+zfJKt_PpH)YK2bg=^J5l7CiIX3M zosrzB7ic5U>};oSo>>ejW$!=D^8ZoD;tB1(bhi1%91C8o9N;3Kt6;YCKd!Y{ZF_xO zlVxye!Hrz9yQj$^Dk$cTp(d)l&Y|lje2y4+IOQNmx@0Y^+R6Fp%~xV`^R;;SbwW{w z7ew@)y>Mb61wRDBwROrNZw(%^Ek`Jp1&;wq&Ij_1>PHZ(LA;P+T1`B@=KbGSUfTeB!g*Tq-jlaFKGZyuofz)zz9k>f0Wb> z-#D^Wfj7`72+t^ZTZ8_*bs(VU-VX}AoqC^bQy_@T@eYNeP&#BzF-S5b6;HHHMx>iT zD%I)HpV#9xhfDn{iBj8YATA{4ZJw$n(0NR$!-htaYlrj3Uh< zna}}G#LMP;9y#Y=9dz-o*v8?6S-aKjJKgo(-aCjCA+HayeKT)6CnBL>ygpY5j7Zxb zBAD1XU;m{?{#p-_x1Mlz`f7g<_>i%5bi>o|r>@Q!J#p@x2 zks{AxJq?BbYA*LxYnITNuzsVG9Hx*y@li*WZr@9lUi~a3S(Yh-LOM6u!$eMm>lLb4 z`;uaD5{Ef=Uze}MUom)TB?e=y(ryRe2nD80woc52NY4c9M-PV1y{rZo+>xC7#Do=g zgkMr$oa3$Bj=6hfKxxpak0O+xK>_h2vPXKG*k>erK=uT;6b^1ER_0Z5s5{_BIs~tC z^`_F-Q#mGmpr|?nbfExYavz999&C&|%jB4S^mhi?XC+;{rlL*}x88J5jCYp?)H25v zbjm_u8>29lz0MguvvH@aZJTXkknT_8Rh~h{3SrW+rN1P@zTo)2FNYlks^AYLmegY}S(@{RqPE^G7j0{;+QS^H`5E8GA@iDOo zV2n)&r}uxrs+N*vJ0T>rHtdWW09A*E+C7FtsO%Eyv7ZVOaUBshjDT662<^O5GL3`z zEsdC!-uij*^m?zhJ^vi*$y!QGClu&8AIx>o+fV)AacR9(;8EOaIy>4o^q zBJle|v9&5}Jk#QNN({|g-_$Q}@oSp?-ePagMs+{$_nqh|p^|T8kY037uFEy4~^`C+E%&CA6Vy4}ZC zS7I9vJLQY#!naQ1-Sp_&mLp3}k+CN(dJb@Q>WS;es6}pl<@Wp5eeZMytu+%RYD|(A z+?%qk?-w2Ig~x2YkL~z|!(y!|082_ZfIUc`=waMzu=4-GqC~^qGRtV9s#P zePD5UER13|sx5r)L^Lr_tXpDrak2|AKbdPkS58nFT4Gv86zUv!B%Qdyf+amcOfDOk zw5FX1ZziK11KGTD1}dtudO9AqM`zVVC;n%(E>Z*hH)5HjX&)%cQp_DhQ;Rg`&XLW- z;bM=}J3W$(`m=TSuop!jYbJ9`NRArXM6%qynfDy%#xQNYIbb~Sz4S?FCk(@ruiUh)amHq*SNaY#hy$5smQmsFp zrCNW#jdAGd2Cm5v7P{+~u_Py#u_Nf|S(n14*_Igm^@2MehxG%xU~OTRMBy?~l6U)W zE`ab+FuMHqNPpv|55~f&5@OVOH~28y{V=%!yZa$osaE%t_xkiToyW$Pis+e-n$;9N z94u1$$>VZ|$91O?Wtr~~WVmDe_Ga4>o9~JrAFdgajlP5FVEj#eD&>8E=1f2 zQ;Mt}vaAShP9!yz;UYV*h%3XH<{-Z+pL{liZmc*$?+Ry-cp6=$%)P0os%q8C^?oRyYNdK3)3rWRNt?`RR8V^)V z^%W$;jc|04&>6|gLzE6pK=fjxVDhMMZlb5*MWu<N9%vtnA=0CBgqMGUI)KG`?>H5Zn{?W0^bMHXi> z>`O}dHSv;n!aa8;>U%lTD3WfS9*^MajNOLnmPhC=Rrh-+x1Bw_d#(F*vCn%#`RDoj zoux-%S)uFgu&!94y!Gb1HxG8fc43y-p|w#|FUdabmZ92o5ktW^;WiG}%5rvV$_`TA z%*C4&z`dp6RVrr!XYUxUa1>W5ciM>)E_ZJ@qZ_)L7}=0KUPnLCFJ6`=`G4lfw+6b* zHwS_}^=QM#eS1R@l9|75u6|j>xAE(%-!?SNFuwQ-)c`w+N(`+;+=<;Df()65hdcJ} zC!l0)^)J50-S7XcYprOKD3bf`RKblyQ#ewmJXUx7;<3@9j&=8Dt`hcF`NR2VS~h5s z-kDYE>G|Xm2cF)?Ma_%srbc_?PNj3YV-&x=OvHFNr)hO+aFx?~;dk|mF0@LWSnN-o z7zkZylL^D#Nf}6f4bm`*+AQ-Z4n{P%;^blMa!2*Iu8vA;HRWReCGki5O=uCO5{W_=!j3?gDMgJ)-c<-+Q z=b2o)frq;X@qs9GeYIkIT{QVV=v%fRuaXXAh$ zI#YL!rvZxaA;*T^SGSmN-|xlboQr+M5uFQ=%*3pEE|Zr4QYHIVu~Y8BiB#`QEC9gX z1G|2{ZD5|0Y~K3!?U1ax6FMVfb?1+F-Vi+q_VB%9DEu*3?YuZ;YJx9wu{T3G0@2G| zVv~qRh!nYS;<)~{uT?w~sQd3C{auYpgV8e=r9 zWezf|dF4;p4R0zB1vD-8B265E?s64nzvhZGEe}jH=wNyd-gU0;Q`)6Cj zTdx5(+rGz3pk_7C3tGlbtBn@VB6WJ!*wGlm7>{pYjYLR1#&q|*A67{V-I-7K2OXK) z_d7Y8@H|rtyA55jr8`&@ch9MR_}^EL33-SvUVJr!KqmmXy52S4u zSyJ{CqNF4wTh{oV8$G}G`MvKS{gb)x`&!O*u5&);bIzGy9r?kRH0|Ttcd_eL_uSy~ zRd;52Ng;{)N%8&|)o+ilCK03e6WhPjM^sn8K4(WUv zMr!1HXWmTit}l1Nzxjo}c=7AHtg})7PoBY0|zrrrXR|&t~uXcOl{L+H3*FPGFrKrnm1y$;rN%nmCG3{4pa6X1PSNO1CmvW&QUFRwt-V zAO!7VI-gQOuhOTy%b$y6N9$klP@3{6#J8qU5x-zDL-)_QHgG+iPfT1bt{i1b>C*58 zSDZFj_PRY(?5~!O)s?=0Azg3O^*Zg}R==K>7ZUFp%ly!k?Zd?{aq)d0NB^?!X&QkZ z0PnSc{VqrGedZEjbOw9Ra~%N0>Ly-^j@Le`w$YjX{*vOIujU@dcTu*iB{?%b=AnZc+9BeBrCZ`HN5#hL z$(_FEU42(-a&0kJK0MBFL$UD4o*eh!G1EftbJBD9OGlTSR_IR}GH}_Lihz7=?mQp| zPxyIKq!Y-#f9bU+b6?oahJP|rKO%8Z;@JKBF+VKPjpa3g?O?$Cdahi;C6R3QT~{*W zLCmXngIbYH=5&3`Gg@%X#u0q@qr>zgqOl!ynD|t04g3PHz{hZ|2XAy9+;e7USuEdB zD%vW~r*Y;CR4JY`n_}vf6gBa4t+A*W<7mEkBBxr?kH}l+6Es$6e3D5)Tzvnf;qfeCClZz*T|J@_?H#VmpLv9MUfp>Qe!1_B*i>5sO_w#DJObAYd~$F z!vD|J&Cn3-Tb}*;4X6$ASBk4N5xjr?T8k6!;Y$&2#xi>a9@DG6;4fNmq#r83^1vNai=P2csKk>MLnUUFN#L?^VJ*lhCRp-8RYwQl*< zC$e5@{Z$t9r>O3Ws`n8ir3ti;o@$N3J9~P0to-^aIM*IcHTUzdNsBny^HcPhpZ)J7 z^;P3W=<~WgRRd}A$&t%yVV|zoj1?H_7u)t}Gu`}Ad}hu0euf|b|&GH!y+@M*wMB~I}H~q@z50sf{0Ya=i zSAp5n+~adCLcsGFZWMWR538)9c{jjd|1r1T@gLp}0s5+cZ;TGWHLkfn?FsP<*TG|( zpj~*g`!eaLt0wf5V`Rey?-ger zCFks0L;&351vSEKQNcQQ66cqH^XVQyKNaQ(dw%W(Rmy{=NLmRW02JqGKWWhzT}XUf z57e_K&{E38^Vz}IN3tqwTh4w-Mu5%D@K8%d%k8^On9_U?2b+4`)92b61kpEAzeS( z?n@L7T0(#z>rCy`TjzmL)zj@Xp;usrwkwlBcz5gjTnt$7BRa{llPL0C-Oi$2=c+rc zy@E400gyX-Gpsow1oeQs%1n1QrHl6Wt1nH@52ufa5{+G2snj-ia?z8aqFrS7y#Cl&J);E=slJb;fSKpA&Pk0t73kuKSgGd}_Q38+oDRhKs>RCPy>+ ze_>D}%0K8&JU$-V`taKGf@{B5Iu5W=Ixf@$rl;hydSkTP_yvkB2}EUbNN}LO`sPe= zxZ5$Rey1SmZ%3xBUtfH6)H`kHncBNGKyk~$Z$DEp?`(M~h}u8i3jo-no0L4|M_DSU z->LMXprKjuvdj1Ou4!K=qxs$y#CcN9C-*`H+w1x8s}*V}+8eayv`5MB#6KH$55LjC*kPtNZJJNqo86SteC+6w2X-=x`Qz7CX#8iIT&{u4Ernr8? zbJ|VF{r6j%@bvRlEiGECSybw1)NPlxc6_kQQ{~SL4wp8f=Lkei3gSj_9z;Uz$M@FjBtOW8@U^Pl z@&_R1*WD$}jj^Z2dZvx~NJxJ^E~Z8MS@z8`ea;z>00d5sg0=h~RIdVofkvbPbei|v z9Xei3@6BE#{fR^ShUATTPL<7bJd0d&fWRGepqbICaQrmZCSDSU3t^g=fr0x>lc{WH zZ=+73HEKi^=7KZ%RedddAsqC{_%-V~RE=ABCUr4KA)N}s6e{MxD7o+ld#M=TrrxAP zbA&2r6YRalQ;*&q7&4+0O%XrKWD@6OwqlY#*w%79jc8B}5uVlmgD=7~`h6K-&wCvz zYr2`4nVkISzsfI$+P@zGnS$5JCi!C&PudCzy%KK-)0a(a#)Aq_g7r@_KIbBdn**|Fxb}CuM{z*8WT{>RN|B{EjH4<5`!} zeEuA|e$jQ}PT~ew;TVK|fMjmfc-d|q>%V&GL2N=y^rYOi;aWoGNk$|KH%{E-bdZUK zE;RWqb>sTr+CgP0#T@8MY}5CXd9$49Ea@;0wos(w{Z-Z-}gKTw$vG$TZ;tqwAlVvehgT=^(#Bxg zy}z)9ad+DD!`}u{9p_QgsFA%#NkO(9Km5q&^lt=ewb`j<@IM0ONEGgppq7es4fCEp zgfx7%{`n#EGA&iQGUqCb>N+V=QBBoLAkKNZwn|S0tn$|@?h!59|3E9q?wy)2|G#m{ zaU>EHGg>Yk{#Eefhi#(t`9L|_&Mi3c3;OBeBLEl?a_>Z6;REm4CT{1|1-rUh(p<=! zR(|a4eAB_}UbzujFxa2nJ8eM!2i*~wLtnYW;1MV>F+TQe(6OiZppc3I0_PC&4KO^a zzO|`ZpC45W$4XjOL;2vy9DwJYo6sgL&~L)?y{GZ`?uLUB6r2+GAxkIi#?%Kx?!${8 zpXmT&YNuG(f6e6-B|QY5*m;gzO+apB$NIl+iANlg+-gkCQuGg;PVQVe`SjA;)tT{o z2lRi=*zsS1yfHY87PlS1uwMj!AnM1E@XM|%z4@3Z@b^j-owxaY4{n9i^Nu->X^lRe z$ZKdXfXG2BvsA4Ba_Q|mdPD27Uf1`V9m-2BkqX24C%P3Jhw}Tg(2oIUn}d=UPt1keP zQ3t3H*)VfizYTA*3dfY=%b$aw)WUsgczdt=@e5EGfsxeb-~;+J(T$LJ zF)K5(>`oYCZdcaS>WpAbN0nmA5rMj?TeJp1oMjkGuG^9bo_f}({v zaCmi~h|-P1eN=y->W6>cwUOtL!p-~F-P3OJ{iA9Pt^-6}-3Z?Z9}$|m(553IOd!?gVbIgQG&4P#Obt(7{1|_*8R0Wd zrC;_Z@tYz$C?eyErK+<7&kXoC^+~&x47JDHLrpH2O{{&qM;QGWmD9#Nb4!opyP<41 zKWXIrsryB@#~0L@409dg-@lJ<*#8NuTM>loC%3JQ}(!kx}4Lz`Msm z5ktj+oMI@{tUPad?sN~plLrx&f2mEgs7mY~*+-rm=8w+>+?XC9(Xs7wf8QQeSrBpW zXVR78F1A9rZ|p-g3e);lAS(UY4ji*PQ5#Wp^Au5;Qv=s+F})~;=+x+8Z5)Gq@UT}& zw-*PBG6t)>U@evZp(c~S#l`&R)_(QUb5fzKn;jkVGxL{jUo#cO1iZcK_PwfP(n9;v zx!OwBng}UBRhqFepiJ5iioW2ozyEhl zXh6H!;vQ8;AGvN!Ud(h+NeH8NbllZokY%X?r#O7(K(p7?xY&$g7e_ajiQuIx-^cGZ z%)8o)?>}~o>a5POeH}ejI$o{zamB+o3y(Li{hD%T6;<{tti6xuEcsGOKNA+$ z;*pV#J+ls~j~r+ZW&6h$u0RHNytxE>WXs#Jg^3<}b4>1^{lK;+P3b1eYCL)RloZ0{ z7||(bSzkRO-*1tHEn?e$bZiJ07BtD^t68{^a?O)JmoR#B0pe`ODp0s7iVfo{K9p~T zMMN|p>t$xWmGAR+B^epR-I5kRhOB)I`lcr*&s=aV=~Hz7ogOE3)(T;ljPLgP4W3RR z`ni6x;+G+7wml4eB@}u^^t5QqSgnCUyo7)U?!<{chHhw!^lQkRBKPn4)uFnj;mIMw zs(_h0C_52h;gg`HA6cDoc79@B%HGOsHBQZbTtKRBAr^r8lX~A?d3Bncjfu@j`+j?m zoNaV9rlCy5$K?hCP8mg5ZRU`Xk>Mf*M-!<1#xePDF^mscvtLhn^-{!*li0$6$>)RC znB%)AgZPqA#Nw5NzAhTzKl*$d!i9RJrwaz}2@HF1)^|liVkt^_V-KXcM`vvwGL&(GW+C3KTh8y`T9O@hui{26hmQeFY!)j zXmonI00JExGJA58M2um+46KmLcSQUBL%a2-dzP}DS&0`))s->XW#?$)mDAIouyYZ! z%eDuycMv#!o4u&l)23!A|kE9jJt>)@-!T|iwGYS zYHCmGeUn^pX5!q{5gtq&DcbKt`8&yNdO*a`&=9Ls546tSvcp5^MHW>AB-hjRxYFMB z`d8yLRl#Z(xf%)Q(_`_fpV$M)V*NH27w^0I{yj2vf&uf3uAhZyZyo*Sbb26DzuGa< zT_0d8|MVQ4h!1`{)(}AbQyHAgp{gdnsZTW~bu(}$c2|XI|41_05gH7vWBOmo`ON{Gd|wEaV%$lXy=bWUQp51l0>W zlqyWqAc5CX;&c%0)u~4DlG;pXre_-hzixdzhzWKVOYp36pS8Aw+L!pdno)cbx7qoP zabKa&j(ELQ17|KrEfcKyXEL%wU%_B3b#v&q2<=sMG{UuR5EjMglUHz*Eh zW}Zanoc*Z(@XsBe*|n}`+@e(rRN*Kbo_akJJ2mrw6p)S+nLRdB3l+wFI{osPtCu{Lu!{=k3GvbeeHxnd{`W+%k0<&zuNl^P;T#@ zqD%y$1wt*>PECjvXLl#%PUbOJ7cb&d77c~74i{s}mFM8PtsCB8*vyV4Y*6GAi=%u)LIO$Smsl4&cE)!A})4_$g7p1mx zk@Fd8XiVzKhq0bNIx$sau{y-XI8ifxuPK>w^2mo$8LFe=>x}yH2bS&=ID}Ou^OEn0ngvWL4?FEKa2CR z_!uSzgGX)#n?wHF0^$lK{?1Vx#$DBADE+GPkVj&{Njj`3_G&gpuAm# z4<6JSsJI5n(oP{{@36afjTM407|avrZ}{<4!8Z@29@N$zcb#8`E36GLG~&~V>}+9q zdDLgYP?4*a0qKE7nfX|i<)ThiMui_QUc4Zb+mjYOmH>Cx)3b3kZ&k|?*ie1|*alqL zIU5q-k(`lWQ~P2fr}fjqWI(()ovJt%zstZ=n%1B`>dI%QHZh!*UEuL=p!Vb8J_}Dv zOH0~_d~x(pat3Rq86%7EFr+17JJfsw>dV17F`{Y?8v=hC+YuQNK~zHT#=ltbWw~_@ zeWmaTw1bsbyIspSvv;$&%Ws#`rSsu}W~@UGKdaDhzJ|~O_FC=it$7ZdBpb3A44n6l zI1J>6AJ!xpJPL|$f4u)CRzgCL&;2vy_%Q98MIomNNSiXJWaX{^zH3SWFfo4t{7AgB zf8#gz{dV=*bHYJS$hNp4}-6j_(FS~}0NwhllA ztpaR8%gPI)s?T8;?Z z#3+PCMah%5`x^A%IvSDe$Ns?Ne8~6eMVf( zuj)5d4;CA|>Og15MKs2FE__K=_Ik#fZdK=5o~+_CP?abDLO^&!_4ch>WOes3zxIqf zHE<_4Mnc2yas0_IuQ~KODce4DU1ORnfaZ0V$9Ac`3_Qu2C51ZEZ{_s}8{enqaw9^{ zI$kagMXgl=nI`#-^k*OWa@;t8a2aDbSs>|AO`y-?{eJgQ%L?zvVz^MAbC0Q<>3SNRsG}xn zi%a(ikc}TDS%9aG%`4?>Y3)4n8S-m?*5&5rlCchjOjzCVQ&!<4LUe|+sa;a&QL5N? z_Vs63SyXIfgxj6fB=1=8I~CfGQfa3WIOcAtm&b|98QWZ;n zUt}f}M9*%F7FU$wq=+D7(EGbN`p=jdl8y_QLg82eV7I9jo>#_PO21}%_n`LKgL(uf zvSsS5TyT=~C{(uTO)qf#m#shmRaF27tltL*`4e zfM27)u1lKf{PCm!?D2Jg0di%iu)MMGUUwS-6X9Jsis!r*s0BaZ2CY zyEbX~w#q46NoB;qMoTLzTTT})rgiCmJyCVt*!cbsXttl1iL1f~4it7NhGOMRo8O9k z_x+k2viMh>Q{C($i^@}aes2S^NlCG$>xvS`&49hKEe9mRsCwoli3ARq1WkPTvX>hq zOeB~8W~LkLd?%ow&_5S1%EQBBFWp5(6E*i8cQ;tf#;M{9kG!IHzZU&C;r%k;E&2A} zkThSw9sbGnw|37x*N)Ha%9ce+dcvo z1`Z)Ec4*Qv*+>5!UHsJ_v$STRh8UdW?ug{-BQmGXVfjzqI6vA-B4EA<1*5Lf`gs1X znCd#kuUv@RN0R?R%jgkQ@WuGGV<1z{u(U0cVbBpSW4fNTx%qh#t;!`F!bE4~v*~jv z3`}+Y@v!DXE9c_jjk8zKL`Yny9?)!Q0#8p%jIt|R8>5SPrLi~b5MHKk{UXtT*&1clxR+QeyF9 zW6z@4-N)Va^x}bydpM=B*q;$J!eiFg<8?I+iHC1QH2hrp)~k#Qt(cp7+Ci?j$(M9O z6CYT47LfV!(x0&HiEf&rD2iQ&E6?m%m=?8?(C}fpqcL7KhC&O&FVjo=Rue~kDVSBx z`}=?sc6s3Z=V5BW;0X(kK22ugTquZXgYHdTzW63#Fa@ifMW;n*pN^k1w4!}DVNUzBkQ7uS>ASAviyAl#fRlld{1-`u6j33nt=Frmp#wqoG)=Kic}S z=Pl94(%k#FHR^0}J%$mcnG;J;z@hgCUG8J`miC7V`k(d)srNFAA0J;|neQX-QTn;fEnfFG&3` zG09f#lG>0OfrsRIhH=dK$KJ4xm-eTk>aVQt z9R@xq@51Z9OwUJ`*tFvl67qk^NE-!5hlhZrCrk0YG%J%nUWbB;7RSgsq@1D40*v z~)L|gM`W7yT0dKiEb5i{dGi?6(1bvIXjSz25qiI)hp zJ&-DScAg7WD8=cO$$d&BhWJB1XTOOBPztIKHw!QoR(I#p5l(Y}x3=&uI4-ay-ze{H z?A#vrr+C+Ae8gbûRotcY%83RFi@i6uhP!v3kU%6j9HCYd#$aP`$WTSIiy za65#?kj%rMBZtY9T@SS%rkvweqLt1M|HzY&2+C-P7OZ(IidaPpn#X~3nz;gmPAR5% z)ja{ds}$!{BQBk5WFna&E>@B#gEnnx$X+|)TL+cj=TZR4tImGGW~?QKyl%zrm?;y2 z>iD{vP!YJ-Cr((%z$2HPSZHQ<;a*$>v!%NJAV2A}i2BI!x@^*!VAQCDz=+yL;G`s7 zcXXENqItMC3l;v2_`(X^bKE&QrHAN-8Isqv0WVE&1Wj1F8b8#99ved$E-w}Rt$BSp zVBZkFeq=siDa@{B`KCkPh`bB}f7P;BaLJ5s0>*04Gap8!&H3k5{%Zh<(zAEx&YoXX z)pO67+V{5@43Dh4T~W5h=H++p@$0n6`s@Lb4#U7?e^+>S*uaCl{vT*W1q zl*GLk*l+lO>>ZNf$G7FL;-***TjQ>_Bk^h82iC*2&w8I4yG3~{m||}(|NcVeM`PU7 zN0KeVGlkK)Z|&FA9ph>Kx@`Np)|V4Fc3Im|~!0eqkTbp`s?*v`*i`V%l-L7G5@ZEUslH?YQg5PQb*Xq)Y6x2Vt2U#_38-Le2691eE-R~4nOjx}v9Jf$7XIBcIzsnzu@;KkL%#*tj zq=5nB762A!<}!d8eR$s})s0H?@+j1tYNrWaDxT>cv3SicK+K<^?gx&m;w*am5#!tN z2#!IFu6VBC@$2ulNx!g}7SvHjaJZ1&ol}MN!30NJj~-RN1}NOSQ}5{dNVb8F!a!&@ zI!HH227*d9{J>3geq${h z9zYKK_Wj|t`^SD&o&d5>)zh)~nDnEb3;9!n75bo6loy}UrLIA|SLKS@I06=NpY=8b zXV%BH9fWl3`Pwg|c(c!qcXlDs#}8LxFmrfyF{H|6J^ z#^?`z8}s}mo|Q*5C?hCo(? zbS!av@;q_^D(?a134i@MxL_uGk|+X`0~^C4BjXzy)EHS=p%;gF!IcrDc)=eTP9mVj z<=Htot(h9(y&b%W-&f7{E>(H3N<$cs$;A>J+63UD9LNwRfM=Ab>i4{MeeMlj^o!Hg zsIV}?`0Is}zrkm11O7|j;tfdgM~G;Q%*;kJ?Bkvg(}l(n8-{CqD&D4&HB+*8!o!WC zY*cl1YJo%&ZYiP3Aa(|BE*wKCdKI+71x`*^2mX_Wu}oRbo*5xCVg)dpgr-WTuW?iA z^1Um2pkhj6V`B@L2_zC1IVI_c1Gb=`;0c0UMa85CF4zs%Rw0=1-=Y9)5xD&=@g`Oi zLxSBE*UpMT9h)+TL1P3tUjg{$B9zbhDUTEV;>E(Iegvry{PNYSK3|GH;+6MBF%UZ~ z;4{xp#_<>Jb*xm^me)Od++!|#fV+nqlLIf#{_|q3YH8T3ud0*^gK1ntux^y~-sm!W zK}MrhChE0`oLyAbh0+#&G!-9V6_t(Tuye`@bAJF6&JPnS>RbYvK>l0h#PbJhu_<@@LmZ zil1e~0#%sw(=*J@7s_Hj9Jp)e<*uiA+t?vw*gXGdU_y6;5=ZjrsnDB92ny49#I|ro zT@Y=|ku#$oVt8rVi{`Eri<|=z zue*!n=P+W=o;@qS-;lb0YJJ@sq4szKl4?H)J?wjxWHe^MiYAQe>@p8V%>-}jA0#ls zsRu2~t~)C2txN9CU9=YGHN78ZZ0fpFr)yg4}78GYkwt5I`~dUk}WdHBST*F?-L-0@}DZ z_Gk5Ev5N|#qTP4PF`e4w-2YQlIpI=;%lK)CpBU)1Epp`rGBqQ`-{htHsV71x_b-%n zE2gD)8_&O*A6sK!$dBCG(a|Z0@*^7_!)Bj~*n{EsUCx*{#DyvV zkPN%{M@`dCIXl{RZw5D!fp`YF9a8D)k~+uskqn|0K%E&YCkXf(C@n&VbaLB4T}l>4 z#uy+1r`QRyi3LwbpN!TISxoo*k#Rd4QdaQRUy^xk-S0j(4(FC~b6iPF_XIvi@);&+ za(eY1^^o{g3zNwj1d7$@_gVThJ_I^jbW5<~JfuZR(7RXPfRSJL6Ji-VFFn&#&5lu8q9sK@DBxT%fd#0 zO3=sE00AfY5VmIAo-c|;*^rRSSOa?m{ zJhHB?E)e>xM$X(Nbq1e>u2ZtWN0rZ!QdBfuEPmNrcZ$zrOgG4hAsax#0`&ajRlbH& z;D#BkWp*9i&4F-gSU`pFDmWJ9^4Oo_C(4`;QBh8~6{=$RV?XWn)5I&JB2z<>b;ca? z4D}P{elrKGK!clutLmdbrk6}Owi4RdLl}&@e1zZVodcC&{1_`H1m@R} zPVY~j54Z3KloP^2zm`vgP;`X%uYt-sSU{t_2+XO28M$X;*at^^mMr_?EFZqV-Bv)) z=dR{h`315*n4{{JN5e0tq@hL;zXYtA`5z|zJtt#ogyXFY&U3v-A7_8|b@rw4CIB$F zB7f1+g*^`9Ed4&^X*$kq%9r?0G0Y;4Ito`y1?Hj(?=K1}yq}fsr=BrxPhIs}c2T9y z+jd%XH)3JoLi_!hL`@RM*ep7(a3x?NMh4F^i0$8!P&J(tkVlr)yU~%jBo(VUT&d9k)Ocs<(zdd$d!it$eiAGIslmC~KANk=;VF{zyu$ zKYLrR@|NRF!)aljd2FrYCVtzI*T|aK}>y}M| zJj%WOsN8CsnB1vTn_?yr!ta53ksQT|ycr@_N4I})$kw}9`#>l;(?7fWPv0MfShMX< zL_a|y)F3m=*jzbilSx`#z=i)C_RQ8oX(iIu<;s0r0D9K3irtZg*)Ns0*9voS@AjG5 zU6)=BY`Ekz9qdVYlOkwr>AxRv|NDV0R$GLN{AeMNNnIN;H&8gOk6Wv$mDp51>*18Q zVi$-N75!e+_cyGk`M;N~O>WiB7j?SH&a-$i<7Mf<($$31|NV;Bw3lhn z4N3>|w!be{v$Y*E`x&x;0GbkWI^ouBKEn`gVL7e-KN?&j1&*BjNk0?NQ9KZQtf}7h*-DrT?;_Pbd7kJ+tV{7%;7A`!^?y>Bz}#L1;vfcDSXO zR)30RF^MsV)n&h23em>uPWP1j%426Z1o4oqSJrOZbh-XmF$Q6x0$NjB7~NoFn|{~j zglFkE*Onk-hLIRna5N{iSg<1hz>VX?<92_u-u3ToBN3dc`L=)ae#^Ypy3?tEmBJrU z$J>GDXZJVHVETs2LbKpO#FB!kmLA!Xp)l6m&WR?D%!3iepTzH`QzbmfQ!UOf2-sdd z)C@!pVOQ_K1>DJFccFJ`2;G0_`}o7ryk-(X-5t4X;Fnv32pCn+hLLG}iQ+kXK62VZ zW`Ux$&WTBz91U$6?(LB^9@|`|{h;o^h zaaGYK#cq%qm4AHW+WDdwGcwH3x1HqVu$n-=Y$AXl?_#wScUwbeKbv=T?_O&WzIZU7 zxJN8X$NDTdlUEYfvRg%tZ5hF!|k*=j+R^I!F;QjMAF^re@exj22ly6 zVj3icjG4G+b@^i6o~@+b=f$}2y4-eZ5CvyEGahkS07GHi`oe4Yg4_xEB!j>*ER&FS zh=ZnWWG6?6p1AO@@!pxdi=U^C#a(aXb&o82=+SAQI}x-qz=Z8}-lsd#!z>z_P0PGH ze`a>k8{Vep%oE^ZGxAb+x7dCQ5z?gH8cS20%8tB;CT@v_Qem|=e||QWtvKE^LC|#1 z)!1J=3$fU&&G(!-*oD&VmD(n(pfbUaYC;n!;*AEJ6uHH`3?Tji6NrAYeIXzIA+jq| z(P&+6fkXtJicqSCRDtZy7HRQpDcKet&GDCgY}Gq2QpyBHXTCSK&6>1#(<`KB|RXDD3_t)!M5zs@|DQIG?j@Shhcx#X} ztMJ$xDM=bU4X=`X70O3M!GQnwD#uqW~96^~~h``QL z3B&;Tl9+8v^+PI0Tl8;S?L3$PohIai+9Wtb*8f=pGvuvWJg20h-W|9rr~Zw%=|wrm z>Pl+(2kM;8H?i$HSP!rX(JU0 zj`1t27I(NlA2w-J`6rg$TN_&&8jB|l9EX)UQ-`r#_yaYc&eQZt}@sfB6X!#bxPMt7!?6O_y7WoyLM*ajMfOY33Vq#k}P#=;g=rqC~GXd7Hi#X={i5+7HPlZUP zdjfEXQ$z|RbIFW#c?;uMdy*;+zjxl<^3?}2Jh+M_0~`Bj1?MQw zzfY3X6z_QmmWgrofJFZD!3YhT8Ckr<2&Dfdg)~;xzC)oq;X`jSTg*N)|nk+Ut z9Y-XTr#_3?en}$^*t55BHy~0|?+;5~%n@>Su=A_l5h`F@Uek z|JV9};Lz?o_8x8oA7!|@NuCgPU^8_)$N_sGTXY% z;hAR8!bbIJZmD;+j??!24Tn44!mC4P=L5U9?g;qtj`|Cg-?~`fl|}^ZzDuutw>Jc0 zXWLV%<=fezZS#)-bALw}H+mss`9L@Qy8N*~u)C_KcAs~<&n;^_|IU{#Qo{;{I&?ak zGc)5tcBYV9tdKF_h6`P+GSzk+tvQ==dE0#Wg@9Jpd&?dY`u7?* z9K$s79ryhF_weS74z|GjZl$Km>Xg@bA|i>n&@JZx*7H{H;&-FL(+s8TC3Q>R5;xa& zK9-8u^2oak`}gW2@3PtvBDkKnmy^S_ZlV{KbBcaPGCm?=h&*D@zxOoqNGhCmdzs{& z<=c;(imO3x)IU9+7C0OT{=i+A-NxfPspNOSV#mg6tlRm`{Df!L)^GO0Z}>cWl>Eg1 z^BbR?-+Z0$yodY-wIebe2f1XtpAPBI%8968v;3)RC0$R;*2Y`<$ew17OK+y77- zd4!8@XIUl??TBn|&SYdZGOn;4Ws|*t90o+DyB9^bzCXUR4%s(${*v3hNVoNurZ3=& z{CV|H%3kkG>BNNRd_Jr}^eBO#9kQcc#P;n7w5a7ZUPPb?oICRxLYxUif?|smB0z!| z#Kd;C`xkOhk-gLEf2e^B--2Q1FWiV5+Zda?`nQZF*{3=2p{7se@Il_`ARB$}rNa5E z@On9J@q+8ex5O$1SyI#cFNNbL^)dli0A>I)WxT<_NK7mqOS8QqQ!L0EyFq4|*WzO<&@Z<1$*^85#*08tBzo6s#>3# zuwxzp@!`rAhdR9`B{L8nvfqnoG*4dsWdbrCO4Yvtc(1zU{N~Cy|G4qmEU>u?_A#um z&k1<*ZT_9{ONsKlzqvlHItRp@g8oI|pysQo+^LO4Ie&o_w6@{%{;jU9s~ZRT5pjvk z&p$$iF+7r>j#N@#US7`M0~vJ4Fo5Z(<>>e;v48iTJqMBO1bhSo<~?#osF11wNQOXf zfzc^O-kLdZF|<5AR}=RgJbKjJcVp%9cvk8+_Hp zwm#%J`3PxU$jHwA;PKr_!D)`wc8;la2?8iDHmY7>~rFPm5tEr?k4Gj&X@K>_e?%xoaY5Dc325QGi8r6$4kdaUj$q<;O$RWSlj%0K= zIXU4IKA0I;-u&cBc<${XLPpBltw4o|l&Dl@>;niHD#?@YT*!tVefGg6mC!)Ncz-xP zUq`a)q{xX-h~tpQ=<9bWqYT|6^OJyC6~{@u&9>hfPU&t$^1Z0lpHHNwkX7#>x^Ih{n<9Hn%qh>*=R}|Uz(^#Q# z7jOu^4n3vNtvgxXdy3yjxX`4;m^uXoK!f1eC}_LJ*FR`UcAr`2e`EZL>;9O>o2F+@ zc?+lq9)w~`rFiQkXb1)=#ZD@oz6KaH7nks-nwuuFQ-HrmgB+i20!TSOR!XQ-=nB+m zMYf$ri+o=Qlt!tLr}`qaWK^oCX1_LZkH#*TNt76q#vWP-xyL$IR;g|A1|A_{g`H!0VDLD2l9%@5i>F}peJCkAa|&`FI@)LzI}&QR#v*7Ih-M>gYzFk zUjFdGbo~Bkz|d~=GVDbqJ&f!S1KC?T3_4UvK|F>-RaMnBL-WZzR0=%0ghde<85xoC z&+)cMA$>?&l7K?<(PLv5J_0dZsjwMI5yHo%^Q%EYiEQhpFnndiJ|%+kcc!){hBCo6 zzbe17xmrPNZ+2y0h*H5Vz0agTYQQ3X{g~C>RkqtSJSg1oR-vOJ3E%@Z)@g zUP!jEI-ocL$(K7)u_iD5Kes0yCTXCA8L(l{vsUteekyUz{+#C^5{<5kd4Rb|hB`nu ze)eiZG%M9*?jgdeR=zu~KJyk>mxcR)3yGbC#DY>6XVx(>q*gpq-D&Y6nOG1@ge()J zA{4nRJXVVv2cgc!eJVn=q6`W|3-w=?q3JJ`1O^|s$@4z3IaijS91d6b5*QnyO=^LG zw<5K8kKX)s2x#^uF27^i)xsb%|3lSfNQV&uidWpp6AZAH_&X?Jn*3sC2kARw^aQ%r zR>&6ZtM$?~IY9{G5bwYBCE5TbTlx9a$&>kZ_CT&Mcq=~3Ygo59+suD#Ns@w-KI@#; z>)%$LznvkI_vdkxMZQ%xwOKF?mA1MQmG<5DuhNABp-E;j;qBY<0)JDz?>HK?3O&Sl z2;|lip~oX`>_!a#Ki?4CtzqgI9M@TE0a*cQga(`&`!?m62@>c|^2-|Og8S(k#D$F4 za0chqSgNQzv96G+z59@9I+ZLlfnnv-wfeugf9SoIe#OVdaq-Ar2t_zE-%js2ep2Oj zEj9Y{>JVl4Gqurk>6~>kUG%$Fc&wP8*^u{oI{xIos06upkfB%597CUJ&gj8wJn-sB zt*DQ@a=Kpf-Ox##l@Sir9&FU>_q>tF@a~8MmH(VV_T=0pHzN2cn6cnt?oI5;0;K1diX0y%v`y^#9oV^Ju8U z|9=>_@4_HUc4He$h(fj*>x^wMmLw9%5JIx6#**FGBQe&-*o#6aB3s!dO9~@PizTv@ zy07W|{@lNFet-PF=XdV^?(_Mp(@Ex<*XvrI*W>wETH!W#uBPn~FhUaatMD1k;Z&O- zsJ-bOWn2X#l1AyLY(9l?87{2BunLF78nO?hGSKlIX}QsHGBX-CY=v*NiOr&U<2~TT zRPb!L^P);kjRS_Tu|95g+HPf+(1K$@LZEHU~o^u zu_y9E$^i9xv~_ica&G%miWuQwK2XG5r87@9sL%?OC-V@;;b{zVaU@Z)Q9M-y2<^q7CNfG>YR|0-T=Q`FUscgnPHx1c|7@{(dy%jjYTB zV<0S2m(y1eXjtyMFtx@Dp@#y-j2FRFVhChp<0p8Ei{WYAPf{_MXZQ$$>sJEhuGn*h z06jIasaz1;nGDSoviRD9$^|6oYU59)chr(dN(czG97AH62{*>H!8vcCgTjYI;HF7Z zA{5){a2-}YC5+w|c(FY6G_elxk*=1-8*JRDTjEut_Pz27$`XlOW=x`JAvnE2Jg>9z zQXOQV+o|Q z({>cF915Y@i3dGr`Mc#Ssz%g}bTjmBWdqVw@VFqcN4QC_pyIj^{qny09K4RiwoOn` z99`hpv8b_0iR74|SLcc^D#M%f`=$h{MqS=IQOc%Mt2Qh{#6N+*bQA5D{T75+aYRzK z4pCqm%R)@w-#8Je2y5Vgs40%g zr4bO=bLFBCXjCSV8FuXDYRw~cfZru}vy~MZ+hM@c;-o^VQ}FA0+SWb_StwX&{4?-- zutk9V#&w3SNt!|#_YRHvem6;;Z3)icwkV(zdID2yAPv7Rt_M8E^FahG%Fb-P`hjvU zN2|mRZ2C7{&Ni^qu^zcQ7SEeuEs$JW^LMCE=#a&hn55wvVvO;>D+!$1KR?Ki$mmI9*%*u^?eCoHD$btDGCc@NJ1bAP$S=lrz1)C3KL4#hIGp z*VeA`4c2T6roc+kxv7KV*~td>)aD_t!e9<{HN({$=8T4f$6r7rTzggBl?jaNpdJ5y zF}MVCDP5ZA{LEGMzGor~wtMFWWhA8lU3RjXyCz$?3$9EF*Y0|BtLg48^fZg?F#Dbq zbH%5-p^9>2yl^bEf&E>(gbtb=IDp4^kktu@=JR(#yGJ-L)g21ZY%=E<-xnb=laf~=aV#-q_jMDDqISJ_F`U=F zQ^jQgH0uzi$Hrw|(t^%*3&cSOdTg5>9j2i{TViBsn zDEesq35{gfTJN1P9mN4e;2o~sCAzd%UnVjeOX+xj!mq@8e&$E^^h+8apyBaca5c`} z2q56-WcT_Wrt&Ey?scJYG38wvrh}qt_alJC67s0V0s*Ic>I=6omsqA)6cB=QpD!Tb zJOUp6OfMcwapuNQ(cU6{hu7W9SrhXd-Jw1d*x@&oW&16x8+jlEk7I#?DvTtqP!RJ2 z%ii6HX4}!jw3M;ab*r1aI!l%>Fh{u1b$Mo|w=(RUX}=w%WS33oju;vkunt=Wn=~Qt zjP|)eZMFsk)fG1}^C^R5=BQ`~o2?61?(oz?ogJ*o!OlkkQLeb%$S_goCNpR9GV$&f zjw|het)KX&J6m6Ew#QU>u&{YO(o{n1(I}hkX(Ct*%byza@X~{uDBWZJjax zPUTYVBbG7#!lhp!K^Q7&`ZS3pWyg)qTt4|U3cc9jmV*$AfwGsPZrAOkdC+aLyINN; z1l&~7|LjHw8Cj?Fs(Nygb1)%a!zaM)H6dRt@9um3==ihL#MklQF>f z__qwI8Odjzq8a4qtH>}gAkILx)(1W(65twSq5dc_92Fh^j#yI$Rlh;X&Y7s-^GCNN z9vK+r>E2dlX^PReJfPH(=}Jy$@oNUfn0Djiuq{FH-&=>DH?J)L>UtBq*9hLcrQCJPMXQNm(oX1ObN0aw$td%lA(UQq5H%Ub3hE*3AAyL-$b zR@rcc#kk6O_3+DBG8C#$ya9w+xdqXV7y^nJDL{8h7KIi+?Di5f{xoX9suH&2u@?+H zW=VXsnQ*4_?S9$cPnA4;B3l)!%Obniw@J&KFHhOgC^wXCKQRqTutcDgg+>krE#3#R zC$a{|oU2RFVQK9wsw}M!bF&CYt(~(>!nJR43`L@}nM7d!{{VRI$>qn{<&T{qY3R3< z^u}82eimH&LC4I{RcW0~KnyYO_KxqIQN**BmT1Nb33e>chu*I1`yxlWO6MWWaCQtW zqSN7Ay(0HNOOeevcFbI8oIz^It1~e}J=@d7#H3pK^Q=)@Iqu)Zp)68^L3On(UDt-0 zrXb%yU%VAj<_{)xvR0-vl;5-xos`O@njpsngYpY%Tmw}{_Cda8Q02yhxZ`aR<>pNg zI$>{ddM}Xy%8lwz=9ZN-vCGeW$ty+N(t&V5Wn_f`U$%Bk2KBmGF8cmPLeR#8S9<&N zK`1A1fr}beq(`z9u@>{$Z*~3ZUQMB6t6sXi%Y)=J#$2MDuspBhOi7>Jzn7Iu^caqw zXF1MCr^bBI*b$~X8P)Yk#D3o(j$5{h&`p>FwPenX?!jQZcE3u)8Vdujyd9~0htUB} zkEMr$fOLN;!sw}8ke*ni+4VN*T1N1-d&Lgh!K-tnvmg4#{ zwAbJW`Z4;jIM34ej1wP64X7nU-dEIw3?{AIoPLVF;bK8LZ?ejrd-DvM{5y#Nt8iu( zB-k0k#ju+98DcqlSOeMmdDzJ}`%mM8*a5eC#j9Tv&v6zzZn&b!>EKip)1^)p3DxPp zI-1N<_XFD}^-X(e>FF?ASZB)biYkg}Rq1vG9D!cVXnU(tVi2^>(ZH9l7zt;L=p zI_WN~ve1cFa$niPoR{e=W#oq~H{#R-FGij`_>GQ=X9;1$3*Obvq+E+AWnr%dpiGlj zSIq!vZVDA)s=LK|g=xuz!*~eaLa-UML%SKWNigeFC<|6fsqt4_eBzl(5<=u~d+^(5 z!6A1I>G^_=VbL0l_RkHL{mt_X5HZVQ1r(>6z3^0hNUW-Fjd6_h7gIF)I8N@?cPDj@ z78cm_NQo?d68@%Srb-@%kU3j5}%$LM0S#n!FoT@hUoh^qqO&Twcc zsYV$L_P-Uj*!D|$K3-$lEaUE&`j&?0!;O6-v+i~Jlb6qnOlJxZ9=hK4OVM6*elI#w z{ZSOhV!Mb=JF(UBN8`9NeCv0pb3{?;LSHX|4`V8np)VLEV*F|E(=k6%5o0NS7`9RM z2#`N-9DgX>eDX5F=rRvC5<;1LvXsg9!0Rw8_aau4Dv)N{C)`bwe{?v;eeaO*sif=Z z>IJ7^k0!l95w>i2>>~fyM;+2a*RtoRz8yBNj_Kun`PJg62FgTf%<6J$QcSL%V222~ z3~$AG^JqU|fAi@1*h1dewY$C=42q&>0TFh`lt~5L=>93WO{-|&aby}h`>R|jktHPN ziXGW+#uv^DR=KjR3Ev%joUFs4A7hPqQIbcKDVkkgRxUaK2uS=CkQ%&RIQ^K-NpPbu zKV_H}Pm(sj*Q37_hZC_Qo-m;Q-J1m6l`*nXdV|sblQ5k?ZMFMcBGy?0YzlZAqiO=I z8pl%R?$sGRb2YE32rV06f(uZ&+>EFN)eWvHO5ULk6V>xNO}IMzPKl_Uo8*I7P8ub9 zn@gY-M?r{d5?s(ZQ}EMj@~*(%hD7N8Vpv?4%w=}JD4{nbCH!74k9o6US(tf-VcdBq zt;7FTQXeH#N30ALHt^OR{wphf?&3C&KTzt?0Mw zW@nx`1dR_y6bxbNc3DfXprLHj<^nH<-_S5Cl&sIBK(e)GSW}MT>^2)3+2Y;#dgX^M z;wy!`QI5F)V#&SVr@hVqsP`L6XY5K3C6~AI&D&zVkKa1Vn8eA5|6<~e%6t-4PAL%$ z?=x9e|8d<~9Vx(kkJ;rJ4e(=fipYwGd7Mikp|zmP^sl?;5C9tX7l2FN?#9&6N%x0mcF>Vk2Q{?c5yxju0wEeR-CM`0;Jq8tQRao3vGAw#re``qw7yI8$LGWY7c$IJy)FAhIH2t3$w zI|Ly1!;8ry;E&S)T!RP+eSH@f$RsXB>FX`oiRDwgR3>XKJbA+ktLY2@k+h_W3_W0! zc10?`ju5uqgq0ynOKj1Fws1rFjPBe5f6CwgJga?!gK>iuT~@ zd4clgTzY{)i)7T%?lucN`1e780PyT{dM!{EboDCM1HnY?&iA37JRWBX`=;?N0%5wX z8L$*f;jsg458I=yJPbOAVF5}+FY@y8N&z|e4nPtB$pO*M1r~0C!=eRl13U(YV>TdC zkUHIYa3!(wpu`z;;!qC&XC;?gMDxk*h3pSttvOQ>53s-tK@EU36xe+A-Cyfj91E}g zeNRUo@DW=_OUuhrGBPH12E2S^r|T#y(-|Cipr@BdOAJ(jjpJ0F7!cA+o%2cxqfo>F zG`%@rP30W=%y+Ib{H*v`m+1%aRIdU(%dpNcCFH{T7h#`E=_kPAGBtxf6O&&=CkYn$ ztPJLC1hl6C*7{zXOZfVHPSNOP?sSUD4r-D@@_u-#UfA}%?(weeYSCZXL~g+-faTck z(f7Z3lMNUig8&wn1uFm3Pyf;DFJt`=-8e33ayf;4<@0i6eSJN%uuy?p z+8k|hW-oDwBKnc)2bh`|U~T91^ce2Vf;h?2aNV^mom~V!z|<%PRYo8_7OJi7OhZF6 z8Sw|0fnaVkMn;GF5*r3bMjVQfAbV~F;MVr5UGDeCiO0h?D1D8|XyX?qnCpujmw>kd zWmeg!6_gJOHPkw0vRq;L)%+j~4>Yt8s+m}OdPS!u^zlQ$+rKHT#{@J36wFNoMe#Wu z%n~C~`dd>kPfbUWg^a_5h9f9mI4q^07Jit&v=36*P{0VJ;`ZM^q-W#6z{~DL><qmpe$1Z)?H`FUMd%53AaN*hIbF_GdB(e=1V2}@ zcQ<$LQrJq)4wyXEeQNlRi`x8Tg9-s%Rs`r3h+^*AurRQKHh!dLX1 znpoH91J>dhfQvo6$ox-F%z2u6fKf13a_kN#f`)o5hE+ys!SUM)P;Ui(zcl)hDwF;f zB9ySfRLa0i4P`Ur046z+FM!{AvmWH`t#7kc?qXL8)Uc(QQogr$Z4uqLau&Ydpg%#3 z!e3t0qzL|hPP|0jH0`-j2Qat2f(*J(&6Ea1hvhUELR%EVcl-8j^}z&p{Ajb^2PYjq zQf87Q(XVY?8VIYh5)zp93N@@=L?2^Hx%v5t5H_``Sf*FMDB>s-jrhy0!#@Ba9zK&u zQKN#eRnOp>+^t1pC=uK?n}=ox60x@l!9x4S^wsF=45H@p3lKV0^;;1bovz&I7std; z7YNjs+8i$D0)^T=+3Y1`UVV8P^ht$W*xxIbhJWULnI6ve2VRcd6y9tFbaB|dDBwvu zyqkphp8UHG_;uMl2iglET`RH;Xa-de4sB@?KG5u-z9tI^u6~_weG;xzNL#GvBF{iW zZ8!zkxl>;ZhO0Nf(7Pq9B3$l8m&N|4sLi{>x(s5CG4v;l!-~Q_3)sDlNrC}Jx~cLk zeS^ds5J4*ex|4P^4C%{D$x+sEdFn zPK0Vx^l^BVOGlf>wQED5rzk}cM$U5Z8kln8&AlYiRQ0EjW*#2pD1M5LZEqe(2pN@j zLX~O}tee#!>%g0QB9L`+59|xMz|2#ACO)YgP#t;9(2GG`KO~q9C2(B2m~-M6?*8uj z(rqdNApzG$(sW!s-i<#sX6^28YSSwsQM=ZAhho66oYGB`szr;(k8nu(9i(s)r44p) zBXp~=OcB6KdPHG5g$$|Q)sJ5c($7rz$o2EpuKpbl$m%;|E$V33JmpuqZy8(2wn=Met#QufH0hIgI{sA&O!)95!@e>)v5;*feLbgN zXIo4B72Y3kbW?$_imiqWpX~zfgI-oxQmo^lcV8hB>A_6tm;DhKy?Fr~g zLIXnCY|Q0#rj22#`kLC1mjDPY3`=)JG9U#r<;tqk3kcmmJ_QTz!S$scR1;`IJMd1S zZ1VJo%tdY4oK-IoP|=M+d1-790mJho<&5TUv0Oe%OY)tbrvrd;l0I7Jcz0sBmYyI* z70W@wH<8K!x(92-pD8(V;$VPeEQ^IW0#D=%zAR$(==IpXQ7wW74mDJTt|<5aaA%tM zkhQhSr;~mLTn!y+y420NdoIf|AjRv{@O_9oB>>{~4#!U*(Bd3Dccd6#m%1*OWe{li zM<-#=wq~uP;^x^AX`D;ZppMyw;5FV9>0BH*v5P`9^E`!*d*zr2-v7vfEVuX-BMW~_ zbs?_EZ2D5j!%S5tAOx4)RlM_iJkRme-6Xke4|nfkdHw-#NM&J&XgIS>Z4!%QfT_9i z>PO!Gvrs|D1&0Tgj2Gfb66TsW>C5h!KHbvfd4^+2qzmV2!O?L}Ir8`3Huq zG9$CUyR%3_F73_$N@#LPOJ)){l38h|z1h0q;rz8r_vV=tX>O#32)B1%7V}`)Ijg8A zheBg(PvM`ejl_8eW5Oiw>H85yV_$6(Q%?JFz?fj<-2MjuF4#;=YKKg41SmyL>BYBm zL}>;HF>LI9+PnM-a(Hq_A<~m*W8y0JwVuz(Bu&VAlj~4u7QHq`#~Y}!t8Dar1!avx zo!&v0#MKkOn}`Ah;x9^v$LwXCAT!-ySwKKflZLZ-1{1;jLb2ng($pxiB=keN3l;Ry zuWxCmeO@(%iMfkCQihSk-EPsRs#&w7xgd?js=8aPyK55#cRjm*l&a0~n&(#oo9j#) zrz)+2%Lw4>HpknqW*#B$mvu9{*PYBNwNpnJ;Wg$Rlj*WUm(YNVlDTpey zqV6g9*^0O+uF}I}tR_JZdR6@5MhLt2kNb_7VePjQblNR3iHrM=n7vcrigXVUCv$Tp z>#UcUtM}lB)}Rp91wEV50_s2#P)J@|2?6OcYg}=1p=+tk4OVVAmfBNW+I56L)-Wi; z?_FO3HHw)8D+Dv`9F$s6`si*uWNJr)kCe81coJ4RdX`#~ht#F6yp@#8>L|;gDdc1? zmTpLJ5*~@6woP3EuO-#-O&@WvT^3Hi<%C|3n@-#aj^gDP|A!qf8R_2;Q-PmB?uzO2 zau=MFK*IgO{G@eG2-uF#hVMa)qZ^h)GN~eaNBDckKM%Wq*M)NRr0Z5G)9~8xPrVX~ zmIO*guDy9v8X(YuFpg}p(D8ckKmbd^&5&d1b+8({{%f}+H_*>U%{IR-uG2@CkWaCn zxkS%v$iKiC%WWS@ZTfCYOT2mX8tarD5n<5GpD7CsTVt=VD*+pgxZHTST#F3=#=ndY ze%=lvV2NLV?Sm=uPGLdoXBDZ3qsg$4aU}yyN>xDarYKx{AUm{@@HjYP*Li%Wvl}Q~ zpSW$DDo>KX4@-~>S);z1Z3REd2G^F`P1-L<1UYs2=X>0}iV3cf3^{79sljdTW~gMFe4&LHnL*HDw23j;8^&c`;rP(YS^`RB zW5N~YkyOZai(9sm9{_%2y#^ zyNE_N$7kqB!s!FzrZ3$mAU-NV9gKOTi1?JH5O9WWa#?)J*o7ea;|O`S313UjA%Cds z^9ov+ds0%p0U*ojRihiI8-%Z z94t-~wjK$_SG#(cCP|K0VWSI(FFr|;6JT*L=J*)30hy(D(U|^qYaC0liVNeBhj^Z0 zGFv-C8A<`L{3zxY^!I^rdYhwrqU)Xbc}`JB-U^K%vUOWZV8E{UXnpAwI#Mlujo{5n zW6W`+VgbAya$F@_5lhrb@W9Lpeqdd{>Qi5KED!y_D{ z`KQd^i%#q+2UbX(!mE4TO8aBRZ{6M6a9Z$!#I}At6`eBl%V43*)JR!V#AuKSc9Kf@S`@AG|Z}((b>4te?EKUGC|@k57^~n%w~u;xLB^`@aV2Ixe)~Hj4rO^Azu7 z_rHB>cT9s;q2G>;&%_&|fSd#i(=XcrrYY6&S^vcPn>1B% zRX#0i%kqPH0nwNy+P`UWM?zZ>Feif9mSVmNYGjP_X>#9Tp!2rx+B>+#%**H2joX z&f@q<3{}iU4Za;7GIH%&_)3lgsi@p(MCu7W?*c2>kevGWl}GPUBic{=PQhk`UJ3_$ zoPmOO%nvbtgYR)9WwsWC%_SnCyA?f_UTDR!bDHed-v{oO6pZRLwWK~V5Fru6yMM#Y zke8c0gcmm;103TT1ajv%qeuU;_`u6Qrp`AMjC(ytifWAu)j&^?Mm6ZOdCr^5u5_gP z%BWbBw66!pNDP@eb-XcL$oS)p#e4b&I9#j&M-l5l?AW9R)D3?X4?Ftf!08E0EvykN ztM?9Yzu_Z_YRJAraHYggeo{w#ZN+71LiHbN1#3OpKdEb_&1lavd8D-%wFwMPn5CFh z+x-+ird0k|%eyl|Rk6CZU2*qu5LDUgb=qpTrZs&$9n;k{4TeN_x9S78DdSzcQTysH zCYj;5)!^Ff?$r~AOmcvp@hj?|DC7FwyUGbCm^j%E;GZsYa`7%ZatXwQ@JCKsb$~%Hd`Hq$h#ch@_MV9x&A)~?5%0`gPn$Z>$i&8=642@PD!$6O zQdW8Igd#(O-i~-wu?(imXM!F$M$+?kn)oI3o%LV%I`1UbP9eTd8H%)?43r|*t@*Pg z#vf;V1aZ&O9->L98PI;QY?Bv&HRnBB z?Ba#J%py+Cda76S1;jp}nX@`!EIYInZ$pI=d0-`5vrX&1;pn$(~q7nCYftThvFsO*Y2z$ zf-@PQ_@BQ+FCF|=dB|mF8!8(5#*PO;_kHGxNt7kOcrPmUJ!7$48hqt}4}Bg(w8KG^ zSL*C`8~QM=$ljR!+qS@BKx9!JQt_XxWEZvekV-+(B* z131#vn?TOM@*A+kH#Q5#KI$;Ubjc1;-?*Ll%lyhESz^M)F41(5X)-94my-RpPyw4i z{pMuG94Viy?gI<6@a1`1U@G4?ol>I>f`d%dZW@+-b;jn*VFEAC$`)=uvkN+J#62)P zh_AAJGzgr37Jst>LHoi#$Z=p~B=5~`{5UD#Onk58>As9=^+uDX*A9KZd<&-rxx$9) zZ!z^n_)emno?_)%eTDDV+n?x*c;lC;SZ17Nv;Ux=N#Y$qVOt*BKC(o5U}6Eg9Cf)D zQ)R1iVFT0PN0)1rJKHH|BVOB>J<^}C!!GRV%c;9=QSM!)kgxW*)LitEOJ&;Q=r=;E zlCYcSjynu z`9Dt&O`GfFY-(9wwPh%*{7~(AbXC&CV@6C6c{p#5J=bH^cA{SP;XS*@MmGhlkyeSPZPzzaQSQeg_$q2}<>wnKU3mW})X z_L<#In;*X2?`OihOJ_}oI!{Tt&z=M+Tle0>E=<&CkeC&r&n|GUx}F6>gJ|h=K^fhu zz4C#(M&so5xU`l`w=S=?zo^8HYpJmx+u#!ZuKtHi&x6Z)78>nE4i(IbG&7)9QTg7< z@BjKHX>SL#-K<}5ojqjBOw*-R6pAT-GQ!FHBQb084UZic>OWtsfRIF-HE)x4k$cGf z+3ZnlSZ?ZYz>ji+SzgGCo`3-7M&wf6tXFb^bDPoQ>AKt(me7PqQN_Q}r+g0aSs6P& zgA^+Q=2FK=W}TAnW+isC>T^y;?Nlt1Q!`ZpV)op}jRy+a>`1TAl>s_{W70rbU$Rx8 z&cMtweZkGvZmu5!z<{;mhm_|kiunb$)^NW4^$u6KZD-GXn~jJ@rd))cS*DDhMWfNz zLx~F_VW%zH-mS^|MlqQF3OrUkB|ONE&V)_M<7c#)lymZFq#i)g83T-{u`i zor2zqYkbpq#ZXC(6LUK{M{MU()9?KUjzjkfQ}E9?ym5n$pB+_pv}$ucqtF8VLT0ax z?T~irX<#eAqR2vLcWGeeIT&csNP`#oYrc%id58CA=x>g4`?sCNh(#K?SCZee!}t8| z61?35;NWia7&!6vd_$vWBxQ*ySJQa^IRn*DgZKj+_^35UMX+dj!A05c9&S(s4!^1p zR6`-Ca$aGVC4L3kz}JXAZ_pR7VWLbq9?VZB7MOd!doQTx{#lq^O(zgkPiol%cZe%S z;8)F+5FVSL(E>w4@{~$B))6S~BIxLI8b=$Q!1Is?8`J!W|9&|5p8q;>g8v;|T-p>^ zppKv)vvljfAFkS$M3Ksgzq|9_(cP>8=uS@(<4sfAVPhBH`*Ex3yWzFOEx?DCB-0 zFstwxN337Zzc~NuI(({ajsZL2{-o(!YqMDR^^CJ0&LLfNt-hOK@3w>gPGcYc3MDQ6so*N&#DJxaw_IT`TMaZQH)}1~@aYSrW#11D0kauQ z`lVZMcrbYMw`EVWiGDL^iQlP2;*=@kcmk@^a<3`zq0&R{dS&h5hl|$m=E$rg!Lzs8 zCcO*fgJ(l>`msS$(JSwQ_Je~$5oow!)F0z)=Vx_w=7Q^|U;UcKw@%QIS7ct1^<;$7 zZ~I3UKJV6s=bUbR=wKF}l{hZeSWRc5mUC2c0U|hM0~rT~|0f&-glxmz3J3*F)`DrS zGL1TqP&by9Y-Fmc=z7!xBYmaC--aW1M&(Doh733Qu~QnN>MC~VZykuoqQ&xK4!fC0 zQG#X+UhY{ir9-)v84rH1Mj%# zkCug>Ccn&t1A|$fIQ96`^cTBA7cPGGXU~dQFvUNzx-VJv<|IaVh&|JOrOk|Fl@@!X zhQC~Fiv5%YeH*D&)hAx>oE~uPe_nz)FDB8!^viv0rt^y@lU0IJ&DDkp>FMcoQ>K}@ z?9#+RSlRNtKoK|QK{g?&=(#!Rk!O-sYoLv$1_pO8B_mdSj{RqdKNuLfPvelhV%qBM z;)~5+mU%df$@y7JGG*=C&tt0*kJx0j{>3l+rB2xlU4kH8?c;^;ymO@4kVe&NH9|P~ zR}Taw|1_6LX*E7q^M5hfgn~Vxv-@0bF@9s-iP>LW-@DfXmD)hZJ7sXC0tBhhRW{+t zC9J}~Y{0J7A9p#IfFllsU(Pi&Ul(gG=rv1CG38Z1QwnXcH9)q8r+Da5!>_v7Sz}Fx zZ5>cYwOLZTnf5kMtZ#Nj45wrYvP(*Z-Uvc|R2gX6_bDLEw!$}t1!QuTPZ6~zPiqcy zH7@1xOGaavZ6e-KCAPblEjE5pbjRWNJ4rJd1o%vkjl^2 z!Sd=(Wc>Pwy!qAPD02|3+)8Kv#C@!6b0kgg$46;&2a`F1x_=-W+=>>Fvw7j5l|sqe zk+^mMx~XJ}7>^>3Iw7j708|orKzulN=(VkK5#5}auasYM3~hBW!3qW>V(J_(A-pS5 zeldx;Co$XdPfg$8>}=}y#j1n^#XI+6Dx9t)#j>4}XRZdvm}T$KAbl`Y{GEgGi5j`$ zPC+?6QLI?{{7ERYN{+{kO~IXGMs}h4-hBe^?F8m`XxnVOg*0P)e@y^(j@%({Zr!sg;zl> zQj|07>OGK!Fa+3EQWSB{;FvC;S{xqle|(sp%XWs&F6~JmEPCZ~)tal(#{O_FxmL1b zOfnDo`a{}_Zw~1h@#h~ok#H=Vtz+(aW~u=O}>WwtSGU-wlw zFXsF?(~hA&vz#|PRpZJ?3BR|eL%&u|L86yoCfR>>bz}2!-XHU-RNd_o45cu+Cm;F@ zDvtCH(B9*|I^;eE$#J|Oc(z(v-Y+@(k!87n$mX!JEjn2C99r;5El*Il8T#^35yYld z_k+~1%q7?XCflnRiv2S22vbc2=P|H8vi=RO0D-_=A80Y_4h|2IsewWT1jKv^p(wH{{qWoDks6H4_O( zQkLN%Rnituj>#WlxJZ1wN0udwQ-uS~&}WIo^KH#V1q@y{+s}7@6~S=YgfUJUB0#SBbNG8Yv$XqQ7o2{an2KFapggMl5* zbY>BdDkwg?K>y{S=>)<%RscSp0|FPPT1~c94AAN`FVHsiqB6l+<;c`srh)d&@iP<0 zm?@s0PsS&8O$a0QzmwoL)!4$xVKW&1X4-zz{``B|&6!AejzaW$BK@4dvmK}OZqYb6 zUFf$b{Sm_0_w5DFZPJE6Z)ZtcMLUVr9i^UruN&k`ob?0W>6JM=Zau0?my6^Ld;6$r z!=U2nFE*RGjf2TNW8E-sy=nIkyr8B5f_WmWsv}|_<>f`BBsz(Jgc*uYVFd^mfCvjK zf{RjP(8$svkPO@@@BT3ZM@!XoICaZ7!_);mlW$wHCSH1M>FueO>%EP9$BWT>jhy~D zHJRP%UT+M8S3LcjBWAWbnOok1QuTLI%*cAjg}G*DDrenn|F~cTjf2U#-*5WbG+uyX zQNfmZSbTt`R=P^E_pO*2e(BnXG6`-mw4n6>wO-GO_T!&(6oKBr@Nh2Z*A5oxL}2Pp z0zoX$oaP19FS*APfPr!BR7?d5b7?AqEzcel6M2g~)$7Qg#yVMi!4Buwot zNAaj7Nh;P9{X%HUXXtd(Q;aX#0mpbYXY$Kyxx}kCuVZ7YKo}d{wpl7)@y<%Q4Vr*q zW6y-mU~NKvIwrOZG=5a9_$A2c>41Lg#0P7viXIQpR}A}|<)OLdTQI-pgzGuCn*|m~ z|7VlZsc7TA=V#1o2|EI^>QOZ1byf*#d+VW|TB4oChfrB$4tZiNr3IGypK98W60{?r z9oj%47g3}Jb{27gJ3wh_1_H;g0$$6`a=M<@zOLRu4CR$H#1Oh+QDZ;y%)Ygsu&IEj z7`}BPKAn2g|MV$Ai}&=z>m#$99!3|Q6Vv*w0=#)&NS-!A{j#^Kh}0C#S1zsWuiuXE zdoKBg5A&C`W?{PN8-kf-C}S|Xx!ii_LgsXE$71LA6QEp~%y2+`M`_U4k3h(oF@KLH)3|)rn}$mRyo7 z%c+NtDmxC(rDbdklw>xoLk6O+)Hzl#+CAaRMfEqF+|Y4IWK-~B(Ce(FEmaTr~aX^zin?Df!vc@x>bnLQpqLqytD`_wKe^$tU_y>x|d&u-#(L))dE?6f650olFq zvo<|usFtKS5#pvdF1pS3-2Sb{_$ZJ|8%w7zpS;)i3wEO7@_$FF>8} z-2G4*s~l2=Nlde~7!xO=RpMT+*_B=?4>Y|`*K}QSk^7qF&kxrlPn?UJ!Lb}ya*k;8 z!1CCL;`!JdDak2t`jfJI#coF%SgYUb;Jv;Ey?U=5wYzz)=ga%#xw zCs%q|><*(Tj!e%A-vWT_G~)>&yzu#3g##nXQVkYQoVVXH$seqL5k1m}ffsmRqVJ$Uv&9A+>(E0#xmWt(bCTq=Dt|t_*%Q*ZGxumr z;*J-%EUUN5j?HAW$DvWB zl3R9rrL&dxKdQNkWYx=Tpe28=H^1F-d`2;Z>UjeAPz)@R5)vLjN3#*31N3_tg@r;h zEdgagLo@9=gYqPrPM`(9wsXcX%p{F-RHE}H{Zy9}$*slzLjlV8s*c*l0YNpZzNA}g zW8{l3Dhb{FFdT~n?7F#+dWDLIS27M&rQ<-{FKoYBPDhqT?M>d82=~v25jmB3ZLCB5 zPySsH?mef4@OB+HLCPO5e?M`Cp0}X*np(ZIrY~tI*Q9La&hqMqKHHV``U#J&pEnZC zeq`yof9ZdsSAGgqV3#@m1Duf3a0G<=F0kEr)!M9&ew%;B1bEDoEAm{x+c>4>s0UoV zH>~5x%y+RkW@ID8x>l3kPR{UasX9b6=x(f4f#PD;=`5z6$)O7G$j1jp#^*D;T*faC zU$OJOdb9m`w=E9E`t4c*N%s<~{r+5tYl3&>_i%7^ybh0JSW`*-&|B{<436ZmUlrF1 zDjo=kaASsx#goU4Yw?05+csxZFF=j}0Rd6NPRHw{=!F|>=34{S#ub1lLb4)qwS-Xu z!G_7#H29)LbUb3S&8l2MMLT6PaM1+$wuS+mYc;>Fkp{)DTY>w^q+gc8lx_FJCDN|T zz5Khcw^V*aAe+n4RUfok_1ocS&4n{Z0aI_%l-{3!U%k=({1paHuUWx|N!yh4cRf9I zamBP^RIKWJaIB#t5p=j&rfgGpNJH_Ar}}MOD@fSe5id-0!Y+10bvu%FA8>kU!tD-V z(X_&I8%W_|vAQLbdEkteL21I;eW1iPpw42&swiw?itv--$FchEM@;4}xruHf935n?2g`lt5(%@DcQggqy^$?Pt-fE)UoFq`R_+rJ3PIY_<9o2BT+J(o(k77i zrlejeTxEpQTOoJuO(YxemS|>*~2!AZ&)Yd=0?@Y1%4ZN`!Gg;n_nxqO!xkhfs1^G~eT&pZV z=A0@jf!-g33Qrm2?~m9VS4IxzWHmJUmM;!hjc@&wgG|0i#IO*RGSd2#_fB7{S1%|x zFkRGn_a$EuWWJzchMt>!?#M%JurHU}PiQoky1l{a=6<2fr9Gun^35hcbMVbSpkFsC zNUPbm>wnsG8mm`j=1aKE1^29;*?08+p@^~vnsj7m5rab$9_Y(`s{e(dmnKnJT`eyRQIKU+% zK)Zwt2zj|61~VV%?!52U-5q-h=+PjR!7$N2u}hTgiT(DM!qB)(TXutRWa{8v&&Sh- zA9`|6J6Mwr-(_gol>wmUqEw?+-uRcT1Og&B`=tC?d2u@(O$(BtvDwxq>&%9q-pkAL z{FSVv;wAE4lw088q|^B0#A34gewO{)-d#ZTpor7>GnVVJ$J4C_pYrWc+iWn0 z9ojqmEt3CrXa+EWv;?ldvtwyqV%zJq6&IdMRo0NN0eq$%Io?;3d#B}k6<4m1JXNyX@&Qj( zE~muQL^ta~kUed4sq4F#Cl(PeJef(v^XHJ?RA|csowG5P3AuxbV5SMESmwK?0umAB?_|eJ8Ze6?*G)q_K%PR`@&^G=7MS6K zz$j>!i~&s~O+}@ZZ`I0y(}zCS?#R-d23(J-*6<~OLvsKynN(1q5l~t%7y|Mcs-;rK z6pqu*f9C|QAgw?*F+!@*64h7$ZkHM%zglK8h0mJeyrET}*A|;qrQWV_#TeuWjFU;+TR)E{vqQwGobVEjuI6qi(rea+P+mu~iaqG^kAZ znfKuAyDa~OR9LWPoN_acw9LYs<^}6ow1CPgT-=EB(2^(1&bUL02T zP5M}#I)1~s)9~pjr9IVIZqt!ZbH2jtgOUJ5vo@=I z^FGv~$R%`GWkH0PHW(dQSAJboQQW}E^m58a#G6_{8oT`N%#(Fk9_J~;4(X9%LL9Wn$&UsP%ZmP1=d`@S3YL<{xp~G zzt>U?3A&$Yd;jOPn}yE-KpVFnQ9HPY#}rFLK66Wo>smv;WhnsIO-p3cfdqcfe->dj zI()mA55y%KrS`iS-3R5l8OVpQqa6lHnm4W2wgKwEYY5-00%);tqah!?UWkR0*Nv24 zrRLxpLu+R0V2|G5ZJDfg37MPXiAzR*ZuYpQt@cjarX7^*uJClvi(r6}%&NV~^)o5& zJ_y%$l}BSd!6tmW%6Ota*Z54GV#4~g>AR+H7235{6r*$e$UjCRhFr1Eb;51{K@Xsx$9@5bmb}GN~pBCZ>9@FH#d%?5GzBMBNGD}DQI-Y8GVJ4NTH5L4LDKz2)U|4 zn*XoX&d6P7wrT~*a|UVgXW9Cjm4Jit|F&`dx5M@Sw&q`<6cHjuOzBOtD-HVUypAU^ zlPn<5BFhvd0m@T1iw9{feZ}?4df0L*KR*2pF^h3fQM@D8&M($8U{b3a%HWe3$7=> z|7Db^edI^FS-W5|d$61L^JF?`ipvqN1MAj+^Ve8O^z;%k3vB_p1Sjknc#gjnJSXQF z(&?ttwFh1^;;schon!=_a$!zEDV|PO;Gi{sT31dBe)XO*r-nTT6N%wjefFnQJ5sxq z&AI7>Q`5*v#cFbwq`=PPHr>oI_OarXk9o+Q_YvXnsgZ7Qr%Be#oW8THJbMrsFX zBVBYeVZ|K_j=E8=bhp1PvW12EpG$OXu!EfRZ!%GsgXO3`Kvq{$8q8mmvE!jtn3B4i z{m&7zt8bq~-t^LR2$ZhNP6_)04YKU?mB4;x2>W_CymPqCcDR;xeW*{yu=1bE9A7FOzUqBRLD9TCJ!Y>*OgaD*`P1*Q+jb~a9*An zmz_5MZ2J#8wW61P*cBJGr~ORK^_@mO(sdXx5Um(mE|wZ{@3T=5rUV!%0Yc=<^<>k6wofuP`5`=E|>>d5)9*d97-h7pS^KMs1+-i1Ot@liEWj zd0VZS6aJ~?R`XB+_j9w-OD5N|I9zIVsLpX*U(HA_wvF+Z{|4b#b z!M%dc`5AKKW5dEp>&?nNNaWja>`cT%a_~i&+O~@t;Wc^fMGq=Em;b0Ls^7&8c2ip5 zLB?pKro49_&YD&$9j>Zf@(jJ*wANs$_4A@qdUUqZ^Ypng@s%nVF*HX2aFEqVvaO+$Fx0OWK6&{MiUaTXN1v=QWWvQ^Jz@DwitzV$YSkUr=m&U_6 zQigJPb^D>WSw^uWTm2V%*2A+-e-2=98~+z5i~Td+<$k?P9~+G&>RlxjBWmnuSBwwa zf+bu(IK1QVm&aDs5FV@ke_HKdx&IIRf?t%I{r@lKfbRc+6Y$vT|IpI@-yu-wApiRm z{tr*VlZDh?_oX~|KXtsa&l5Z6p3D+iJnh@D_{W~9{TU!9Xf9mq`enB{vwcRF6#!Q9 z+Tz5YkwvgyFOXT+I<4}q32(m%!S3uPs?8kdZ~p-}Ju~c3#gdF^?-w0@xv$qV@ZzK3 z=H~Y;BEhNg0>s`wFZQ>ORCM$5VV=M0!pK9)G;fpzqZkG~?CJ8jaYbz7O)Hnu&doyQ zk2gM!vz(!)G?6p^FZSLtEXuWCA6^I;AR-7-(jg4p(jW~|!wg7EONde;t%!8P(48|w zgD6O;ARQ7T($Wpm4evE;t!MA2zQ4!+IQED2L39oG+*kbS3}N?Tp;W(pgIHnfVyj02 z`Y|bBUhS=%=`t9bcDC80>*w04#kR=7Mu_e(0#P6%jb(L{+qT^#4P!|nS}G#hHwoIg zU!`a_MK3&>tq;yYMrs0vwN4-_MvMrXrG7MnL>l(R?WJ$W4L(iCw~0=SDAdh2sQ(t! z{>pz|edq%rYxTFAst%GrwlkH*r*eS-DfM2yd%lk?9R{B786o$01=SyL~jM=v*SCMrKCOne(1)Tu?USlr&L zav{lzoYi+sOje9yaa>&>74T_L&`jg3RR4i!$_I>ipb_PYhw)jV^sjp#mBOaM%p1F2 zAs0tJrj@E3nRwt_jhSm}uakLn2No})Hnxn2d+^I8bVWpm+Kp~4gr9WCH;#qafPtue zDGh?cLN|ajtYWv(X^ZvYB;(HS(l0y1zBw*uk?hAy4)Hr=U_Q?6!0I-Kfcjsspd>a( zK^uMc(QpR8+Cxz;_86E_`zJB*DH>y3F%dRH!@k!xWd^CE<4f)e7FZV{f==s!Kju1U zL+PA>De(?gzZEO8dtWm>2CJw0tp0WnDGZseW+!BL1&pQDYCb_Lq2$}8#`5Tqn~3E6 z%qW?B{ZvypZ0yY{sy3mCR9jT%42`GqEexJtmDPN7O(-TmVrH=6TwrLbdNja(gnf%MAGRcDBS$dA)rBk zT2nYeMkYG8ed~JSCM{ac>apUs$Gk1S4r0E`0B2_)oMdsHgN*>)G5$k+!x^{0LOc7! z!=!IyganaqTo4AGY7QcWBlAHgdeR)sC*@?FmN2B`(*|d2t;o|`6RtDT`5)%|SW35F z*^N5C8|z)4oXoz^os^y0^Q`mpl{y}&2ll`Fin*q`uz>uqdDf(2DS%2j-NqpwX)eM228!m@3vR_46Ct zo$1SqkvjcOesuub$lzJ|B9$+xPYHgw&_cm9KXxXo z8dlCCfBGY4$gDNNiyToGV4pEeC8OeF$z0%NdgsRY2Ze_}dU-?k^;M2)y5R`KK;`Bw za2JSoZZvc`m&Y=ZRBo_J*#=lGnHHjv_86YqdMUkyY2QuX1$6O%OhK6X5cq6%M^Fh` zZH(0nR#~fuhK9y2WqSA3DR=U@hS3-9VJ)?Tr&Nk7 zoSz*Hka@1pS*<*%NEC83)mVH^7>_%R#blK6`G!r8@W*AYbK-jGC5;@sE5_TBxo?5l zYcf`5dSD?-atD}sMvwWS0vmhnyIHUciECV>K@CMp!&g$f`zpCbM__ON_d3>eBbzMC-{se1C zzHp79ejzA6Wimq|M^4Ww?EN0GtX#&qEal#=6#6sv`Kw2E?@+pDhGsdN^#p}`VwpqE z^n(?hRvh|e0j-65@I#9N@5{gI5bAV7v$i0(g1uRGE{L@o>BU9}hafqvnT;ATgdbw__^vg(eCYqEI z5{k?F5r&UI?_IEJ!c_~{PP&T96~%Y(m7{htEKi$5(Nc0=apgT3hU(8m#e=aU9Ju!x z1B#4}i+t^e9vp@l7m9zKBL$+<*DI>-;oIOeRHx<7H5LE5$ou6-cxaHkNSQX62Jo=} z>F085cd0)qBqW6JF6`*7d1AH;(O$vB6}$aZMyL#z5ge9sd}b5;l2N6L5`FUg8_OZr zs~$WK+d^!&&EcR`<&dPNK=qVbz)i`7%3INQMXE4rRaF=a_a`njbLZS2+lJ9!$%~;v zAqv#T(K~0BIrzo1=@Ie$(;;y-qkg21>l+-+O!RMn6kw7ZY_A3U-hP@M6E^r7V9SPw?M9jWS+1?6E7 zlevO>=z+`*)7a!D=R85Rt*ll;8!rJkfshO~e{Zob!y8Z=Lv_`{Mg!2xx) zk(7Rdy>OR$n(2kTwM+*Gr?9F%bG%GjXYObXs~|?d?N%w{HM*M!T;g|KJD%<=Oz(36 zWbQsqB7WGJ9jzR>>w^7_^`9o_HdeKtwJUM&-Gdb#r7-)==m1Dlnb*DEA);sxeQe>> zMG3fYq?DAe1_JDNhHsF;3dmA!!{&BaKds?ym(nqi7yn4uK|s`HLHjs6Y84}U06mMek(VOe1f8x(1t^ER6alT|3}PS(k7ML2QjG& z4BpENuUovMYTce|XO%8{gQxtGPeWTyQy7d|0$S4479cgO(l!2HEZJbBo)B%D@mkEi z8zoshe(*IFrJ>Hg_8!mvLOpF-EyTh{B{jo%^k9iRk%7NIZk@S*3D+kc;{ax#0*x)h zs*P>R(L+QH1j&HX^dsfg?=3FXYm=jL$3PZfgQ5-o?y4FkENe-4hI!aYWVeXQ(fla1 zNezBuRmNCYq5Ytua;TM8=;2v)n(EupaDoG2aAvV`K+ra<&Y)V43=k$rIlDQ8U-@wN zw|bC@b=C=Z<#*RG@0;e$!&&UU``qZKVfS-?m~UfMm4DOBv5{?Mn*2SpY!=7OLIm#? zGdbi)#W1)0=CY2)*!$n{(k5JN$lEbm68GMe8$S4ODsnv@lG2Ht0VqtFp1@%Mx?KCw z;ipyZF#1jr9$a%ua>=s5_wS3hLo7i-8|&vAmyuEGl6+mGiLrMhYzmK{*3Jk<2P zV-cut zv<3U_VBfzTz_wB^tr`mkLaaQ!T5Ub$y#0PzPFNtvipC)Ab~N+F13KpE#%bB81~QH< z8G6QMy2!+oI}WPxC0#CaZC*^x-p$!x(Z0wi^t5fo$?1%4Q$xaTDc1r-A{WZK&7rT^ z1xIGn5z>k1k^V4!_KNOLj6i;b4peR*$DQ4)?QYZnO(Q@ge+>IDUOXtnVeH?X+I>uPk3IIPN+cNaaIW)qzR+ zYss_UEs)LTOTQb0rM%_?8EPila>GyHp%qBu?P5}f8+N#GR=C_983zK$BcoPW(!V0B*6(xI|43Ln_|_#jUC^1?uWNlNYFG8dyH-f0 zgiF3~1o2kpv6eQ<@7Dc{H;Z}K+YbTIGT%EG?ecp-d%qeaN(9vqq(xFmdG^$d7~xs@ zjw*%i-Vj#R?$wVR>=#lkr{k<6VpADu<)rPGnSSDb7z_eG+jBjlL4E{r@&1g%+^8rT zf=2H%<(n!QsS`ksr%0L~xWt>y5xOMM({5`87)#eA+GjRds!OQ@O^MeLP;%(gkS5F0 zRdsUDCAMw#F!JX42Pje7hB;@48;}Xt0_$jUFi{0q>-$9ft}V{x@E6V2ZT~1`m-wDf zf%*|lNarV@E>K-k7WP~d6ahr|8~5|FK*f)VtHLcTkfv(;pGBp7=7lBvoWGkGiol=` zO{n0r`bYcAXZsJUd2ZtaX$W}n=EWexfL5Eaj@hP6I!iRIY9ej_H+55D|C&$q)CKa! zRGBRrGoc)Q?WU0l(Zq@!TJ*v(&COvta9H;eMz8p^)C{VuNgJ-$nFA)=1OQ{em{^+h z^V@OM?lOl#o2lgpaZ~KVfbDs{NBn>Y2|kV{2=TDD1k0E_a3(l%a1fHOD(WOn|*yP z2Jjq;XZF|BVW%W&#sI~(38t4>%$G+8lUH}X?2@?5X0RX2C}#$QZKa7~;`gk~R;C+M zeOw=?wZ`^S(t;HA<288GaF&d!#vEb&A|$ld8L=u4_lC)EnAL5$o@%*?s4BEd3FR&6y7i9&_=(W1bArzxM`Zi;xBQ-V`j>cNHGwW6D=PCzFCQ04D@L?fqHpGxeD@JZ6fmVxp@xyFz>z@Vq#Yg+J z`pQhR5cMtonuU`h3q*$i$Y8N^y63?~o5Zo?yAIvCqc~MIIZC&KmqYLc^X3)^4zmpt=P7a|yAK(TK zn;in)=D+|Z29(0x^2Ad45ESqpT$ngLJDa{d&k_qjFX`Qgwgd4>JT4Q0gT92ZPTNDe z^pPy?tE}ouLmY)2)YVtcKU5gfPgzeK>isV zY)*25k!m6k-&e<=;ntmTtZ2knLf!_3=Chq}*R5Gxxdo%K0LV6h9lfX#b3&pw^f465DJ~DWhNoJ97A~mrH$joy<(=`w zWpSSu=uEHr_8WkFhVU@a&Mtchm6~LAs1GZmh@}l4cZ1o!i$?36L}s&F(Se;gu)TlQ zh<9_1)i0DE%FibY!5b!zfn=&-2M z3S@26V16$m>NI*kXz2y$n2+KUKym{ILF~W(<;$0@h_yW9I+sD<>2v(Mg*d}Iiyqu2 zE!E&;KnQ0->0s83k{n1$Qf#3~4C=DW4yfwvzNtVeWox7>|-b{d%yE*z-j;s(lm}7ia-7g)_tQ^$=Y{dU$hXBW#uKQU&xruB) z)Oy0-q5{-}ms-wj^C}7{TTTW1ny>FpRo}-Wr!L&O=pkv>&6xSQctqukjyUk}0I<#+ zVsZ!yb_dFRPljg0$WVt353{3zZ@!mYk7YkXuO0!A(EQQ%Tu|L;=$-p8nxHs~vTVrH zI6uuN_4sM|wvXH-+w&vwi3a->zKG17tMdZJk&-9D4n>}?I`bpI=!_YABOC8#`(&K^ z>S>|{*pm@k5(HNcEtT49|Bm?15@6_|ToKSE{RLd!*DJ67{YNb05@_T8GoAAPzi|hG z;tl42U=FNa5>U?8_q#mzSi0K}DsVvR{A-4yGDC*QIzL$}-vR%1Yss9?PhiD3|CJbu zNZJPOM4eHz5`V!JN$P;SXn}eX56B_80U2CRo#gVc`O;n59=ID~DVl$MW3pcC%a{J} zd|Dc-LyGzE_8M^)76%O+s0NGmc@g#Jpf|e8ul3gh{S(WyAp#320RqifiOkg{;OU}g z_`#6_2vuV3g*t(}CH4kd87jieW&TYZ&BC7Om|QWpg(=JL1hsn|xCH(iv9%!r2ydzs>y^S#57fHAZ+zL=*%^C{?v?DhvI)?D zRto^aW}-_Zi+{0F|9*P|*)9YEtKeJc%g{e~Aq&^gNE6XG_W`1G>cFg35`^+q+yD0U zKxPGlKQedQ#3GMx++phg5~*0kE<9T1;HTy6)>gR}-7An4U?Ca4*jnOY*j2;^4i1hj zBp*oh-3MN^aX>6p_uL2QqyjE|l;rVz{D1#dKqgidz~$!wd{fW25D1rI5}LM}FF|X0 z5{O5EDv%v$QM%BX|Lf8bLGqZGnHR8q4l8>=y70|fOIXP}ToYE!N114NLGTwhb z;s0`f7pxg7<>cKGLO(i;Xab#8cec@EkeFky)VSE#FC2amP~dU;kL@v#r))jn5sRg` zJR5r->$Sg%l2d$*_Sh>_)6`4^{W;BNyJY`P`t9m|0FbiixQAwMfskbfu)l)ulN|xY zPuQJ(pmA8KDxNnCFff0O5EXZUUmdn%%MRE;@qHN*#hE$Jn~n>;ZoUl?op|vUt3Mga zrS7lIK{IoU?G(dNbLehaz)j#c$#g(Egy<-Cfg+TkL%&aWlYs7R3 zW`4)t>Xd;m=U@Eq|6y2rQXv6X(6xx`afR&3;TA7AUiR?hz(BnHUEoug(?3_U$~Sz z010+a)VabtY9pH+#Q#7XRmOrRcgduR8ki zd6wgJlgU1zB0U=5SA}1N$h8)iHyH}+T!M3&uKemhzm|1Rmdxd&JAEmFwq4*1a)MPY z*d3_)o9O@dcFzNG7?_`dOr{%Xys&RI9-5{Y*`@sF$}fvzRg#;Unt&v933cA{sRjeA z$R^eS(UrIVuk-LfcO36i8=FEH3aW@zP6i$HcyRZl(GVeRUe5n~Ex3(EeDAh^8d7Z9 z^Cp8tZX-6Vu(0lV%uVxQCoFjiGRY)d z*|+tgt ze3UA#2?3nEA78>*d97j>^4uHW4c7J>cBa?r2(&v@Xs|^5gk&|$)cT#9n$0UK_gU;8 ziNWSi*;VUoQ-*TZUh6td@l?yfou1l^|1ra4M(q(6F#a1az%xYkQKMu%%O% zH(z{ORer}B0Zb3ey1H9hTHx`TS~zm#G-)ijUhQ4yz^I`9`39%QMupRas}X(Oeh{EX zpW54vc1mL`dL>Gdc+Sh?h-EYq`!!oT2Y5-j=-Znk<3S%f!GU5gp62dhEKTB1W+$Tw z-7;fowTB;_60REk7-h;WpJG`^6OGjq1F2k0E`L{l6rB`AbkZ&UCb-xEBo%lzmKlqA z9*o*%c)tpAw!E+Q0-04}Jgu9!WF(`vPQ&V_2rxS*=zW2?f$ge;x-V6; zDLIMJi2hp$Yc1lWHdj5>u{|HVS1X3ztv$kV{FbHjH%;zq1eL_!BUFyhSu0ri-3z^+ z)0khtqWvdqhj%W?%uh#(NE@s)E-|~9u=jx@m6%Ju+vbD;wWt#pRtg*d`)6Y9syq!? zV=om#&ggh{(%#LL-;RB5nQvK+7;Kmmmj`MUPk)#I*12W1uJ~c?t%`F zBOBF4cfP@ICOfD2rz90BRfb%Mih2U2)e<_AkK(gY`_QsyQSa>^fgK|z10<=Fe1Nef zn&hO+q#jo7wnv1;UG8m88UYPuCZJfF04Qif;BuoBwg}7r@H^JlKPvuQ=E0ozT3fh` zXv{%958`v0sLIi^sOIetp%~H^ElMco4I7l_BwM#~$}9d2tzt|Ac9u?<)p$91a9!MPeTM zMJBX(r>U8?B7z!uN~le87uB4CIawW-3|r`~Cx#pSP@CUaD=<#G2UOV~hJ6EZ3(LDf z=Sp#LJpxMKz#UJq#`I^Eb$IH@Qs)67bsldDSErbC)l#i>;qN z?EuLytnhg{JCYLUji8ADo=E_Ebjvgz@K0_9(Ky9C8~=bJxBPyZ%>}Z@7dVwq{z(Rf zEXu*}0C?DTfT)b#5qxjY56+pkPD)^`oWWYs*MHa%~!>7_mf zDSL*u_bmcQ5Kw)w#|45XH_2e8Z2$=kyQ2w|yH|dcru6`igG4R^Cd;;nSm30Iwc-WF ztHXJhb{xo?JPdb%w8}r^USe{OfdHR12x!nEP1?@hUIfUni~Vv0ZF*w#N)1sPNaR<) zOdz{r4%A`B02hM`7ax&SX*a~y2~7H!-O947&;WsL|E?JX?#Vig5Bq=q2D$jX;bB;c zr00&9??bH&@q!OU1qGfypXmSXRaq|fqzt~d(`^xO7Fq^Dzi1c@@6OX0{x1(bT5TKM z)jZ{Q`V2eEPB$X_-2c3BKM@ZP4{Mp5VI&1!%)c%}H-#who+%}p>jDp23fglxx!D4| z?0^GbLWeDPp+Z>yNRR_{gP6QJ;Ue-gs+|Mb1BPiJ)53t2w4h=~Hbj6T)3mC(+PW)& z15J-O$l3*>;8^ZF(k9*HliT*wJU0T!Tfz{0kd0y`f3T2={HIv@k?`xE&HnzL09ycq zOMKQpn*e5{`$vE%TKEB*Qvd5}Vr_DPH(e~q%K*(K5UV_BZ%z5=yhY9LdP0i#jtuvI z+0sw_fb--_Fwc(zTJqfi_=GGnGG@Mt{@?fZ-0Bt*jpo6ddPz3}zQ(_7c1GYWfapmR z)xsJj1L<=(XG>sUAaa$X~@+ZnEkm4Gjx}z73vhi`0d0{>Q1P08Mkd(W-*hwl>I(g{>`K zF)^_!a8fX9e^i>cT*ZXkxj~}zTM__G{_Q%lh!9Z;(Idp>s{rbcmd*h;e;larAa*q` zW**jbO@pFI>|6i*U-|VvZaKo&j?~~5aa;Z_r&n&uE#a{VlQ}&)Qd#QDfX`NA3-G_v z%DexuBU_V!w6_|zKm=Z)2rMZSs6285Gy$OI0%>qo&^P$#bG|>)*C+*?%>U&a{Sg2G z!uTMaDw-$$yeG{fi@hj7I{^L68g!ceZRfWp15a)2fMy=lRN6Ak_vMA*z)BM{+*}28 zr2g%5(ZIT(%Ch|ysH=JhieV@wkhbqEhxsb~=LOm3wlM~W*R07kqatZUGpS%)>i@R# zR~V2Wa?JyaNLMEy$pu=q`i;b0)MT)$6t@Yl0_CcIC?xTNQyUAa3UPnP9~uK6pHFX| zp&aG@HJlKBhag>CUE$GhyLsv#f#6f@ZP|Jio%?$h0z{|Jx~DKh>|`GtjgMT=)XD9|jhcajJg2zSfMEwD<0#)Ey>G}tF)S2&N&ttovl9~hk)_x0+4DXQey z;paO5iRIeb@66QP7JjxE0=PPz+tpP4=Wpu*AmwXbc9Xt&?^l{Y3iz`GD0noKEq3u^ z;gC}#QU3a?9ud>ReoHc`ds{uY0ch8+K0pmZ_7fX-f|F`E%uTc5D;DiojSruHw#Ks4 zr$M?v8tdT*+v!-Q{D_qq%TLaxVD}1Kb>*M8^_XyOHyFPSQnBiG{XIclZK-3su=P^` zI97z?S`&9jQL|=zV^$IMrWrM8pN_@y#!85$yKX>7%Ey1u&|AJ^$K4tSOsDjEuK8Ke zIqm68V|7^7>rDM{?$7U@W_iliay|)2MK3|vuBEL8{r}-5V;&G_vQWP~2At{Mab855rz>mmw}QA)pVYavn|4^?8Gt7PT2ka-AbVtTd0-de zv*VKA@S*&d$N9mV>&G)KS9khH*v`tN7?9$vW^A7Py7`L3kL~V?L9vL&50y!3?SXYb z8%rq}0tiXmIW*-7*#${z=bS|<=lFdueCXd>1qGTlSMB|2WROpp{q_7|?f1EfHwFO& zF00#l94_tMH(d7P`KwlPo{pZWyf?mtl}w!K<*r}!96j&|whQQTp1j_?2)SY4TZhM; ze$woa#2iqT4q;qdBnyB6^3G1;5wnR z1Z5@*FE(5P=T2Rk5cxKTA4uItY+5WE#fDEaCFCIY+o_wkGB-3lr@?;xG5JH?qU8&; z$S(|5mb);EDbU+^s@h4t(($capggYYrkt={o=|#)CX*p?>}rlPbfElGLh)l&2&@c( zSrv`x_#n^vY)`E-9X5sle!5_rcP*wraDJ#)@|e-n(-SlVxIpMfv_;Z%g9tPaN;Kz9 zxB!?dB~R_o&Y2I^_`+4ycM#rX#Bn1BL7n?+Yii;5WbCHV=(7vE2%s&6MJ=y6f$En_ zIpC`+76*e>UqX#qvM}(;VL_}ei`V}FeB?&{u5UVDJsIR0*Bbj!!sNYqD>lg^xD$H zR5*z@T?Ne@{nM~66PyJb5ZuKmC0=p?y{iap?t9#^YpE#9+I_M)85?Sv$8?*{0LOBv z^~vZRRPv!L>DWP8K*!^`)W!$5Xbu*S;u+?QR1PP%tJrtvXjc4SO0)Sr17u6J(mC`GE@ z%~hduI1rnF-bAhk;2?TCZ8AW$+X~#juq@V^U`#T<=Pc` zkK7Roj_>K6jL1UE50mg%oA0T%L<%x%C)2#X_B<4-el95--!;hUmvJ{&vzin0-hwK} zZfr)W=6yMG=GDjXtkJ|Qq^->0+jtiMfB{!C!|Dsr@ z;<;zTQr@WAO9wsW^YnW1O&w0Hjp+iGTsz6mQ_+w8r}VDEdLvDZ6L-&16wm!}!>CQ7 zUVr9mWfRsLcYZ7o-u0E$51+!U9NpG|52-+#!!$3cAlYGP%vDbclgM1>ZjG$9Obc0S ziknDBO_8;PUde?I?E=2WS5QEbsQ3jMbR1WYj-v(*$hA9{5Z`Bms7R2|Q40E|s_tirfh;*J8DD1Zgj{ce)ZD}{Z#5><-oLQbn}TZO?(S0Jk_$de?l|Q)A+fym zh$pddpZ5iYWEbCWyOKN5yruM zOsTaqWkD+*&1^1U)C1V->c>Ofxa!NOnIh*()2_}@i1`hk?Vb%)iDyHM%EHb3Hq0T} zYP~!nJbJsBZ#(PlpX#}Ykr7#}DI9QSUfB)#g3ZBKY`8MMiG>e4LSC3f(A+KbW9!M> zirP_5$s)RfUL%o4at62z-cC*o`CJ0S%V%Ze6UBuG?}ID1E;CE@+v~6p z9&N0#%IRr|X(RQt?_tL7avhC`ReC!vB!y%Dcl~JXS=;0JVS+E&H)4~kgl;8G-VPUO zL}&Z0X_xpwh|+E)I2JmAYXY8#4|7Olu>VnT{h1Cuj;izZ9!~-;+DrSzFU)4)44{u06q~ z06%Av!i9|Tnec>QEFBmbkX6od4GO)Pkh;gIGVK}Gp0d^rKc?x`bZLGS-PdQYlaJW; zTC%yCt-loZ^s$8YWM`}(a>5MkIU82FVJSk`4pdo+0Ua{R4YB3_rY-e6Llmrxzn&t5sLr(@gA)Z#xuA z1kI1oqD0g`T8URym)t@c?@#8y?s;+E*pk%WF{Hob!w)YnEj%bpeefyF@WA#!(gPf) z@GFoT1zkNcaTV---yCav7~XqIwf5CRX-|h-Ck$>PLzb`1j77$NTQhfpPkAkDI<;OWjb-D|tU zN-OI0T8iCHh*%>D$~EbpFV$HqmDk{Qh(^CjDYMD>JeU51>na+u7)Gg>SLKB`7_&Za zF^I?W@YnAZYUmL~Nynbd+4gpOv-_!9y55?q4-op^l+;EO;YPIDDC(M@b*d*!^CoUY z4=smCGIa{O<-Esr zF09yM7rwINWCckN2(%_@S^9J^D`HvEC3vD9lYd)za@wl(8}nhPnyq+8_?W5QP| z7Z=%UJU@X@qBBK+6t~h)UYc;Ys8;oI^myOf0DYF}bnwa}34?=h{Dihq=?p2>ak(P=s9by|qC!im0vAft?3(y(rr`5nR z7T4WgSW!n%Ci*Ij*B2gAtj9_z6~u%7PpbU8LUQYUgBQlD94 zuXm8JxJ<^aC4G9?={`U8PQ}lyV2p24hKGQp<@A*gTde5n^Zt5yS9j&95E5SufA!L* zVzAXtt`W$7mJ}vHgfdt_nQ|p?mEzDs~*9GoZh)s?^)Z&yN!}Hx~SnioD+zd zvQ8JbIWYdmeGZ!E4FRCdoKss%*)O>pvAcZ9BUbP|RSKz@KR5ms_29^cRQi&g>QdUY z>ZapbUM0H|O@QIMH$Tr^nEXx~taRP0`?s~ayI!mDc)6!)>QS^-Ksr@=Hy!Ws)_JRR z2CbZ2h3yz-{mjU<(ekaiuh{s*Y(wJyiC#6$Xe>F@*zgs<&o4s4cgwKiFws7S<|eDX!`cz z=mZt}^C+!+-?6$I8*gs|Qb;@CxM0=~$T{^ML|<$v-D>Wc^VV*BmHUTiiK@~g=pTkJ z%OfyGL)*h|jBE0>0*V_xF(N0q$kwF0biCmyzPCIO1>V||4-}sT(KXxOBHc<44m4Us zOPqGDbJVRU_e^jO?|)lAUZ@%N?5`&ssP0bxxUILjP7!l2`t!}XrC*;yV+u}e!2NFk zVr#QW(I}bw(s~6%Ri$tJvDd_^=|t*ldsHevD^1SQt@O~Zv4q+!ubO|&uEpv}QeWiE zAXDMQqrOGCKOXdD=lVxsi-XGvFqG#3maqW;u`yG(l

?|(mQs7hl{%lg;van)J} z0~V)r1MW`5Bp+4q7@dZPKU;%d935pt7x1h`C|{@N%5{{9 zeLwQ>A9j4jL0yw#Y0Mzy@BCgeTflV9<0IP>OeCh__v-b1L*8WCH(qUP6KGXFkqL@C z8@&{%5tD_;*J$_&-f+amoAlLZvDyVH8;#^NiP1eBckS8W)x-RBW~;VvS=Puhp}nf- z@79Ik?vHsDB9`A)Xma08RM^#i=Igt@=rch* z$GD4$N}q{+=+#d|;j(VIGA9wGL80NZFIL?7pBml^nlF)kWO%O_LkANW615oq!~Hx? zVn8WkJskbko1vDE=E8ZvJ|!!v`*heddfnrPr-x{tgnEVtVv3eq>^Ubc`ff%N6Hft$ zkB^T|CA4I647Iwyzq=pYY!@s2IkHY0z<*!nBzLW|s_K zrh5p_H8@Z%`Hp^%OZk2`{UZVg-rd)&EiBNz+~TACC7&WXtoYrS2Ad%$Dg@qkfsZDy zQF>u@Juxw|4|6NvA z)`^Fe70ZtjgS^!IU2|($R@Uyrc}>46&ktOO@gHUGX||NN4?d#b+N|;-hK$~9bRMsJ zIg5|KviX6w=Bd%Q3_ZgrV&rA@_;(?MCe?KCV5=LR(2;0^tR*>FUI%5!Lgh4${3h1i*A*8F67%V9{y(`J>jnP6;*vIc%1im zsu?5#&H#Y6l-HjD$3Y}GMqJEK8JA2W-nM%r4&SMFt#I})TbYy6NpyO!c;qv)u=B47^!Ctzpi@(d>&7Ks& zSNj-!%%tgj8Q;!x6j0;*1$Hr*jNbnZ*mx53zD^(l&F^+M$qTd;zHbufl26OcGSRyR ztHQAipZ)gKPY-2zx6JoYNfy3inBjT>q zN|QHTkF=VDbJU$ikNIHcD>K>utYDkZ57Jh9^C`T+Kt%mia8wYs+7Ziq1h{}1(b66u zY*vY#ENbmyl$Oilq>3aPOUfM$sI@z5=1Lrr#&`#Ys{I;GC_a0FbVaq z62?@)Y$m2i6Ao%y?Hbwq^mhczAZ-+_u2t{e_JN==uhtblqUDzkHwW5U+_~)ixF{)Qc0z1pfQV` zNGU%i0&h))NIwzeg?_e;S)R-@7j8q*^tXDEj!>7f$fa!O_Oj^8iha}@;)z4QeagSeNo-S_dUiodA>$_T{aOT5C)$gD9CMhc4!E8LbaIS_| z@Vl!5{xFaI;#sef0_?&>=+!+#QGv0`^1(N}Ce{L$kCVh>1B?z{elDQjYnhKe`B_+a zZ}~0Vs1&UE$CL=>4lR@|PkvFzts~h!OD!ub3Z=VO{vO4ynf9jj^GOG9kGX6C|5e!h zOH%CxTiaZ<-k0{J6(v+6X6hxXktSie=7a-d&E*5v_a4$@O?^NnMoBLl7hDtB>}+4` zchNZX`(7$?ucyT!*!7S2bCes;G45iqWyMJv%)9GWfsOen+(PWrvBr@fsN+_1vd2WU6K!9j3M@>n|!}ZTM~@f zYKcy2#lxo@#Kmp;6&|YKUWKJ%-+KG&uKBMhbI@tf;s-l;5EmEX4J!QV*KgrX;eVzW zK-^5$+GS?Af>L|5=Awo)*m3@Bfw~v=k3l#A4bPB}z0a?^=)m7` z{d$7f`YR}RY*1>a7(Wo^x(Yks00BwC>xry&H2Ax_FDkk`{%Up1zr2Zj#FI{a7Z-90 zo`Q)<3Zyv7;Ij!IJYJ{=4}gygagn$RW24X{0#D}~22zYzI`H#6@Y(qHN_I^CIi3|^ zmNV-Z6-VFTcDFn*g;*=R4gN@B~>F;JxNw0}lqmwhhj;VjqgX zdWQe^dj8*fJ&9LgmWjDf*IkA`?G;`oYzr?W`}FLIZSs`k!+gL0DB>ME-jnyvn}23B zmwaqFX$I0H5&zWarKrWZ<2n3S7rR#+g{P#wSyZE#_wJF1sJ>eEu0#U1nYsDQT(`;F>-RJy^c?{51l z%(_paeRbF7&oNHNMhPJ@F&E5xI_SkI>fYKgdi;6>_k3%U_v9xLQp~>6E8^I7^0Da+ z{dA|`I+ z2!OO|FZa1h##1Z}Zg*l|!9pYz%xp{RTkK&RW?#L`(rok5m-6#w!vZbY-S^H< z6UCL_U~t#gUU;!kDULYy)Dn=k`Q|D3XM1%1C74($&gn5!x8o_y3kCWYfUe3lBteNW zEx)@2xsGAT65%EZPXXQ1EGTW7Va=+b=n11Sz7(}H$U0k;4d<4O+Z1mz> zss!mkwgzk9yHiGnj%%2?^8wEn5A>dEw7a%T%CxkBb$gywGJ2`@fkVrjj$ zZ;x+2mrCj5!4B^&6YvHlF#5Iq?u;b40#PPq=y19$9{wpqZ7#2iJ-+37&A?B$S? z;ObiGT7A8`7Om^tJO5(1hgf^096i&tJjqJ7^-TS9b5zpJugOyVCf0CS_k}l%Pkj_M zj`V)NX_+}vfACHcTh4BgM zF?R^Bi28DRNq~6g&oBmLhr#~n{xlfr(Uy(s2B9&k|4tRP|urKg0zdbWSkHlG)RktwhfjHw-Y#}v{ z-5>T3sG0j>ErjrLRGjCtOXbU7t0f+oc!Dwy!mei-lBDn%k((~=#t55eS`$dm3 ziIG;Q#g;8A%cDJZSa{iMoFrw7O)1sdlLI_AYRagN$NrBD$%6OmU55MLjslJ-HnN#u z$DLJ8@!n9rsajL(qHDj#IE!aTYbt9G3s;wiYM5LEaesG?tFE3bbUGx%aWH|z*Pi@L zEL(Q>aM)|8MnH}|k`TSxQXe5qgPeK&sUPeM5u>}4CSdao#*6H8D8(_W6D1c)5Wuuj zpX?zSm9$cqJ1X+;*9c{IWxlx@Dj!2PfQN$9Fo`uZ5isuNNvzOnQ!T+|oL{8umMuwc zS%CBbj87al0Sn^GC-*fzr-=u9%LBOIzI~$^*K(A!nO_|PZ32S> z%ID?Ekvie9&dQ2F$3O*E#=LesvPct~3m*@}HypTk)-&SF5Cj4T5lGRftS|pkhfm3rydX--y~2v)@m!QSa-f3AW-9gdGc&TY zg}O~HbzF$8aLJv(s8acgh1kbur?15Q#8QLK%UN@jOc|7jZ?tSC-NE?L*<`htScCEI zY3SsI&(+l$0>xf7^2*f|p_&q1wXYo!0Sxa~Oz?@DGfMsFdZdHvu|eE(9E^E`0BX|w zWN&48s3>~1e||%F#P}^BV9a3oH_|(2kMC+zjTA^pkdlzE1am6s>s%%>tQPeTR+wwt zB(>mtve2NCK9;mPu=Q8tfi$GzSOGGP@AF+8 zc&De?rsUQ(^3JqYh6fHvM+C3JMD8;V=PHJO2O! zXG(C2s`64-1DHTu?|_4uAECY3%W!eJ#N@my=#r`U_k>9Cqnp~*MG!G`c2Hid{jK@0 z)19`rp|o|^U14zbb1SNBu-V|?7C+}L4Jt|fLMbdk&6~y3dQeV)$y6j7hd-Q5%t`is z9yj)2^6$fhGKC~rP8||M&_973P7b(aV3zjrsXywp2hi~m|d#EQ%ySj2w5>JIr=r%>q84kKF)Pr z%=kr^cCh+q$t*2|V%>?G+TiUJdQHw83LfEf(^{{&v1&1BvwQ??>{|ndW~a zH-9lS{MpC+VR7ZnkJ`}8D4WpZC!95Z#ro~b%gl{^ymG!OBBQ|8Jh|brnWwZs$|ug> zm46BlqSGGSqrM%lO@bpQ74%Lz*Npd3aqC7aUP6i1X--&9j4)PbTy3+zNGoAg=C-&- z*27Jftzv#b`F=OY&V$=3Oiyt>_$Aj4nQs|Ebea*05H^YsrI@(RP_-$J4IvtNp+oIQE(7 zAuk#ddQ_>KYgR)I=LL-1K;*T)NwLJpAog;cs{**h20HyET9BWM9 z`xk49V^o5Re+DaQb|V|DW<(A&qohJtSQ#{Hu7dGeBrJ&`*Fuvz)W|2J-+yw-@YH!z zDPa!}3WHA4a)N?xN8DzAXj1mCzlXnh}b+kb$E1;aabqB>J)WfW={%6A)InH6pADw4nWZagwzT@(6 za0G)gU`zQS*|ZUEY9zT1j*f}mfGEotol*Z@mHD1t?1GG2-u*Ev%ymPDlqWR z);IwY@F6=fNw`FFuN{cqF)n-{M5dtS^+*30Q%KK&_6H~TEAKg!6~ z&g$~HV-CfRI;|Woyp%hWIyUEFbe!6X=ku<*3J*HIq-#e(jH);9SRphtvmx_b{=8-I zpNf(D!UU*l*d1M|2vGYZ2HQf9>drt{-C1VO%?SGM2V!7iuiQV&ni`qwwL5uk+APyh z&>#Q0HKk%#wB(VS+kh&iaTdYpTwO)Ey95_WYh^0wbRoY&|7b1R%7KcJt}Rw_vHwiT z(&9S9V=akx=I=Cm-!BjIeHvmnR*}W0EBH$}JTucR{FO!T+xb>9TaO}e%oue=Y?L{u zAhZ*#H*~sTp6~pdf~F|w3{UxU$~`{5(^Pg`!~TrUh`KNB=$5mqc>l^T&Urbs2H(SL zEyqx-`0)Jq1A*Lyl}08#vjhGCW+#GkA)h%@L?c2?!`@F;&tB+EHM@y=9XT?3x4=w& z@my+{1DUFFUW#OavGkUVvUF76b>-hueOAgBN=7=CJ|>}%d@bMrq-NNOr3iI1WZx5< zJD&oBvl<&4r5zplhCbT{L2vj;?a7ASUms2-iaN`xsgY`HYpb+$?7pHN&aMeB+CWjp5b9alh*BAzUp?eP@vLpwNN*`85hp!(mLI&Ie@9Q4gWpPUx#N39nBz@L zS!XEBwtjG3duigXS+1gxYU(Y@1a*+FxSj70XI0<46Rsd$=q0VsCt>qtZuW*FF<|uY zS=@uM0VlNH$kt=RBvH*Gy5qNGIhKR@MHN9)aRYo3OmciFSJFscwZ6m3+?mhMP`ts# zRnJ{IlJ?nb*AC_rN(9pj5IZXNRr@1}y zWc(~e7jQ~KR;%z{N}Lu)%RpPwVrY?gisMtP5Us)vvq z??B!Qaflabw$fQ$7!Px{p!V3!S*RA;+tPsd`0ln9Rn5V&*lc=h6X}iesc|4A+JnrV z8z&AiBBLu8{7bx{A6W7+L-*-40x;o%C zWfkxAohcK{Q0ubCJ=4D*j=k{m{r5*0k#SX1YJ5nM>2A><{C3LHj`vvh!@}yfbzZ%P z?G7q;D$#qB2Wr%^Ibh84YL^42pXC6b-00_F+<}Xd0r>K)uNVJ3?eB5(kr$VsS3<7r z4Wi3tc6i?y`83QHF=RgPE2fhFjS69zm-sn`xq~l@-5WJ42hISkuszH6%J_=M%W`lp zPNW7f)#>RQ>kTEt{H1er1|?lNCCO0fx8_9+Ue$W~i?!u=Rh2WeCONqEws$Z+PPLcMjM{by(+6&EIwmE-#@(|Np}{$ z2~yX0Xz#O(6mNyHt0q6)<>Ep<$*0Cj=FUy7F74%nWVqL&1o(E#c&q;;J{gU=t1zol zkGo^I(xs3%0sYB#w8HtsnRsvfdSlc2E|t`vH_D}xYkD*Oy}5c0oh}>Am6bFYnq@-5 z8bS1+Jq0U`8qhkY$4D9Iz@AoKk!h!K`}s4uGB#~JPsPRVSiK|^Y|VYvdPE{6vNx?9 z7zLMF^!Tlrysk2gD$214@c#KkAVG$s?I@;Fdv!0MY4;eXd${ddQE|`(RdR|8GXb#| zzV$Uo)aZWm^T7<$oGr6@LZ9}Np!2;8^+*MWarF zaohWfrw{aW^MB@=pEE zD*9dRyJdSD&FKvOV$E+|6b_*mUkb}cEHug+1_UsiO3xedSJ}Rbt*p$PIhT5``0cwQE+(htyY=s^ z(VaH1c&8n4^x@%h#;O8;dO{-t}W>fZ%(e=gUq;S?as#Ev(hOPve=R7Gf zTDUrEJm5Mu-xr79xSOp#ExzK#Bx(1z?$`Va{VS4d4OeOgQ91z}4T#-h4QhkJ5Q@;v z7qh3(*HSgv5~e>^qd)YMot#<>AeNz~WQ_Is>$P!&kmcuD3k_uhMz9^OS1U+f4kqzd zzB4hksy7+Q_1k)R?&n?vDm{-hTLneaWA|2{Av56BK~CmFEu!PO+cz0YZSH<6R)nhI z7??4)x^WJ321b-ovYr&QbDHtZq19!4lCEOPA)2b-QbGRt_0jV7kKhH_>anmAE*CqSTpg3Qy%0V_m9>-gbG#v+C@X`DC_4e1 z`RM3g0z33%jef|$VD`E3=DH~{C#a;qLDqhZbC{=*PjbEqovca66*>63C@b>tR@uX^ zrxV0oyZ7dt5Hkr!OLy%FJ6WF(djm>H{vr5 zDeo$Sg#&A#i@_NT5Y88Y3752`?m85iQLfy{N3AasjSD|&t;e~$8JG_Ky++$WGkNmY zhewwEd2}$-DqTfTDP~XyI$#fDi?sIZNByh# zoSKo`S9@$pIsbgNN*y-0^SXPq7XOJZRoa~U)!LKUDhc}Le`*MVJHW-61hPoI^MG+c z6YlC0=(jC{`C!m^{C>k^X!fz0w6jbkL^^iSLZ_i%E_52|;_%Ayy!_{0dV47Ieum!3 zzIf#@PM1l;Vso-kW~pV5-o0Ycl=bKEwJRtwfi6}F2Z6)CIXOe8U$5Ue-HX_*Ey=G$b%_Gl3!$~>Ry`^O%D$VNg zaY2Dc<9|FW(St@5J5#6^>j&h`&(-@hUjj*C7(@a2%bM1QsEZ+nzG)v@O7i2T>}2yP zRXh7@rG`ZIQ}Q)T4RvoQNi}2sEMvMO7qM}#x$s+Lmz@e^8*HLO^%D3Q-msF3n?Qd$}5Tc9^Bl=(?9mA}{E%x#z8QeE~n`)B-{getDB9nSeTuvC38&fXW$7pX8z z!Cshgbz`ICvxjHjz}DJY-9W*_Y}C3y&F6&TDuIKttd@a>{dI>j6N`;7_o8WQw_HZE z_F)FI?eT9tGxocYhaT}Btvwv3r9j8p1=>$S?Ie6T-dG33y;*3aIBng$;#3^wS~0y- zp1hIYmw9Z+F#9{CaAkI9)~gM>?x?XJGdAM4zU91qos+qz+-TNz`<6zuM;kOg{@k4J zSy;K-O^%ePV{KQVH+vGeOW%?tz_uW<@H>Fn!zVj?q0Dg|%_#bU!acEpJE!``K+%V7 z`VzYnNgGeE8y>+y^>GIS6JLD8LZ;l+nO7qCjt`c{Yhk?ztB)svZn1tknog%Xv6U-EWmkX=jj=Bds35Lc?jOoC)uhZcu-0~#| zxViLmQc0uZX3g<3Hq2mcgVXd|w#%67z>7)-L`>Q!WuVmB+itp=jF{#2TUK1cBG=q? z$CgLR8FF&wX(c@}_N$-nm9*?iZr?X@Oy8O7~8g`aL(aE5K&S0=aV zuXN4cULFp*EN1e$*8A*;<07}P{i;uiL{6lE_i`#$(rAAfXmT`?l9DgR<{C12W2Vf( z<7YJMIbiI9*Ia%@WBNsMs-*N+?FrM?PUc}^)8OxYIG!IwuuHJAo>GnGa$xF0Y_}7O zi$OpM>vNU!1|G|8KR>@!aMy{2{RR9s1_~-FKYojL;jlS7fYMy!;(Au2C?_|g<>e(= zP_clTb>EIiObh{L$IxEVG5VgWwDcc>c4JH8X|>=N%$9@>`>A%e?bV+_fE|5BybEBu zM8&`nFvD5N3t9qPtpP;;1jWsrL5t)Zl5``M-;bdw1C_E5g9;K9mmUG0_s{d^?~$b# zt0jt3pFMlFeJ3z5TfDiYO%P74qo-5)L=vma z?zsz7AQs@CJ0x}zL{|W}DH=vmevSau{kAowxwW-^eWoQ*aq}vFw(Oldwx#ujAmVZ8^8{{Sccuygu9ZIv6;p$YGT}P7v z%*e6=I1R%a1e-z^-1P40g8Luq_LwLsqZIk<%gkCX!ChtfI?|n~+NXA*5|O_9us@)R zBoa2K@l>3}vutzfx#Q}Dj6fi=yZz@&hPLwT0SPZ*WnM#J4i>YOshGr|`FhL9(sLJq z0aw)?YZ@xqq5OHaE8QU@z4cH1Rb<isj&b-WUe>o0>JxtY*%Oz z5FB!Q(-lHwVR~v8&~EaeKhmN-!9h}1innGrts=zE!QmE!`hnZOho3+tO&I8w8JxVO`En!OEuzK=+~LSWChD@B*KSbo$dS9J3tlF?Ne7Q{ zMfO-8X#EQ#J4=I89xs!|JT}x@6GU{j(j3RS1IRCnUW5a%%66HZ$0f^S!uMS7Ne|FK zzBDj(0VU{b`@TgeRO$Ay|`!bnjI;MCP;UV9Otu1n>+wd^La$^q zQudKJ?-`>m?vz~l^A=~fFFvR6QaJCG(aM3nKdY65Ib;%E^1};sk06Vx+4xzP{0oG9 z8`j;y-y!Y~CBcWgtCO_A2@~A``gk^zEo>8?Y2`fNN_`nVU%p5?FC#c2xcmtz0SdK) z7JPGC0_M+PrhkHSSwG#GO54ob9Dh@>l;l-#z3pI;eA|Zzr>%{RZ-;vudQUQPp+|NK zbmH%jR`t#GBCQE7%SLC;oy(0CsHpzY{a5YIm^;b?H?TE+ax7GHxUM?#eApI0S$(uU zjI>CKGUsO@lgb)ovvVis4At?+a7Lm&TyX@?mhQ6h1r{=&Ru+_MUGS$*x1CcXrM8_pXucCuqq470i=NXp z&-fViV+VHc(zbGMm_v*|6~}LJUh?G7X;$kN~G zHv!hvXLFrS*AZZ;8sU+VFM+&jc@p5Er4bQar2?Sz>-f0o?{dxq-FcO5{<-nm+BoX#s&eQs`08+$ zHGnkVOKKNurt@4Rx%G6+_(~C%(m<>{4;(9UVN4RAq%>&e+t8m&x!Pi^Ka&a$`dW{7 z0t5949Ed~9zBiCw0a;;#L6%wW8~4!C&^g&?42^H)S6iy4{a9L3o9}W&XW9W+Pphe> z-MevE0HhSHXm8~UoAnnogu@mTO>_n=RL#|!I?96;RGm^5%_rASR$&7VhMgwIj6n8y zJX(Sq+2f*^i+a_gGo+4pr@V)RADkxL^E3f$Sc$rS`_d)NH#Z8(PUqz0U^x8S*jsC|GNM6)3o||5(yd1L%xjan=P|G-s_Go1qUa zfQ&8`*Vav^Isa84(TLuduKZ0%$ag?Xl}OrI`7pUJ-4ak$i>@rO8Pv8_!Y^{p^A+)= zFRWKoD43WP5|rNwT{8T+J|2CEf-+*KdbN%Ol`>T3kiLJXUw->KR;;@}TZ;gxzinhE zEm3He7c@}aZ?q}j-F;C)=DXY?G-F3n9+bLN!cqL*NXQZ^eNVAypdT#%dSzVh_T@?% zs1^}3Z;`LZ!N*{M)Lm{GT|#fl_DjY<3nbL41Zw#^1546SD4C<0j~FB}bL3hTcf&_T0nWK85h0Fqlf`#rN%_Pp2&mWCx&-YE{ab z`7+GW-P~a8Y6}4thwG!d8!O7|;(^!Ap@Kht$gHhBQrK`bo4A7BQs2P)67CU<$(uJf zGo``=D_xggeX(!_*N5_iz8o(j#g?17t`M>ORuMdop&RecU6At=9u!50)0x$=RZ(Ku z=gyvGWoMTg)iYckQvTh*CW$`n)zru{x_s#h z+s+hb{-TK^pd>AX3U~h%ZTsAry)c#QO}Mm*mzQy2PHP-ytg0NdWM;>i*y!4P^38Gf zDxGJPn1pir4TidwMV@r*O4UYaKmV0CfzMc7mT^-3aA+Q6#cCJnhm1_fZBz6){3e_5JILJbKQfzvwQr|6b@YDlBcif;IC3h;WLV>D6wW zO4$KHOQ7$>7M;(c<2zaBoJ(qJG+ z^+4$Y3tJdpkis&S->REkpn{c#^zyfO!>8qE>FDTC?*NcTUURf<=&a8II^Z(q{4>@N zpW7?IYtxdNT6b%4B%({iaa=%%pwWfJx7`ZNJGSDQJIY$)a1{Fx0BAQL&mX6E-k9rb z;-6$>dhVM_N*jO+%UOH@ps+qH>*#5Q@~6tCS7uS(t+Uh9DM+O7fXAQLk$Yf1A#TPzBCn$U~Uy}^a+`H2`z@g6)au^qlda*0cm zt>-!GS0fXXq@e*nmo>I4D5<;2k|DVT6)MIpXpwNEQnS}@NZF%Dg^g+WNbZ|oRJOY! z>m7!ZRpntBY(*cFjO>-?js8Aavql#M`1n%$p47=!esan@O$!h9~P zzN4oh7C1_a!6#4Q34L%Zb4M<-N{OYMUTG| za~XbCJ2Kplu}Jp3mpi6g&_+VT!zIyBR6Te>Lp=32wie;@*%I^4a?kR5?y`l6r$yJ4 zo6`*X9&)t^lJecr92khpYw&riq$8@%DmCELWt;QIng9p9_7sC#`>i$8D{Hl6$d~?q zj-vnb9|{pk_ur2R;+6ULQ-zq${{8$Nrmy2QB=^oZ&9=**93QQXx{Qm2QNh1Q2r)`| zF+xtFxH49~21&Ol{Ix3+{6_o95ERi{_vd|us6e;u_KVerYDwaBFwN(Ss{lWYhxh^H z`Uo7(A27Z{u93)f+XF~9B{n7ZSa&Ex<~|uMlw^@JuM^m;mZv>2!7z5F@tN^!+j0`GxsK!(5ZjyM8!f&LL zlKQSQtkLZh?4T~-1nUj$s$an`#SD1w>&Hhc-nn5!#XbwrfCnl@Sl@8b^r8~Mz2~o^ z#g}2!gUIJ*C#~@6jVK&3@JT-e53t^GcPxZ$ows{EQUEFs8pMoO2fp zGK6q42+TPZLVEl;DgeSQu;{?XOm#~sOU)qZ;+Dwt z$L=iUadwHc)*2ePV1AqouS85$6jN)Ts&d%14qQXa)U-RUL7Dt|ae4E$QFnY(xsnv8 z=^du4o^gy%>=bk*A-hZ4D0()pUH|=ij+O|2H$@pei`7lwOov8BUfex6NQamS?|}U! z6G!~$k|y{44}T!_Nd${|z_VUrhSrq583O=?6eOHp{DkRKIcou80Mfp%r^C@Rw6U>4 z35rKml{k{4Y#!jkS_0ENmOeHx>C9fFr+*7!-xQMijMwIVjWD-{j9p1ZMb_T_CQ82` zI4_S2nL>s1CiXo+9+yct@{+?a_X3o4gI(P(t`7?-#>Bdk&#LhKgx zvAm?24}cg*DUWo>IhV_(&O_;h`PY@=Qg#X)Q!a!Uc)Pd3!MZZBq`E2{db+w9`@DOS z{RPH`734W*tP!sM!Yq3D&q0MW$iDCajKcOQ`(8%=DeL8k{{)DH_zsRRoY|Z8@hKyV zAj+;+uaHNP#_D1QQ3c%b1i*~krlR_Zs4Rum+w{mFE$T~1gctH09y%4j9AJ~>O5^h& zGfIbBkEe5p&_vz5sVSh}JyC`1gHl|3OlCxBT=E_ZIawcEH_ZEPDi?FGy_PjsZcCD2 z@bdD)mt!SLrk^;f(Q^*!wDe6Fxy@%bwzYk)a^2|ta}+@lK&YNWhvr)cg-LFj40tVf zbagL2t^6%G8&P{zLLZ3e3)ToS9ic~L8x+4^@Xye{V16p?d=p%)ghQNGo0*hk72>nSfc1M=h*Q479TMsZ|TXap`h z+f|Dl)Z!B`G{2(U$fmmb@{A@#7%X*lb@q^w=04%u8sb|FJwbFk6|{kI4cgFx>vVF zsOpuU{GxY=v{#iJ2X1EXfo$}SGHyDKOaaym%`q>J<}5!Wz+1hE-Iy~~=4JQyqoN97 z6e;?Y9KkXRs9}Bn$V)s%*`ZC^qHtRYkDIq2%AQ8ef6qvEjPO*Rf>9M}`N)8twpQ%# z3)A-@S{-UMcDuUw&ck+|#O!r+xQQAe!R7Pk0AFrUk)cT_h)4x!3yYX->!QAwRlKt^ zj=x&zobQ^I;#ktwLFc1PH0`-Whpar`P06 zH_MT+G)JnAf%z#tNFuu+Ba;#^>)uon=49;wtZss$s=0MjqywGtg(ADC;N7N=MebJ{ z9R^!uqSur^a1@mNBzDgF3VAv{MO8MhHeNvkS&g$qI7C-5cWseW^Hw~~5X;Rm%u+^3 zWQ?}gAUw!m#z`wJ@46J6>VvcDD8@&{Tplt$(!W6%N<>6mZ-atPdhayboBKRt1@L>_Rf2*H2C=|46lu=H|VUy4l7}c7B2Q$?C zZF_#cMWs59!3z@zwt?x0JT^@N=1KJFDnEgZQhD_m4?MsIe14q1S_Yu=-L@b29l@c7 zy_xs;`&*28^vnC*6G{{uArZl4<&B%`Ffk>&O~Dv?(mgUy;q`JaK_>rRfqCX2;@7;< z%6%o|&AD3up^jmb;5lza*=z_&<pC6{k_^SwspH%v&)TI}-o<(v?O2rbcUM&{aipcokjE&JDQWXtXcO%$(dUBVYcg z4Rl>SJ5EzD1U|Z7glO< zMRQbBUS2We>C))w-%IM#tjwW4&wQ5{dp)NGSp+(gPn^a4HQ z01tT54T*ul_>7hs{UV{LO&y*BAC>JSR0~|_NV@GbD)u|icu){y%(41sGvUe=th7pd zwadyF#@+$?1~P|LDYW+LAmV8cYNf|cD94fW^TjK@f7D_g<0wQ;m+ur2)?^#P!H%^W z;v+LuSZD|K&jFnpjH9VuxzdKceQ(_;W+PT+OM?Jk+!UC3=pyV+EQTDadT3_#K|>X{ zD@1ASoD$)iB7Et}`)q>@&GK&i^I`1lSh2OHwnyxdqGBo78DG92le!N@=-5P3aoGI& z+Sz7>{F^VI(ldZlK^`zhsyht-d+o1zc4sJKUaAOwjbu@YZf>F~M*>IzRMCzP*9LCz zE}xmv>8nJ^>~fg#nrh-$qn=K)uxz*;bPl**3JM$O~L7NG&gIz;3Fi+kpr$urjB9WaJtcR@w#`;rV{ zxg?KE8;-0ob0PW0pL+qV=947vMKG#cl0 zyIm>-@55WQE|Ou2V_0c)tS#Q*gfIB_ULr&kh2B)(euIJ2Ic|tMU9Wqh@%U`FbRfjf zmyoR1{a(|Pg@M9C6?%GoS`L{oamBbxR+2AY8bXaG3lltj>Dzj_nUkaIjfUpX?xxX) z=x=SsE2wMn*3F|PMr|RtWq=-|@A4{w!;jBhHBN|VIAhSk)0UMrvULpGphC-`9FllJhXJh)O0_C z#^l9U*Nm5EwD>+;EuMYlA;lMd&5+Z`btVe49qmC*^<0^pb6C2p@nPnY45}g18|SMq zXot74z;O)7EXa zlJoajcNfVJT;oacO?g-5E7`@dUnS^zagwV0-18Roa9pQ7C1k4}MACy=y>0Vbe3n;_ z5?b%OY-!1cO09k&SPDJtYZl^*A z0Tqjb$|3stcy literal 0 HcmV?d00001