[New features]Add function node in phi_kernel for MKLDNN #51073

heavyrain-lzy · 2023-03-01T09:22:17Z

PR types

New features

PR changes

OPs

Describe

card-67001
执行静态图时，为inputs准备数据阶段会根据expected_kernel和每个tensor的定义进行backend、layout、dtype转换。在phi下的kernel借助了GetKernelTypeForVar和kernel内注册的argsdef进行转换，大部分的GetKernelTypeForVar函数与动态图统一，但由于MKLDNN的引入，存在部分op的GetKernelTypeForVar多了一些专门处理MKLDNN kernel的特殊代码，如interpolate_op.cc中：

  phi::KernelKey GetKernelTypeForVar(
      const std::string& var_name,
      const phi::DenseTensor& tensor,
      const phi::KernelKey& expected_kernel_type) const override {
#ifdef PADDLE_WITH_MKLDNN
    if ((expected_kernel_type.layout() == phi::DataLayout::ONEDNN) &&
        (tensor.layout() != phi::DataLayout::ONEDNN)) {
      auto attrs = Attrs();
      auto ar = paddle::framework::AttrReader(attrs);
      const std::string data_format = ar.Get<std::string>("data_layout");
      auto dl = phi::StringToDataLayout(data_format);
      // Some models may have intentionally set "AnyLayout" for pool
      // op. Treat this as NCHW (default data_format value)
      if (dl != phi::DataLayout::kAnyLayout) {
        return phi::KernelKey(tensor.place(), dl, expected_kernel_type.dtype());
      }
    }
#endif

    if (var_name == "OutSize" || var_name == "SizeTensor" ||
        var_name == "Scale") {
      return phi::KernelKey(phi::Backend::ALL_BACKEND,
                            expected_kernel_type.layout(),
                            expected_kernel_type.dtype());
    }
    return phi::KernelKey(
        tensor.place(), tensor.layout(), expected_kernel_type.dtype());
  }

这段逻辑较为特殊并且不符合规范，如果需要自动生成，会导致自动生成脚本变得臃肿。综合考虑后，决定在kernel中增加一个功能和GetKernelTypeForVar类似的函数指针，默认缺省，只在GetKernelTypeForVar有特殊逻辑的MKLDNN kernel中注册，如在OneDnn kernel文件interploate_kernel.cc中增加对应的函数和注册：

phi::KernelKey InterpolateGetKernelTypeForVar(
    const InferVarKernelContext* ctx) {
  const std::string& var_name = ctx->GetVarName();
  const DenseTensor& tensor = ctx->GetTensor();
  const KernelKey& expected_kernel_type = ctx->GetKernelKey();
  const AttributeMap& attrs = ctx->GetAttrs();
  // Only input require reshaping, weights and
  // bias are having shape in NCHW order
  if ((expected_kernel_type.layout() == phi::DataLayout::ONEDNN) &&
      (tensor.layout() != phi::DataLayout::ONEDNN)) {
    auto it = attrs.find("data_layout");
    PADDLE_ENFORCE_NE(it,
                      attrs.end(),
                      paddle::platform::errors::NotFound(
                          "Cannot find attribute %s.", "data_layout"));
    const std::string data_layout = PADDLE_GET_CONST(std::string, it->second);
    auto dl = phi::StringToDataLayout(data_layout);
    // Some models may have intentionally set "AnyLayout" for pool
    // op. Treat this as NCHW (default data_format value)
    if (dl != phi::DataLayout::kAnyLayout) {
      return phi::KernelKey(tensor.place(), dl, expected_kernel_type.dtype());
    }
  }
  if (var_name == "OutSize" || var_name == "SizeTensor" ||
      var_name == "Scale") {
    return phi::KernelKey(phi::Backend::ALL_BACKEND,
                          expected_kernel_type.layout(),
                          expected_kernel_type.dtype());
  }
  return phi::KernelKey(
      tensor.place(), tensor.layout(), expected_kernel_type.dtype());
}
...
...
...
PD_REGISTER_KERNEL(bilinear_interp,
                   OneDNN,
                   ONEDNN,
                   phi::BilinearInterpKernel,
                   float,
                   phi::dtype::bfloat16) {}
                   phi::dtype::bfloat16) {
  kernel->get_kerneltype_forvar_fn_ = phi::InterpolateGetKernelTypeForVar;
}

PD_REGISTER_KERNEL(nearest_interp,
                   OneDNN,
                   ONEDNN,
                   phi::NearestInterpKernel,
                   float,
                   phi::dtype::bfloat16,
                   int8_t,
                   uint8_t) {}
                   uint8_t) {
  kernel->get_kerneltype_forvar_fn_ = phi::InterpolateGetKernelTypeForVar;
}

此接口也能用于未来其他硬件的接入。

When executing a static graph, the stage of preparing data for inputs will perform backend, layout, and dtype conversions according to the definition of expected_kernel and each tensor. The kernel under phi is converted with the help of GetKernelTypeForVar and the argsdef registered in the kernel. Most of the GetKernelTypeForVar functions are unified with the dynamic graph. However, due to the introduction of MKLDNN, there are some operators' GetKernelTypeForVar with special codes, such as in interpolate_op.cc:

  phi::KernelKey GetKernelTypeForVar(
      const std::string& var_name,
      const phi::DenseTensor& tensor,
      const phi::KernelKey& expected_kernel_type) const override {
#ifdef PADDLE_WITH_MKLDNN
    if ((expected_kernel_type.layout() == phi::DataLayout::ONEDNN) &&
        (tensor.layout() != phi::DataLayout::ONEDNN)) {
      auto attrs = Attrs();
      auto ar = paddle::framework::AttrReader(attrs);
      const std::string data_format = ar.Get<std::string>("data_layout");
      auto dl = phi::StringToDataLayout(data_format);
      // Some models may have intentionally set "AnyLayout" for pool
      // op. Treat this as NCHW (default data_format value)
      if (dl != phi::DataLayout::kAnyLayout) {
        return phi::KernelKey(tensor.place(), dl, expected_kernel_type.dtype());
      }
    }
#endif

    if (var_name == "OutSize" || var_name == "SizeTensor" ||
        var_name == "Scale") {
      return phi::KernelKey(phi::Backend::ALL_BACKEND,
                            expected_kernel_type.layout(),
                            expected_kernel_type.dtype());
    }
    return phi::KernelKey(
        tensor.place(), tensor.layout(), expected_kernel_type.dtype());
  }

This piece of logic is special and does not conform to the specification. If it needs to be automatically generated, it will cause the automatic generation script to become bloated. After comprehensive consideration, it is decided to add a function pointer similar to GetKernelTypeForVar in the kernel. By default, it is only registered in the MKLDNN kernel with special logic for GetKernelTypeForVar. For an example, add the function and register the function node in the OneDnn kernel file interploate_kernel.cc:

phi::KernelKey InterpolateGetKernelTypeForVar(
    const InferVarKernelContext* ctx) {
  const std::string& var_name = ctx->GetVarName();
  const DenseTensor& tensor = ctx->GetTensor();
  const KernelKey& expected_kernel_type = ctx->GetKernelKey();
  const AttributeMap& attrs = ctx->GetAttrs();
  // Only input require reshaping, weights and
  // bias are having shape in NCHW order
  if ((expected_kernel_type.layout() == phi::DataLayout::ONEDNN) &&
      (tensor.layout() != phi::DataLayout::ONEDNN)) {
    auto it = attrs.find("data_layout");
    PADDLE_ENFORCE_NE(it,
                      attrs.end(),
                      paddle::platform::errors::NotFound(
                          "Cannot find attribute %s.", "data_layout"));
    const std::string data_layout = PADDLE_GET_CONST(std::string, it->second);
    auto dl = phi::StringToDataLayout(data_layout);
    // Some models may have intentionally set "AnyLayout" for pool
    // op. Treat this as NCHW (default data_format value)
    if (dl != phi::DataLayout::kAnyLayout) {
      return phi::KernelKey(tensor.place(), dl, expected_kernel_type.dtype());
    }
  }
  if (var_name == "OutSize" || var_name == "SizeTensor" ||
      var_name == "Scale") {
    return phi::KernelKey(phi::Backend::ALL_BACKEND,
                          expected_kernel_type.layout(),
                          expected_kernel_type.dtype());
  }
  return phi::KernelKey(
      tensor.place(), tensor.layout(), expected_kernel_type.dtype());
}
...
...
...
PD_REGISTER_KERNEL(bilinear_interp,
                   OneDNN,
                   ONEDNN,
                   phi::BilinearInterpKernel,
                   float,
                   phi::dtype::bfloat16) {}
                   phi::dtype::bfloat16) {
  kernel->get_kerneltype_forvar_fn_ = phi::InterpolateGetKernelTypeForVar;
}

PD_REGISTER_KERNEL(nearest_interp,
                   OneDNN,
                   ONEDNN,
                   phi::NearestInterpKernel,
                   float,
                   phi::dtype::bfloat16,
                   int8_t,
                   uint8_t) {}
                   uint8_t) {
  kernel->get_kerneltype_forvar_fn_ = phi::InterpolateGetKernelTypeForVar;
}

paddle-bot · 2023-03-01T09:22:22Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

…an't be template variable

paddle/fluid/framework/data_transform.cc

paddle/phi/core/infer_varkernel_utils.h

paddle/phi/kernels/onednn/interpolate_kernel.cc

paddle/phi/core/infer_varkernel_utils.h

paddle/phi/api/yaml/op_compat.yaml

paddle/fluid/framework/data_transform.cc

paddle/fluid/framework/new_executor/interpreter/data_transfer.cc

paddle/phi/core/infer_varkernel_utils.cc

paddle/phi/core/infer_varkernel_utils.h

heavyrain-lzy · 2023-03-07T11:19:38Z

@xinyu-intel, @YangQun1,Please help review the code.

xinyu-intel · 2023-03-09T23:33:58Z

paddle/fluid/framework/data_transform.cc

+    const AttributeMap &fluid_attrs,
+    phi::AttributeMap *phi_attrs,
+    bool has_infer_varkernel_fn) {
+  // According to "GetKernelTypeForVar" in some ops those have MKLDNN codes,


Suggested change

// According to "GetKernelTypeForVar" in some ops those have MKLDNN codes,

// According to "GetKernelTypeForVar" in some ops executed with oneDNN,

Thanks for the suggestion.

Thanks. I will change the comment according to your suggestion.

xinyu-intel · 2023-03-09T23:38:01Z

paddle/phi/core/compat/get_kerneltype_forvar_utils.h

+class KernelKey;
+class DenseTensor;
+/**
+ * Note: GetKernelTypeForVarContext is currently designed to MKLDNN kernel when


Suggested change

* Note: GetKernelTypeForVarContext is currently designed to MKLDNN kernel when

* Note: GetKernelTypeForVarContext is currently designed for oneDNN kernel when

Thanks for the suggestion.

Thanks. I will change the comment according to your suggestion.

xinyu-intel · 2023-03-09T23:38:18Z

paddle/phi/core/compat/get_kerneltype_forvar_utils.h

+/**
+ * Note: GetKernelTypeForVarContext is currently designed to MKLDNN kernel when
+ * the related memeber function 'GetKernelTypeForVar' is special. It is
+ * possiable to uesed for other custom hardwares in the future.


Suggested change

* possiable to uesed for other custom hardwares in the future.

* possible to leverage to other vendor libraries in the future.

Thanks for the suggestion.

Thanks. I will change the comment according to your suggestion.

xinyu-intel · 2023-03-10T00:00:53Z

paddle/phi/core/compat/get_kerneltype_forvar_utils.h

+ private:
+  const KernelKey* kernel_key_;  // not owned
+  // Use AttributeMap in namespace 'phi' to avoid depending 'fuild'
+  const AttributeMap* attrs_;  // not owned


seems that oneDNN specific stuffs are hided inside the AttributeMap. Do you mind making these as members as this is only used for oneDNN kernel.

Because the attribute name that will be used can be "data_layout" or "data_format" or other for the future vendors, it is brief to using AttributeMap.

Personally, as data_layout/data_format has been defined through the phi API, it is natural to stay in the structure directly. I think you're right that AttributeMap will make it easier for the future vendors to store more specific stuffs, but at the same time, the content inside the map can be out of controlled by the framework. Both are okay to me.

xinyu-intel · 2023-03-10T00:15:52Z

paddle/phi/kernels/onednn/interpolate_kernel.cc

+  // Only input require reshaping, weights and
+  // bias are having shape in NCHW order


I'm not sure if I understand these correctly. Do you mean "Only input requires changing data_layout"?
Usually, we distinguish shape and layout. Take Tensor{shape={1,3,8,8}, data_format={"NCHW"},stride={192,64,8,1}} as an example. data_format means the shape is ordered by NCHW(a.k.a channel_size is 3). stride means it has contiguous data_layout(DataLayout::kNCHW). We can reorder the tensor to kNHWC/KONEDNN but keep the shape unchanged. We can reshape the tensor to {3,8,8}/{24,8}/{1,192} with the layout unchanged.

I think that shape data_formatandstride is a whole, together to represent a tensor。Take Tensor{shape={1,3,8,8}, data_format={"NCHW"},stride={192,64,8,1}} as an example. If you change the data_format from NCHW to NHWC, the shape must change from {1,3,8,8} to {1, 8, 8, 3} together. Otherwise, the data is wrong when you use.

YangQun1 · 2023-03-10T00:51:01Z

paddle/phi/kernels/onednn/interpolate_kernel.cc

+  // bias are having shape in NCHW order
+  if ((expected_kernel_type.layout() == DataLayout::ONEDNN) &&
+      (tensor.layout() != DataLayout::ONEDNN)) {
+    auto it = attrs.find("data_layout");


Hi, I'm a little confusing about the definition of data_layout and data_format in Paddle. In LRNOp::GetKernelTypeForVar, we get the data_format from Attrs, but here we get the data_layout instead. Do they have any difference?

BTW, do we also plan to move the GetKernelTypeForVar of LRNOp/Pad2dOp/Pad3dOp to phi? since there are also some MKLDNN specific code.

The Attrs name data_formatand data_layout have the same meaning. However we can't change them for compatibility.
We also plan to move the GetKernelTypeForVar of LRNOp/Pad2dOp/Pad3dOp to phi soon.

YangQun1 · 2023-03-10T01:13:43Z

If it needs to be automatically generated, it will cause the automatic generation script to become bloated.

May I ask how the MKLDNN specific code impact the auto generation? Not sure if I understand correctly, currently I didn't find GetKernelTypeForVar related code in the generated api.cc and backward_api.cc files.

heavyrain-lzy · 2023-03-10T03:26:16Z

You can find the code automatically generated from generated_op1.cc~generated_op4.cc after executing cmake command.

zyfncg · 2023-03-10T04:00:03Z

paddle/phi/core/compat/get_kerneltype_forvar_utils.h

+
+  const std::string& GetVarName(void) const;
+
+  const DenseTensor& GetTensor(void) const;


这里会不会出现tensor为null的情况

如果没有给tensor成员赋值，会出现null情况，为确保安全后续加上安全性检查。

jiahy0825

LGTM

heavyrain-lzy added 2 commits March 1, 2023 08:45

Add function node in phi_kernel for MKLDNN

be4f61d

Merge remote-tracking branch 'upstream/develop' into yaml_dtype

ca9d1b0

heavyrain-lzy added 4 commits March 1, 2023 11:16

fix the bug in 'BuildInferVarKernelContext'

28e7686

add infer_varkernel_utils.cc

d6b6d1c

Merge remote-tracking branch 'upstream/develop' into yaml_dtype

e7149f7

fix the bug:the first two parametes of 'BuildInferVarKernelContext' c…

47e4ad2

…an't be template variable

jiahy0825 reviewed Mar 6, 2023

View reviewed changes

paddle/fluid/framework/data_transform.cc Outdated Show resolved Hide resolved

paddle/fluid/framework/data_transform.cc Show resolved Hide resolved

YuanRisheng reviewed Mar 6, 2023

View reviewed changes

zyfncg reviewed Mar 6, 2023

View reviewed changes

heavyrain-lzy added 6 commits March 6, 2023 11:07

change the code according to first review

77c66d6

change the code according to first review

4a77a39

merge develop

3025b90

change the mode of paddle_build.sh

2f670e1

change 'infer_var_kernel_fn_' to 'get_kerneltype_forvar_fn_'

f66c2d6

add the error information

1b5b3d9

heavyrain-lzy added 3 commits March 8, 2023 08:40

fix NotFound infomation warning

42ea9cd

fix NotFound infomation warning

3426644

fix NotFound infomation warning

b305cd5

yaomichael requested a review from Silv3S March 9, 2023 07:30

xinyu-intel reviewed Mar 10, 2023

View reviewed changes

YangQun1 reviewed Mar 10, 2023

View reviewed changes

xinyu-intel approved these changes Mar 10, 2023

View reviewed changes

YangQun1 approved these changes Mar 10, 2023

View reviewed changes

zyfncg approved these changes Mar 10, 2023

View reviewed changes

jiahy0825 approved these changes Mar 10, 2023

View reviewed changes

YuanRisheng approved these changes Mar 10, 2023

View reviewed changes

zyfncg merged commit a0a6dc6 into PaddlePaddle:develop Mar 10, 2023

heavyrain-lzy mentioned this pull request Apr 24, 2023

[测试用] elementwise_add #53272

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[New features]Add function node in phi_kernel for MKLDNN #51073

[New features]Add function node in phi_kernel for MKLDNN #51073

heavyrain-lzy commented Mar 1, 2023 •

edited

Loading

paddle-bot bot commented Mar 1, 2023

heavyrain-lzy commented Mar 7, 2023 •

edited

Loading

xinyu-intel Mar 9, 2023

heavyrain-lzy Mar 10, 2023

heavyrain-lzy Mar 10, 2023

xinyu-intel Mar 9, 2023

heavyrain-lzy Mar 10, 2023

heavyrain-lzy Mar 10, 2023

xinyu-intel Mar 9, 2023

heavyrain-lzy Mar 10, 2023

heavyrain-lzy Mar 10, 2023

xinyu-intel Mar 10, 2023

heavyrain-lzy Mar 10, 2023

xinyu-intel Mar 10, 2023

xinyu-intel Mar 10, 2023

heavyrain-lzy Mar 10, 2023

YangQun1 Mar 10, 2023

heavyrain-lzy Mar 10, 2023

YangQun1 commented Mar 10, 2023

heavyrain-lzy commented Mar 10, 2023

zyfncg Mar 10, 2023

heavyrain-lzy Mar 10, 2023

jiahy0825 left a comment

	// According to "GetKernelTypeForVar" in some ops those have MKLDNN codes,
	// According to "GetKernelTypeForVar" in some ops executed with oneDNN,

	* Note: GetKernelTypeForVarContext is currently designed to MKLDNN kernel when
	* Note: GetKernelTypeForVarContext is currently designed for oneDNN kernel when

	* possiable to uesed for other custom hardwares in the future.
	* possible to leverage to other vendor libraries in the future.

		// Only input require reshaping, weights and
		// bias are having shape in NCHW order


		const std::string& GetVarName(void) const;

		const DenseTensor& GetTensor(void) const;

[New features]Add function node in phi_kernel for MKLDNN #51073

[New features]Add function node in phi_kernel for MKLDNN #51073

Conversation

heavyrain-lzy commented Mar 1, 2023 • edited Loading

PR types

PR changes

Describe

paddle-bot bot commented Mar 1, 2023

heavyrain-lzy commented Mar 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YangQun1 commented Mar 10, 2023

heavyrain-lzy commented Mar 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiahy0825 left a comment

Choose a reason for hiding this comment

heavyrain-lzy commented Mar 1, 2023 •

edited

Loading

heavyrain-lzy commented Mar 7, 2023 •

edited

Loading