Move fused_attention op to phi [迁移前向 GPU OpKernel] #51743

AndSonder · 2023-03-16T07:36:16Z

PR types

Others

PR changes

Others

Describe

Fluid 算子函数式迁移, 迁移 fused_attention op 的 gpu kernel (前向)

… fused_attention_kernel

From00 · 2023-03-27T03:45:03Z

paddle/phi/kernels/fusion/fused_attention_kernel.h

+
+#include "paddle/phi/core/dense_tensor.h"
+
+namespace phi {


fuse算子统一写在phi::fusion命名空间下

From00 · 2023-03-27T03:59:19Z

paddle/phi/kernels/fusion/gpu/fused_attention_kernel.cu

@@ -0,0 +1,379 @@
+// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.


2022 -> 2023

From00 · 2023-03-27T04:03:02Z

paddle/phi/kernels/fusion/gpu/fused_attention_kernel.cu

+  phi::DataType data_type;
+  if (kernel_key.dtype() == phi::DataType::FLOAT16 ||
+      kernel_key.dtype() == phi::DataType::FLOAT32) {
+    data_type = phi::DataType::FLOAT32;
+  } else {
+    data_type = phi::DataType::FLOAT64;
+  }
+  kernel->OutputAt(0).SetDataType(data_type);
+  kernel->OutputAt(1).SetDataType(data_type);
+  kernel->OutputAt(3).SetDataType(data_type);
+  kernel->OutputAt(4).SetDataType(data_type);
+  kernel->OutputAt(15).SetDataType(data_type);
+  kernel->OutputAt(16).SetDataType(data_type);


Suggested change

phi::DataType data_type;

if (kernel_key.dtype() == phi::DataType::FLOAT16 ||

kernel_key.dtype() == phi::DataType::FLOAT32) {

data_type = phi::DataType::FLOAT32;

} else {

data_type = phi::DataType::FLOAT64;

}

kernel->OutputAt(0).SetDataType(data_type);

kernel->OutputAt(1).SetDataType(data_type);

kernel->OutputAt(3).SetDataType(data_type);

kernel->OutputAt(4).SetDataType(data_type);

kernel->OutputAt(15).SetDataType(data_type);

kernel->OutputAt(16).SetDataType(data_type);

if (kernel_key.dtype() == phi::DataType::FLOAT16) {

kernel->OutputAt(0).SetDataType(phi::DataType::FLOAT32);

kernel->OutputAt(1).SetDataType(phi::DataType::FLOAT32);

kernel->OutputAt(3).SetDataType(phi::DataType::FLOAT32);

kernel->OutputAt(4).SetDataType(phi::DataType::FLOAT32);

kernel->OutputAt(15).SetDataType(phi::DataType::FLOAT32);

kernel->OutputAt(16).SetDataType(phi::DataType::FLOAT32);

}

From00 · 2023-03-27T04:53:16Z

python/paddle/fluid/tests/unittests/test_fused_multi_transformer_op.py

@@ -28,8 +28,12 @@
 from paddle.nn.layer.norm import LayerNorm
 from paddle.nn.layer.transformer import _convert_attention_mask

-random.seed(42)


恢复此单测原先的随机数设置

From00 · 2023-03-27T05:04:25Z

paddle/phi/kernels/fusion/gpu/fused_gemm_epilogue_utils.h

+#include "paddle/fluid/framework/scope_guard.h"
+#include "paddle/fluid/memory/memory.h"


Suggested change

#include "paddle/fluid/framework/scope_guard.h"

#include "paddle/fluid/memory/memory.h"

#include "paddle/phi/core/scope_guard.h"

#include "paddle/phi/commom/memory_utils.h"

From00 · 2023-03-27T05:07:28Z

paddle/phi/kernels/fusion/gpu/fused_residual_dropout_bias.h

+
+#include <cuda.h>
+
+#include "paddle/fluid/operators/fused/quant_dequant_kernel.h"


Suggested change

#include "paddle/fluid/operators/fused/quant_dequant_kernel.h"

#include "paddle/phi/kernels/funcs/layer_norm_impl.cu.h"

phi下不可引用fluid的头文件

已去除该引用

… fused_attention_kernel

From00 · 2023-04-02T03:18:06Z

paddle/fluid/operators/fused/fused_attention_utils.h

+namespace fusion {
+
+template <typename T>
+static void AllReduce(phi::DenseTensor &tensor,  // NOLINT


与fused_attention_op.cu中的AllReduce统一成一份

已删除fused_attention_op.cu中的AllReduce

From00 · 2023-04-02T03:21:03Z

paddle/phi/kernels/fusion/fused_attention_grad_kernel.h

@@ -0,0 +1,91 @@
+// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.


此文件可以删除，后续反向算子也直接在fluid下做迁移即可

From00 · 2023-04-02T03:21:32Z

paddle/phi/kernels/fusion/fused_attention_kernel.h

@@ -0,0 +1,157 @@
+// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.


此文件可删除

From00 · 2023-04-02T03:24:16Z

python/paddle/fluid/tests/unittests/test_fused_multi_transformer_op.py

@@ -28,8 +28,12 @@
 from paddle.nn.layer.norm import LayerNorm
 from paddle.nn.layer.transformer import _convert_attention_mask

-random.seed(42)
-default_main_program().random_seed = 42
+seed = 53


随机种子改回42会有错误吗？

已修改回42，单测通过

From00

LGTM

因分布式相关头文件依赖：

#include "paddle/fluid/distributed/collective/process_group_nccl.h"
#include "paddle/fluid/platform/collective_helper.h"
#include "paddle/fluid/platform/device/gpu/nccl_helper.h"

此算子直接在fluid目录下迁移成函数式，后续分布式依赖迁移后，再将代码移动到PHI目录下。

ZzSean

LGTM for CI-OP-Benchmark

XiaoguangHu01

LGTM

AndSonder added 11 commits March 13, 2023 21:35

add kernel functions

d1ca7d5

update kernel functions

1c82183

update func parameters' name

72b90a4

create codes for gpu device

9f09c23

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

8d35f67

… fused_attention_kernel

调整文件位置

2e51625

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

0dff777

… fused_attention_kernel

fix include error

0e1f451

remove dependent files to phi/

874a5ee

restore fused_attention_op.cu

8499ee9

fix dependence errors

ece744e

paddle-bot bot added the contributor External developers label Mar 16, 2023

AndSonder added 17 commits March 17, 2023 11:29

fix dependence errors

fdedffb

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

3bc1501

… fused_attention_kernel

fix include error

e668f49

fix all depandence errors[build success]

a09202a

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

95f32a8

… fused_attention_kernel

remove useless include

8bb886b

recover useless include

f7014b4

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

a8c31a8

… fused_attention_kernel

use phi::ToNCCLDataType

e2f7bda

fix namespace

8572858

update new register code

33d480c

fix error in fused_gemm_epilogue_utils

a226b16

fix error in FusedAttentionKernel parm

c4ddcab

finish fused_attention registe code[build success]

0b2338b

add paddle::optional

0486460

add sig file

5cf94a5

fix conflicting file

c846453

AndSonder changed the title ~~Move fused_attention op to phi [迁移 GPU OpKernel]~~ Move fused_attention op to phi [迁移前向 GPU OpKernel] Mar 20, 2023

AndSonder added 8 commits March 22, 2023 16:38

trans some fluid api to phi api

3c146a4

add #if

59951e3

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

08eda13

… fused_attention_kernel

update test code

8f58616

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

2d9b5df

… fused_attention_kernel

update test codes

2e74808

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

4dc6d77

… fused_attention_kernel

recover test codes

cac3a64

From00 reviewed Mar 27, 2023

View reviewed changes

From00 mentioned this pull request Mar 31, 2023

[PHI] Functionalization for Fluid kernel #52395

Closed

AndSonder added 2 commits March 31, 2023 15:43

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

ac4285d

… fused_attention_kernel

trans fused_attention to fluid

57116d2

AndSonder force-pushed the fused_attention_kernel branch from 483de44 to 57116d2 Compare March 31, 2023 14:03

AndSonder added 2 commits April 1, 2023 10:47

move #endif to end

5bb81b2

move #endif

d76da5a

AndSonder requested a review from From00 April 2, 2023 03:04

From00 reviewed Apr 2, 2023

View reviewed changes

AndSonder added 2 commits April 3, 2023 12:26

delete useless files

a7a7884

use fused attention utils and recover random seed

7a6f1ec

AndSonder requested a review from From00 April 3, 2023 11:23

AndSonder added 2 commits April 3, 2023 20:16

remove fluid include in phi

944bdae

recover fused_attention_op.cu

319ada7

From00 approved these changes Apr 5, 2023

View reviewed changes

ZzSean approved these changes Apr 6, 2023

View reviewed changes

YuanRisheng approved these changes Apr 6, 2023

View reviewed changes

XiaoguangHu01 approved these changes Apr 6, 2023

View reviewed changes

From00 merged commit a7ec895 into PaddlePaddle:develop Apr 6, 2023

0x45f mentioned this pull request Feb 25, 2024

fused_multi_transformer/fused_bias_dropout_residual_layer_norm to phi #62049

Merged

AndSonder deleted the fused_attention_kernel branch April 23, 2024 13:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move fused_attention op to phi [迁移前向 GPU OpKernel] #51743

Move fused_attention op to phi [迁移前向 GPU OpKernel] #51743

AndSonder commented Mar 16, 2023 •

edited

Loading

From00 Mar 27, 2023

AndSonder Mar 27, 2023

From00 Mar 27, 2023

AndSonder Mar 27, 2023

From00 Mar 27, 2023

AndSonder Mar 27, 2023

From00 Mar 27, 2023

AndSonder Mar 27, 2023

From00 Mar 27, 2023

AndSonder Mar 27, 2023

From00 Mar 27, 2023

AndSonder Mar 27, 2023

From00 Apr 2, 2023

AndSonder Apr 3, 2023

From00 Apr 2, 2023

AndSonder Apr 3, 2023

From00 Apr 2, 2023

AndSonder Apr 3, 2023

From00 Apr 2, 2023

AndSonder Apr 3, 2023

From00 Apr 2, 2023

AndSonder Apr 3, 2023

From00 left a comment

ZzSean left a comment

XiaoguangHu01 left a comment

		@@ -0,0 +1,379 @@
		// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.

		#include "paddle/fluid/framework/scope_guard.h"
		#include "paddle/fluid/memory/memory.h"


		#include <cuda.h>

		#include "paddle/fluid/operators/fused/quant_dequant_kernel.h"

	#include "paddle/fluid/operators/fused/quant_dequant_kernel.h"
	#include "paddle/phi/kernels/funcs/layer_norm_impl.cu.h"

		@@ -0,0 +1,91 @@
		// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.

		@@ -0,0 +1,157 @@
		// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.

Move fused_attention op to phi [迁移前向 GPU OpKernel] #51743

Move fused_attention op to phi [迁移前向 GPU OpKernel] #51743

Conversation

AndSonder commented Mar 16, 2023 • edited Loading

PR types

PR changes

Describe

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

From00 left a comment

Choose a reason for hiding this comment

ZzSean left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

AndSonder commented Mar 16, 2023 •

edited

Loading