Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fused elementwises kernels and ops #51427

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
bd117a0
Fused elementwises kernels and ops
HulekJakub Mar 9, 2023
eb1081b
Merge branch 'develop' into fused_elementwise_ops
Silv3S Mar 9, 2023
2525681
change fuse pass name
Silv3S Mar 9, 2023
3c8b7d3
adjust .pbtxt files
Silv3S Mar 9, 2023
462f006
adjust quantization attributes
Silv3S Mar 9, 2023
b72bd52
add missing arguments and fix others, review fixed
HulekJakub Mar 9, 2023
90f9217
merge current 51427
Silv3S Mar 9, 2023
190e797
simplify fused kernel registration
Silv3S Mar 9, 2023
4f72943
fix elementwise unit tests
Silv3S Mar 9, 2023
1ad3f16
reuse one fused elementwise op
Silv3S Mar 10, 2023
f93b1a7
adjust proto
Silv3S Mar 10, 2023
e4be4cc
Merge pull request #1 from Silv3S/fused_elementwise_ops
HulekJakub Mar 10, 2023
0dc0b06
Merge branch 'develop' into Fused_Elementwise_Kernel_And_Op
HulekJakub Mar 13, 2023
079fb83
Add supported datatypes
HulekJakub Mar 13, 2023
e52e35d
Merge remote-tracking branch 'upstream/develop' into Fused_Elementwis…
HulekJakub Mar 14, 2023
3651b8e
Change 'Scale' to 'scale' in tests, change some tests to onednn
HulekJakub Mar 14, 2023
f4909e8
Revert breaking changes
HulekJakub Mar 16, 2023
b587ed0
Fix unit tests
HulekJakub Mar 17, 2023
734f4fe
Merge remote-tracking branch 'upstream/develop' into Fused_Elementwis…
HulekJakub Mar 17, 2023
4322409
Delete obsolete test cases
HulekJakub Mar 20, 2023
6d21d73
Delete commented out code
HulekJakub Mar 21, 2023
b0116a2
Merge remote-tracking branch 'upstream/develop' into Fused_Elementwis…
HulekJakub Mar 22, 2023
1705253
Merge branch 'develop' into Fused_Elementwise_Kernel_And_Op
HulekJakub Mar 22, 2023
f0d836c
Fix codestyle
HulekJakub Mar 22, 2023
a3c7d2d
Merge develop and fix conflicts
HulekJakub Mar 27, 2023
0f36edf
delete temporary condition
HulekJakub Mar 27, 2023
e1d5eb7
Merge remote-tracking branch 'upstream/develop' into Fused_Elementwis…
HulekJakub Mar 28, 2023
f119f6f
Merge branch 'Fused_Elementwise_Kernel_And_Op' of https://github.com/…
HulekJakub Mar 28, 2023
833c2f1
fix conflicts and delete duplicate fusing
HulekJakub Mar 28, 2023
db0831f
Merge branch 'PaddlePaddle:develop' into Fused_Elementwise_Kernel_And_Op
HulekJakub Apr 4, 2023
93b0b48
Merge branch 'PaddlePaddle:develop' into Fused_Elementwise_Kernel_And_Op
HulekJakub Apr 11, 2023
2c20932
Fix code after merge
HulekJakub Apr 11, 2023
4e600b4
Move tests to new directory
HulekJakub Apr 12, 2023
24decf5
Merge remote-tracking branch 'upstream/develop' into Fused_Elementwis…
HulekJakub Apr 17, 2023
8d60160
fix tests volatility
HulekJakub Apr 24, 2023
b977541
Merge branch 'PaddlePaddle:develop' into Fused_Elementwise_Kernel_And_Op
HulekJakub Apr 24, 2023
62ae00e
Merge branch 'PaddlePaddle:develop' into Fused_Elementwise_Kernel_And_Op
HulekJakub Apr 25, 2023
a9dadce
Merge branch 'PaddlePaddle:develop' into Fused_Elementwise_Kernel_And_Op
HulekJakub Apr 26, 2023
d93d827
Merge branch 'PaddlePaddle:develop' into Fused_Elementwise_Kernel_And_Op
HulekJakub Apr 27, 2023
8920095
Merge remote-tracking branch 'upstream/develop' into Fused_Elementwis…
HulekJakub May 5, 2023
47a4745
Merge branch 'PaddlePaddle:develop' into Fused_Elementwise_Kernel_And_Op
HulekJakub May 8, 2023
7d2fa1a
Rename test_elementwise_add_onednn_op.py to test_elementwise_add_mkld…
HulekJakub May 16, 2023
20c3c33
Update CMakeLists.txt add mkldnn op test
HulekJakub May 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion paddle/fluid/framework/ir/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ if(WITH_MKLDNN)
pass_library(softplus_activation_onednn_fuse_pass inference DIR mkldnn)
pass_library(shuffle_channel_mkldnn_detect_pass inference DIR mkldnn)
pass_library(fc_act_mkldnn_fuse_pass inference DIR mkldnn)
pass_library(elt_act_mkldnn_fuse_pass inference DIR mkldnn)
pass_library(elementwise_act_onednn_fuse_pass inference DIR mkldnn)
pass_library(matmul_elementwise_add_mkldnn_fuse_pass inference DIR mkldnn)
pass_library(matmul_activation_mkldnn_fuse_pass inference DIR mkldnn)
pass_library(layer_norm_onednn_optimization_pass inference DIR mkldnn)
Expand Down
8 changes: 6 additions & 2 deletions paddle/fluid/framework/ir/graph_pattern_detector.cc
Original file line number Diff line number Diff line change
Expand Up @@ -2260,7 +2260,8 @@ PDNode *patterns::OpRequant::operator()() {
auto any_op = pattern->NewNode(any_op_repr())
->assert_is_op()
->assert_more([&](Node *node) {
return node->Op()->HasAttr("Scale_out") ? true : false;
return (node->Op()->HasAttr("Scale_out") ||
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can Scale_out be deleted?

Copy link
Contributor Author

@HulekJakub HulekJakub Mar 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There isn't a consistent naming scheme and many operators have "Scale" and many others have "scale", so it can be deleted once all operators have the same capitalization scheme, but it is outside of this PR scope

node->Op()->HasAttr("scale_out"));
});
auto requant_in = pattern->NewNode(requant_in_repr())
->assert_is_op_input("requantize", "Input");
Expand Down Expand Up @@ -2288,7 +2289,10 @@ PDNode *patterns::RequantOp::operator()() {
->assert_more([&](Node *node) {
return (node->Op()->HasAttr("Scale_in") ||
node->Op()->HasAttr("Scale_x") ||
node->Op()->HasAttr("Scale_y"));
node->Op()->HasAttr("Scale_y") ||
node->Op()->HasAttr("scale_in") ||
node->Op()->HasAttr("scale_x") ||
node->Op()->HasAttr("scale_y"));
});

requant_op->LinksFrom({requant_in}).LinksTo({requant_out});
Expand Down
12 changes: 6 additions & 6 deletions paddle/fluid/framework/ir/mkldnn/cpu_quantize_pass.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1045,14 +1045,14 @@ void CPUQuantizePass::QuantizeElementwise(
"X",
input_x_scale,
is_x_unsigned,
"Scale_x");
"scale_x");
QuantizeInput(g,
elementwise_op,
elementwise_y,
"Y",
input_y_scale,
is_y_unsigned,
"Scale_y");
"scale_y");

bool is_output_unsigned{false};
auto output_scale =
Expand All @@ -1064,7 +1064,7 @@ void CPUQuantizePass::QuantizeElementwise(
"Out",
output_scale,
is_output_unsigned,
"Scale_out");
"scale_out");

++quantize_elementwise_count;
};
Expand Down Expand Up @@ -1314,9 +1314,9 @@ void CPUQuantizePass::ApplyImpl(ir::Graph* graph) const {
QuantizeImmutable(graph, "nearest_interp", "X");
QuantizeImmutable(graph, "nearest_interp_v2", "X");
QuantizeImmutable(graph, "split", "X");
QuantizeElementwise(graph, "elementwise_add");
QuantizeElementwise(graph, "elementwise_mul");
QuantizeElementwise(graph, "elementwise_sub");
QuantizeElementwise(graph, "fused_elementwise_add");
QuantizeElementwise(graph, "fused_elementwise_mul");
QuantizeElementwise(graph, "fused_elementwise_sub");
QuantizeFusionGru(graph);
QuantizeMultiGru(graph);
QuantizeFusionLSTM(graph);
Expand Down
34 changes: 19 additions & 15 deletions paddle/fluid/framework/ir/mkldnn/cpu_quantize_pass_tester.cc
Original file line number Diff line number Diff line change
Expand Up @@ -97,14 +97,15 @@ void SetOp(ProgramDesc* prog,
op->SetAttr("Scale_x", 1.0f);
op->SetAttr("Scale_y", 1.0f);
op->SetAttr("Scale_out", 1.0f);
} else if (type == "elementwise_add" || type == "elementwise_mul" ||
type == "elementwise_sub") {
} else if (type == "fused_elementwise_add" ||
type == "fused_elementwise_sub" ||
type == "fused_elementwise_mul") {
op->SetInput("X", {inputs[0]});
if (inputs.size() > 1) op->SetInput("Y", {inputs[1]});
op->SetOutput("Out", {outputs[0]});
op->SetAttr("Scale_x", 1.0f);
op->SetAttr("Scale_y", 1.0f);
op->SetAttr("Scale_out", 1.0f);
op->SetAttr("scale_x", 1.0f);
op->SetAttr("scale_y", 1.0f);
op->SetAttr("scale_out", 1.0f);
} else if (type == "fusion_gru") {
op->SetInput("X", {inputs[0]});
op->SetInput("Bias", {inputs[1]});
Expand Down Expand Up @@ -178,16 +179,19 @@ void CheckScales(const OpDesc* op, float scale, float shift) {
scale);
scale_names.push_back("Scale_in");
scale_names.push_back("Scale_out");
} else if (type == "fused_matmul" || type == "elementwise_add" ||
type == "elementwise_mul" || type == "elementwise_sub") {
} else if (type == "fused_matmul") {
scale_names.push_back("Scale_x");
scale_names.push_back("Scale_y");
scale_names.push_back("Scale_out");
if (type == "fused_matmul") {
auto const& names = op->InputNames();
if (std::find(names.begin(), names.end(), "ResidualData") != names.end())
scale_names.push_back("Scale_in_eltwise");
}
auto const& names = op->InputNames();
if (std::find(names.begin(), names.end(), "ResidualData") != names.end())
scale_names.push_back("Scale_in_eltwise");
} else if (type == "fused_elementwise_add" ||
type == "fused_elementwise_sub" ||
type == "fused_elementwise_mul") {
scale_names.push_back("scale_x");
scale_names.push_back("scale_y");
scale_names.push_back("scale_out");
} else if (type == "fusion_gru" || type == "fusion_lstm") {
EXPECT_EQ(op->GetAttrIfExists<float>("Shift_data"), shift);
EXPECT_EQ(op->GetAttrIfExists<std::vector<float>>("Scale_weights")[0],
Expand Down Expand Up @@ -710,9 +714,9 @@ void TestElementwiseUnsignedAndSignedInput(
}

const std::vector<std::vector<std::string>> elementwises = {
{"elementwise_add", "ElementwiseAdd"},
{"elementwise_mul", "ElementwiseMul"},
{"elementwise_sub", "ElementwiseSub"}};
{"fused_elementwise_add", "FusedElementwiseAdd"},
{"fused_elementwise_mul", "FusedElementwiseMul"},
{"fused_elementwise_sub", "FusedElementwiseSub"}};

class TestElementwises
: public testing::TestWithParam<std::vector<std::string>> {};
Expand Down
10 changes: 4 additions & 6 deletions paddle/fluid/framework/ir/mkldnn/cpu_quantize_placement_pass.cc
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ void CPUQuantizePlacementPass::ApplyImpl(ir::Graph* graph) const {
"fused_conv2d",
"fused_conv3d",
"fused_matmul",
"fused_elementwise_add",
"fused_elementwise_mul",
"fused_elementwise_sub",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove if-block cpu_quantize_placement_pass.cc:90
// Remove this condition when all fused_elementwise ops are merged
It was temporary solution, which is solved by your PR

"elementwise_add",
"elementwise_mul",
"elementwise_sub",
Expand Down Expand Up @@ -87,12 +90,7 @@ void CPUQuantizePlacementPass::ApplyImpl(ir::Graph* graph) const {
return;
}

// Remove this condition when all fused_elementwise ops are merged
if (!(op->Op()->Type() == "elementwise_add" ||
op->Op()->Type() == "elementwise_sub" ||
op->Op()->Type() == "elementwise_mul")) {
ConvertToFusedOp(op->Op());
}
ConvertToFusedOp(op->Op());
op->Op()->SetAttr("mkldnn_data_type", std::string("int8"));
};
gpd(graph, handler);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.

#include "paddle/fluid/framework/ir/mkldnn/elt_act_mkldnn_fuse_pass.h"
#include "paddle/fluid/framework/ir/mkldnn/elementwise_act_onednn_fuse_pass.h"

#include "paddle/fluid/framework/ir/graph_pattern_detector.h"
#include "paddle/fluid/framework/ir/mkldnn/activation_onednn_fuse_pass.h"
Expand Down Expand Up @@ -62,6 +62,7 @@ void ElementwiseActivationOneDNNPass::FuseElementwiseAct(
GET_IR_NODE_FROM_SUBGRAPH(
activation_out, activation_out, elementwise_act_pattern);

ConvertToFusedOp(elementwise->Op());
SetActivationAttrs(elementwise->Op(), activation->Op(), act_type);
elementwise->Op()->SetOutput("Out", {activation_out->Name()});

Expand All @@ -84,9 +85,9 @@ void ElementwiseActivationOneDNNPass::FuseElementwiseAct(
} // namespace framework
} // namespace paddle

REGISTER_PASS(elt_act_mkldnn_fuse_pass,
REGISTER_PASS(elementwise_act_onednn_fuse_pass,
paddle::framework::ir::ElementwiseActivationOneDNNPass);
REGISTER_PASS_CAPABILITY(elt_act_mkldnn_fuse_pass)
REGISTER_PASS_CAPABILITY(elementwise_act_onednn_fuse_pass)
.AddCombination(
paddle::framework::compatible::OpVersionComparatorCombination()
.LE("elementwise_add", 1)
Expand Down
2 changes: 2 additions & 0 deletions paddle/fluid/framework/ir/mkldnn/mkldnn_pass_util.h
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,8 @@ inline void ConvertToFusedOp(OpDesc* op) {
const std::map<std::string, std::string> fused_ops = {
{"conv2d", "fused_conv2d"},
{"depthwise_conv2d", "fused_conv2d"},
{"elementwise_add", "fused_elementwise_add"},
{"elementwise_sub", "fused_elementwise_sub"},
{"elementwise_mul", "fused_elementwise_mul"},
{"elementwise_div", "fused_elementwise_div"},
{"matmul", "fused_matmul"},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ void FuseOperatorScaleOneDNNPass::ApplyImpl(Graph *graph) const {
"fused_matmul",
"matmul",
"matmul_v2",
"fused_elementwise_add",
"fused_elementwise_sub",
"fused_elementwise_mul",
"fused_elementwise_div",
"elementwise_add",
Expand Down Expand Up @@ -119,6 +121,8 @@ REGISTER_PASS_CAPABILITY(operator_scale_onednn_fuse_pass)
.EQ("fused_matmul", 0)
.LE("matmul", 1)
.EQ("matmul_v2", 0)
.EQ("fused_elementwise_add", 0)
.EQ("fused_elementwise_sub", 0)
.EQ("fused_elementwise_mul", 0)
.EQ("fused_elementwise_div", 0)
.LE("elementwise_add", 1)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@

#include "paddle/fluid/framework/ir/mkldnn/mkldnn_pass_util.h"
#include "paddle/fluid/framework/op_version_registry.h"
#include "paddle/phi/backends/onednn/onednn_reuse.h"
#include "paddle/utils/string/pretty_log.h"

namespace paddle {
Expand Down
2 changes: 1 addition & 1 deletion paddle/fluid/inference/api/paddle_pass_builder.cc
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,7 @@ void CpuPassStrategy::EnableMKLDNN() {
"batch_norm_act_fuse_pass", //
"softplus_activation_onednn_fuse_pass", //
"shuffle_channel_mkldnn_detect_pass", //
"elt_act_mkldnn_fuse_pass", //
"elementwise_act_onednn_fuse_pass", //
"layer_norm_onednn_optimization_pass", //
"operator_scale_onednn_fuse_pass", //
"operator_unsqueeze2_onednn_fuse_pass", //
Expand Down
8 changes: 0 additions & 8 deletions paddle/fluid/operators/compat/elementwise_add.pbtxt
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,6 @@ extra {
attrs {
name: "Out0_threshold"
type: FLOAT
}
attrs {
name: "x_data_format"
type: STRING
}
attrs {
name: "y_data_format"
type: STRING
}
attrs {
name: "Scale_x"
Expand Down
10 changes: 1 addition & 9 deletions paddle/fluid/operators/compat/elementwise_div.pbtxt
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,7 @@ def {
}
extra {
attrs {
name: "x_data_format"
type: STRING
}
attrs {
name: "y_data_format"
name: "act"
type: STRING
}
attrs {
Expand All @@ -35,8 +31,4 @@ extra {
name: "Scale_out"
type: FLOAT
}
attrs {
name: "act"
type: STRING
}
}
10 changes: 0 additions & 10 deletions paddle/fluid/operators/compat/elementwise_mul.pbtxt
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,6 @@ def {
name: "axis"
type: INT
}
}
extra {
attrs {
name: "x_data_format"
type: STRING
}
attrs {
name: "y_data_format"
type: STRING
}
attrs {
name: "Scale_x"
type: FLOAT
Expand Down
10 changes: 1 addition & 9 deletions paddle/fluid/operators/compat/elementwise_pow.pbtxt
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,7 @@ def {
}
extra {
attrs {
name: "x_data_format"
type: STRING
}
attrs {
name: "y_data_format"
name: "act"
type: STRING
}
attrs {
Expand All @@ -35,8 +31,4 @@ extra {
name: "Scale_out"
type: FLOAT
}
attrs {
name: "act"
type: STRING
}
}
10 changes: 1 addition & 9 deletions paddle/fluid/operators/compat/elementwise_sub.pbtxt
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,7 @@ def {
}
extra {
attrs {
name: "x_data_format"
type: STRING
}
attrs {
name: "y_data_format"
name: "act"
type: STRING
}
attrs {
Expand All @@ -35,8 +31,4 @@ extra {
name: "Scale_out"
type: FLOAT
}
attrs {
name: "act"
type: STRING
}
}
50 changes: 50 additions & 0 deletions paddle/fluid/operators/compat/fused_elementwise_add.pbtxt
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
type: "fused_elementwise_add"
def {
inputs {
name: "X"
}
inputs {
name: "Y"
}
outputs {
name: "Out"
}
attrs {
name: "axis"
type: INT
}
}
extra {
attrs {
name: "fuse_activation"
type: STRING
}
attrs {
name: "fuse_alpha"
type: FLOAT
}
attrs {
name: "fuse_beta"
type: FLOAT
}
attrs {
name: "fused_output_scale"
type: FLOAT
}
attrs {
name: "fused_unsqueeze2_axes"
type: INTS
}
attrs {
name: "scale_x"
type: FLOAT
}
attrs {
name: "scale_y"
type: FLOAT
}
attrs {
name: "scale_out"
type: FLOAT
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also float fused_output_scale from operator_scale_onednn_fuse_pass

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All comments to fused_elementwise_add.pbtxt are valid for other fused_elementwise_.pbtxt

Loading