[Prim] Add exp_double_grad, log_double_grad, abs_triple_grad, min_grad #63245

HydrogenSulfate · 2024-04-05T02:47:00Z

PR Category

Operator Mechanism

PR Types

New features

Description

Pcard-75624

Add 4 composite OPs: exp_double_grad, log_double_grad, abs_triple_grad, min_grad, and add
exp_double_grad, log_double_grad, abs_triple_grad to prim_white_list.
Supplement infermeta for silu_double_grad

Test exp_double_grad, log_double_grad, abs_triple_grad, min_grad with pytorch(eps=1e-8):

import numpy as np

import paddle

import torch


def test_log_triple_grad(x: np.ndarray):
    px = paddle.to_tensor(x)
    tx = torch.from_numpy(x).cuda()
    px.stop_gradient = False
    tx.requires_grad = True

    py = paddle.log(px)
    ty = torch.log(tx)

    pg = paddle.grad(py, px, create_graph=True)[0]
    tg = torch.autograd.grad(ty.sum(), tx, create_graph=True)[0]
    np.testing.assert_allclose(pg.numpy(), tg.detach().cpu().numpy(), 1e-8, 1e-8)

    pgg = paddle.grad(pg, px, create_graph=True)[0]
    tgg = torch.autograd.grad(tg.sum(), tx, create_graph=True)[0]
    np.testing.assert_allclose(pgg.numpy(), tgg.detach().cpu().numpy(), 1e-8, 1e-8)

    pggg = paddle.grad(pgg, px)[0]
    tggg = torch.autograd.grad(tgg.sum(), tx)[0]
    np.testing.assert_allclose(pggg.numpy(), tggg.detach().cpu().numpy(), 1e-8, 1e-8)


def test_abs_triple_grad(x: np.ndarray):
    px = paddle.to_tensor(x)
    tx = torch.from_numpy(x).cuda()
    px.stop_gradient = False
    tx.requires_grad = True

    py = paddle.nn.functional.silu(paddle.abs(px))
    ty = torch.nn.functional.silu(torch.abs(tx))

    pg = paddle.grad(py, px, create_graph=True)[0]
    tg = torch.autograd.grad(ty.sum(), tx, create_graph=True)[0]
    np.testing.assert_allclose(pg.numpy(), tg.detach().cpu().numpy(), 1e-8, 1e-8)

    pgg = paddle.grad(pg, px, create_graph=True)[0]
    tgg = torch.autograd.grad(tg.sum(), tx, create_graph=True)[0]
    np.testing.assert_allclose(pgg.numpy(), tgg.detach().cpu().numpy(), 1e-8, 1e-8)

    pggg = paddle.grad(pgg, px)[0]
    tggg = torch.autograd.grad(tgg.sum(), tx)[0]
    np.testing.assert_allclose(pggg.numpy(), tggg.detach().cpu().numpy(), 1e-8, 1e-8)


def test_exp_triple_grad(x: np.ndarray):
    px = paddle.to_tensor(x)
    tx = torch.from_numpy(x).cuda()
    px.stop_gradient = False
    tx.requires_grad = True

    py = paddle.nn.functional.silu(paddle.exp(px))
    ty = torch.nn.functional.silu(torch.exp(tx))

    pg = paddle.grad(py, px, create_graph=True)[0]
    tg = torch.autograd.grad(ty.sum(), tx, create_graph=True)[0]
    np.testing.assert_allclose(pg.numpy(), tg.detach().cpu().numpy(), 1e-8, 1e-8)

    pgg = paddle.grad(pg, px, create_graph=True)[0]
    tgg = torch.autograd.grad(tg.sum(), tx, create_graph=True)[0]
    np.testing.assert_allclose(pgg.numpy(), tgg.detach().cpu().numpy(), 1e-8, 1e-8)

    pggg = paddle.grad(pgg, px)[0]
    tggg = torch.autograd.grad(tgg.sum(), tx)[0]
    np.testing.assert_allclose(pggg.numpy(), tggg.detach().cpu().numpy(), 1e-8, 1e-8)


def test_exp_triple_grad(x: np.ndarray):
    px = paddle.to_tensor(x)
    tx = torch.from_numpy(x).cuda()
    px.stop_gradient = False
    tx.requires_grad = True

    py = paddle.nn.functional.silu(paddle.exp(px))
    ty = torch.nn.functional.silu(torch.exp(tx))

    pg = paddle.grad(py, px, create_graph=True)[0]
    tg = torch.autograd.grad(ty.sum(), tx, create_graph=True)[0]
    np.testing.assert_allclose(pg.numpy(), tg.detach().cpu().numpy(), 1e-8, 1e-8)

    pgg = paddle.grad(pg, px, create_graph=True)[0]
    tgg = torch.autograd.grad(tg.sum(), tx, create_graph=True)[0]
    np.testing.assert_allclose(pgg.numpy(), tgg.detach().cpu().numpy(), 1e-8, 1e-8)

    pggg = paddle.grad(pgg, px)[0]
    tggg = torch.autograd.grad(tgg.sum(), tx)[0]
    np.testing.assert_allclose(pggg.numpy(), tggg.detach().cpu().numpy(), 1e-8, 1e-8)


def test_min_triple_grad(x: np.ndarray):
    paddle.framework.core.set_prim_eager_enabled(True)

    px = paddle.to_tensor(x)
    tx = torch.from_numpy(x).cuda()
    px.stop_gradient = False
    tx.requires_grad = True

    py = paddle.nn.functional.silu(paddle.min(px, axis=1))
    ty = torch.nn.functional.silu(torch.min(tx, dim=1)[0])

    pg = paddle.grad(py, px, create_graph=True)[0]
    tg = torch.autograd.grad(ty.sum(), tx, create_graph=True)[0]
    np.testing.assert_allclose(pg.numpy(), tg.detach().cpu().numpy(), 1e-8, 1e-8)

    paddle.framework.core.set_prim_eager_enabled(False)

if __name__ == "__main__":
    x = np.random.uniform(0.1, 100.0, [1024, 1024])
    test_log_triple_grad(x)

    x = np.random.uniform(-100.0, 100.0, [1024, 1024])
    test_abs_triple_grad(x)

    x = np.random.uniform(-5.0, 5.0, [1024, 1024])
    test_exp_triple_grad(x)

    x = np.random.uniform(-100.0, 100.0, [1024, 1024])
    test_min_triple_grad(x)

paddle-bot · 2024-04-05T02:47:05Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

cyber-pioneer · 2024-04-07T03:01:35Z

paddle/fluid/pir/dialect/op_generator/vjp_interface_black_list.py

@@ -25,4 +25,7 @@

 vjp_interface_black_list = [
    'silu_grad',
+    'exp_grad',
+    'log_grad',
+    'abs_double_grad',


不加这个编译的时候会报错，好像是Paddle/build/paddle/fluid/pir/dialect/operator/ir/pd_op_vjp.cc文件里，然后报错信息是：primitive 下没有 XXX_grad_jvp，所以我仿照silu_double_grad把这个加上了

why here?

如果K阶反向只有组合实现，那么需要把K-1阶的算子名称添加到这个列表里，所以我删除了log_grad，保留了exp_grad和abs_double_grad

cyber-pioneer · 2024-04-07T03:02:31Z

test/prim/prim/vjp/eager/test_comp_eager_min_grad.py

@@ -0,0 +1,71 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.


heavyrain-lzy

LGTM for YAML

PaddlePaddle#63245) * add min_grad composite * support exp_double_grad, abs_triple_grad, log_double_grad composite OPs * add infermeta items * set_prim_all for TestLogDoubleGradCheck * change 2022 to 2024 in copyright header * remove 'log_grad' from vjp_interface_black_list

HydrogenSulfate added 3 commits April 4, 2024 09:24

add min_grad composite

21c4857

support exp_double_grad, abs_triple_grad, log_double_grad composite OPs

9233a9d

add infermeta items

f463283

set_prim_all for TestLogDoubleGradCheck

80ce980

cyber-pioneer reviewed Apr 7, 2024

View reviewed changes

HydrogenSulfate added 3 commits April 7, 2024 07:48

remove 'log_grad' from vjp_interface_black_list

98a43eb

Merge branch 'develop' into add_exp_and_log_composite

3bef74c

cyber-pioneer approved these changes Apr 8, 2024

View reviewed changes

heavyrain-lzy approved these changes Apr 8, 2024

View reviewed changes

HydrogenSulfate merged commit d04836a into PaddlePaddle:develop Apr 8, 2024
30 checks passed

HydrogenSulfate deleted the add_exp_and_log_composite branch April 8, 2024 04:55

HydrogenSulfate mentioned this pull request Apr 9, 2024

Implement the composition of abs_double_grad #62335

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Prim] Add exp_double_grad, log_double_grad, abs_triple_grad, min_grad #63245

[Prim] Add exp_double_grad, log_double_grad, abs_triple_grad, min_grad #63245

HydrogenSulfate commented Apr 5, 2024 •

edited

Loading

paddle-bot bot commented Apr 5, 2024

cyber-pioneer Apr 7, 2024

HydrogenSulfate Apr 7, 2024 •

edited

Loading

HydrogenSulfate Apr 7, 2024 •

edited

Loading

cyber-pioneer Apr 7, 2024

HydrogenSulfate Apr 7, 2024

heavyrain-lzy left a comment

		@@ -0,0 +1,71 @@
		# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.

[Prim] Add exp_double_grad, log_double_grad, abs_triple_grad, min_grad #63245

[Prim] Add exp_double_grad, log_double_grad, abs_triple_grad, min_grad #63245

Conversation

HydrogenSulfate commented Apr 5, 2024 • edited Loading

PR Category

PR Types

Description

paddle-bot bot commented Apr 5, 2024

cyber-pioneer Apr 7, 2024

Choose a reason for hiding this comment

HydrogenSulfate Apr 7, 2024 • edited Loading

Choose a reason for hiding this comment

HydrogenSulfate Apr 7, 2024 • edited Loading

Choose a reason for hiding this comment

cyber-pioneer Apr 7, 2024

Choose a reason for hiding this comment

HydrogenSulfate Apr 7, 2024

Choose a reason for hiding this comment

heavyrain-lzy left a comment

Choose a reason for hiding this comment

HydrogenSulfate commented Apr 5, 2024 •

edited

Loading

HydrogenSulfate Apr 7, 2024 •

edited

Loading

HydrogenSulfate Apr 7, 2024 •

edited

Loading