【Hackathon 7th No.38】为 Paddle 代码转换工具新增 API 转换规则（第5组） #496

inaomIIsfarell · 2024-10-15T06:52:30Z

PR Docs

PR APIs

torch.isposinf
torch.isneginf
torch.isreal
torch.isin
torch.Tensor.isposinf
torch.Tensor.isneginf
torch.Tensor.isreal
torch.Tensor.scatter_reduce
torch.scatter_reduce
torch.positive
torch.Tensor.positive
torch.concatenate
torch.can_cast
torch.float_power
torch.block_diag
torch.cartesian_prod

原pr：#487

paddle-bot · 2024-10-15T06:52:35Z

Thanks for your contribution!

inaomIIsfarell · 2024-10-15T07:42:09Z

有些CI报错本地复现不了，例如PR-CI-UnitTest中的AttributeError（ CI报错如下图）

我在本地跑是没问题的

测试用例代码

生成的paddle_aux代码

测试用例中pytorch代码转换后的paddle代码

luotao1 · 2024-10-16T13:01:00Z

参考下这个：#495 (comment)

…to hkt_paconv

zhwesky2010 · 2024-10-21T03:25:24Z

@inaomIIsfarell CI未通过，请先自查问题

inaomIIsfarell · 2024-10-22T06:37:06Z

@inaomIIsfarell CI未通过，请先自查问题

报错问题在本地无法复现，PR-CI-UnitTest 中报错的三个测试用例的共同特点是 torch 代码中的相对应方法都 传入所有参数且全部指定关键字，我在本地可以成功通过测试，以下附上CI中报错的测试用例的本地代码转换测试截图和生成的test_project/utils/paddle_aux.py代码

test_case_2

test_case_4

test_case_5

test_project/utils/paddle_aux.py

# This file is generated by PaConvert ToolKit, please Don't edit it!
import paddle

def can_cast(from_, to):
    can_cast_dict = {
        'bfloat16': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': False,
            'int8': False,
            'int16': False,
            'int32': False,
            'int64': False,
            'bool': False
        },
        'float16': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': False,
            'int8': False,
            'int16': False,
            'int32': False,
            'int64': False,
            'bool': False,
        },
        'float32': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': False,
            'int8': False,
            'int16': False,
            'int32': False,
            'int64': False,
            'bool': False,
        },
        'float64': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': False,
            'int8': False,
            'int16': False,
            'int32': False,
            'int64': False,
            'bool': False,
        },
        'complex64': {
            'bfloat16': False,
            'float16': False,
            'float32': False,
            'float64': False,
            'complex64': True,
            'complex128': True,
            'uint8': False,
            'int8': False,
            'int16': False,
            'int32': False,
            'int64': False,
            'bool': False,
        },
        'complex128': {
            'bfloat16': False,
            'float16': False,
            'float32': False,
            'float64': False,
            'complex64': True,
            'complex128': True,
            'uint8': False,
            'int8': False,
            'int16': False,
            'int32': False,
            'int64': False,
            'bool': False,
        },
        'uint8': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': True,
            'int8': True,
            'int16': True,
            'int32': True,
            'int64': True,
            'bool': False,
        },
        'int8': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': True,
            'int8': True,
            'int16': True,
            'int32': True,
            'int64': True,
            'bool': False,
        },
        'int16': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': True,
            'int8': True,
            'int16': True,
            'int32': True,
            'int64': True,
            'bool': False,
        },
        'int32': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': True,
            'int8': True,
            'int16': True,
            'int32': True,
            'int64': True,
            'bool': False,
        },
        'int64': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': True,
            'int8': True,
            'int16': True,
            'int32': True,
            'int64': True,
            'bool': False,
        },
        'bool': {
            'bfloat16': True,
            'float16': True,
            'float32': True,
            'float64': True,
            'complex64': True,
            'complex128': True,
            'uint8': True,
            'int8': True,
            'int16': True,
            'int32': True,
            'int64': True,
            'bool': True,
        }
    }
    return can_cast_dict[from_][to]
setattr(paddle, 'can_cast', can_cast)

zhwesky2010

这个定位到问题了：是由于CI中torch的版本不够新，目前已经更新了版本

下面这个问题更新一下，然后重新提交一遍触发CI重跑

zhwesky2010 · 2024-10-25T03:48:35Z

paconvert/api_matcher.py

+                    }
+                }
+                return can_cast_dict[from_][to]
+            setattr(paddle, 'can_cast', can_cast)


这个既然最后调用的是：

paddle_aux.can_cast

这里就不需要去设置一个paddle.can_cast了

已去除setattr

PR-CI-GPU-UnitTest 中出现 comment 中相同的报错问题

PR-CI-UnitTest 中出现我未修改内容部分的报错

ptal @zhwesky2010

补充：本地删除本地project文件夹后，首次运行会报错，但如果project文件夹已经生成则本地可以正常运行

…to hkt_paconv

paconvert/api_matcher.py

zhwesky2010 · 2024-11-05T04:11:12Z

paconvert/api_matcher.py

@@ -3531,6 +3562,13 @@ def generate_code(self, kwargs):
        return code


+class CartesianProdMatcher(BaseMatcher):


参考一下CreateMatcher吧，这个应该有多种输入的用法

zhwesky2010 · 2024-11-05T04:12:17Z

paconvert/api_matcher.py

+
+    def generate_code(self, kwargs):
+        self.write_aux_code()
+        if "input" in kwargs and kwargs["input"] is not None:


这两个判断不是一个意思吗，只需要判断 if "input" in kwargs 就行

zhwesky2010 · 2024-11-05T04:12:53Z

paconvert/api_matcher.py

+            """
+            def get_exponent(exponent):
+                return exponent.cast(paddle.float64) if isinstance(exponent, paddle.Tensor) else exponent
+            setattr(paddle, "get_exponent", get_exponent)


不要这样setattr，所有写法都改一下

zhwesky2010 · 2024-11-05T04:13:32Z

paconvert/api_matcher.py

+    def generate_aux_code(self):
+        CODE_TEMPLATE = textwrap.dedent(
+            """
+            def get_exponent(exponent):


命名规范一点，这个应该是 cast_exponent ？

zhwesky2010 · 2024-11-05T04:15:49Z

paconvert/api_matcher.py

+        return CODE_TEMPLATE
+
+    def generate_code(self, kwargs):
+        self.write_aux_code()


kwargs["input"] is not None没必要判断，没有这种用法。input不可能输入None，out有可能按默认的就是None

…to hkt_paconv

zhwesky2010 · 2024-11-07T03:51:06Z

paconvert/api_matcher.py

+            )
+            if "out" in kwargs and kwargs["out"] is not None:
+                code = "paddle.assign({}, {})".format(pow_expression, kwargs["out"])
+            else:


这个分支也是没必要，你可以把pow_expression命名为code

zhwesky2010 · 2024-11-07T03:51:12Z

paconvert/api_matcher.py

+    def generate_code(self, kwargs):
+        self.write_aux_code()
+        if "input" in kwargs:
+            pow_expression = "paddle.pow({}.cast(paddle.float64), paddle_aux.cast_exponent({}))".format(


zhwesky2010 · 2024-11-07T03:52:22Z

tests/test_cartesian_prod.py

+    obj.run(pytorch_code, ["result"])
+
+
+def test_case_3():


测一下可变参数的用法

c = (a, b) *c

这种

关键字参数的用法

tensors = (a, b)

要与Matcher的分支一一对应，如果没有这种用法，在Matcher里可以删掉这个分支了

torch对这个api似乎不支持指定关键字参数的用法

zhwesky2010 · 2024-11-07T03:57:44Z

paconvert/api_matcher.py

@@ -3534,6 +3553,25 @@ def generate_code(self, kwargs):
        return code


+class CartesianProdMatcher(BaseMatcher):


那你这个能否复用 ScalableVarMatcher 呢

复用 ScalableVarMatcher 时，在tests/test_cartesian_prod.py中test_case_2()中报错ValueError: Expect a 1D vector, but got shape []，但是代码能正常转换，我当时写的时候没定位到这个错误，这才写了一个新的 Matcher 。您看我是复用 ScalableVarMatcher 还是接着用自己写的这个Matcher呢

复用 ScalableVarMatcher 时，在tests/test_cartesian_prod.py中test_case_2()中报错ValueError: Expect a 1D vector, but got shape []，但是代码能正常转换，我当时写的时候没定位到这个错误，这才写了一个新的 Matcher 。您看我是复用 ScalableVarMatcher 还是接着用自己写的这个Matcher呢

可以定位一下吧，看是否优化下原Matcher，尽可能复用

ScalableVarMatcher中 ( p1 ) 的dest_var_arg_value = self.parse_args(args)[0]导致dest_var_arg_value只接收到了args的第一个值而非整个列表，导致了在 tests/test_cartesian_prod.py中test_case_2() ，kwargs为 {'x': 'a'}，导致生成的paddle_temp代码参数有误 ( p2 ) ，而实际上paddle.cartesian_prod()的参数需要是list(Tensor)。我不清楚如果改了这里之后对其他部分内容有没有影响？

这里是需要改动一下，p1 中红框圈起来的地方有些问题，我在其他使用了这个Matcher的api里测试了一下，如果torch中只传入形参且形参不是tuple或list时，paddle的api中的参数会从列表变成一个值，同上边的p2

@inaomIIsfarell 那你这个跟之前的ScalableVarMatcher逻辑还是有点区别，之前的逻辑是支持以下用法的

api(3, 4, 5) api(3) api([3, 4, 5])

你这个API仅支持前两种用法，不支持为list的用法，因此还是重新写个简化版的ScalableVarMatcher吧，不要辅助函数

zhwesky2010 · 2024-11-07T03:58:21Z

tests/test_block_diag.py

+
+from apibase import APIBase
+
+obj = APIBase("torch.block_diag")


有可变参数、关键字参数的用法吗？有的话可以加一下

同上
指定关键字参数时torch报错，应该是不支持指定关键字参数的

zhwesky2010 · 2024-11-11T10:49:21Z

@inaomIIsfarell 那你这个跟之前的ScalableVarMatcher逻辑还是有点区别，之前的逻辑是支持以下用法的

1. api(3, 4, 5)
2. api(3)
3. api([3, 4, 5])

shape = (3, 4, 5)
4. api(shape)
5. api(*shape)

你这个API仅支持前1、2、5用法，不支持3、4的用法，因此还是重新写个简化版的ScalableVarMatcher吧，不需要辅助函数。类似于这样：

if len(args) > 1:
   dest_var_arg_value = self.parse_args(args)
else:
    if isinstance(args[0], ast.Starred):
        dest_var_arg_value = astor.to_source(args[0].value).strip("\n")
    else:
        dest_var_arg_value = self.parse_args(args)

zhwesky2010 · 2024-11-12T09:15:53Z

paconvert/api_matcher.py

+
+class CartesianProdMatcher(BaseMatcher):
+    def get_paddle_nodes(self, args, kwargs):
+        if len(args) > 1 or (len(args) == 1 and isinstance(args[0], ast.Constant)):


这个输入不可能是常数，按我给出的示例，分支的分类有问题吗

zhwesky2010 · 2024-11-12T09:19:54Z

tests/test_Tensor_float_power.py

@@ -16,7 +16,7 @@

 from apibase import APIBase

-obj = APIBase("torch.Tensor.float_power")
+obj = APIBase("torch.Tensor.float_power", is_aux_api=True)


加了辅助函数，需要新增一些对应的case吧

zhwesky2010 · 2024-11-12T09:25:20Z

tests/test_Tensor_permute.py

@@ -16,7 +16,7 @@

 from apibase import APIBase

-obj = APIBase("torch.Tensor.permute")
+obj = APIBase("torch.Tensor.permute", is_aux_api=True)


这个需要加is_aux_api吗

zhwesky2010 · 2024-11-12T09:26:37Z

tests/test_Tensor_tile.py

@@ -16,7 +16,7 @@

 from apibase import APIBase

-obj = APIBase("torch.Tensor.tile")
+obj = APIBase("torch.Tensor.tile", is_aux_api=True)


这个需要加is_aux_api吗

…to hkt_paconv

inaomIIsfarell · 2024-11-12T10:01:09Z

@zhwesky2010
重新push了，请review，辛苦了

zhwesky2010

LGTM

【Hackathon 7th No.38】为 Paddle 代码转换工具新增 API 转换规则（第5组）

91d3629

luotao1 mentioned this pull request Oct 15, 2024

【Hackathon 7th】开源贡献个人挑战赛 PaddlePaddle/Paddle#68244

Open

fix_

23e0c1e

paddle-bot bot added the contributor External developers label Oct 15, 2024

luotao1 assigned luotao1 and zhwesky2010 Oct 16, 2024

inaomIIsfarell added 4 commits October 17, 2024 12:12

fix

7a879b5

Merge branch 'master' of https://github.com/PaddlePaddle/PaConvert in…

20beda8

…to hkt_paconv

fix

1475d5a

Merge branch 'master' of https://github.com/PaddlePaddle/PaConvert in…

f9b5567

…to hkt_paconv

zhwesky2010 reviewed Oct 25, 2024

View reviewed changes

inaomIIsfarell added 2 commits October 26, 2024 11:18

fix

8f1c550

Merge branch 'master' of https://github.com/PaddlePaddle/PaConvert in…

a3f0d47

…to hkt_paconv

PaddlePaddle locked and limited conversation to collaborators Nov 5, 2024

PaddlePaddle unlocked this conversation Nov 5, 2024

PaddlePaddle locked and limited conversation to collaborators Nov 5, 2024

PaddlePaddle unlocked this conversation Nov 5, 2024

zhwesky2010 reviewed Nov 5, 2024

View reviewed changes

PaddlePaddle locked and limited conversation to collaborators Nov 5, 2024

PaddlePaddle unlocked this conversation Nov 5, 2024

inaomIIsfarell added 2 commits November 5, 2024 21:13

fix_

ec204c4

Merge branch 'master' of https://github.com/PaddlePaddle/PaConvert in…

d33bf12

…to hkt_paconv

zhwesky2010 reviewed Nov 7, 2024

View reviewed changes

inaomIIsfarell added 2 commits November 8, 2024 15:55

fix_

68c21c1

fix

0125c58

inaomIIsfarell added 2 commits November 8, 2024 17:28

fix

7029182

hkt 7th No.38

8fcc36b

inaomIIsfarell added 2 commits November 12, 2024 17:01

fix

cc4b66d

fix

d4bb1b1

zhwesky2010 reviewed Nov 12, 2024

View reviewed changes

inaomIIsfarell added 2 commits November 12, 2024 17:57

fix

5d2a45d

Merge branch 'master' of https://github.com/PaddlePaddle/PaConvert in…

217885f

…to hkt_paconv

inaomIIsfarell added 2 commits November 12, 2024 18:07

fix

2cb8b7e

fix

ebbca6f

zhwesky2010 approved these changes Nov 12, 2024

View reviewed changes

zhwesky2010 merged commit 5f91749 into PaddlePaddle:master Nov 12, 2024
6 of 7 checks passed

		@@ -3531,6 +3562,13 @@ def generate_code(self, kwargs):
		return code


		class CartesianProdMatcher(BaseMatcher):

		@@ -3534,6 +3553,25 @@ def generate_code(self, kwargs):
		return code


		class CartesianProdMatcher(BaseMatcher):


		from apibase import APIBase

		obj = APIBase("torch.block_diag")

【Hackathon 7th No.38】为 Paddle 代码转换工具新增 API 转换规则（第5组） #496

【Hackathon 7th No.38】为 Paddle 代码转换工具新增 API 转换规则（第5组） #496

Conversation

inaomIIsfarell commented Oct 15, 2024

PR Docs

PR APIs

paddle-bot bot commented Oct 15, 2024

inaomIIsfarell commented Oct 15, 2024 • edited Loading

luotao1 commented Oct 16, 2024

zhwesky2010 commented Oct 21, 2024

inaomIIsfarell commented Oct 22, 2024

zhwesky2010 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inaomIIsfarell Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inaomIIsfarell Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhwesky2010 commented Nov 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inaomIIsfarell commented Nov 12, 2024

zhwesky2010 left a comment

Choose a reason for hiding this comment

inaomIIsfarell commented Oct 15, 2024 •

edited

Loading

zhwesky2010 left a comment •

edited

Loading

inaomIIsfarell Oct 26, 2024 •

edited

Loading

inaomIIsfarell Nov 8, 2024 •

edited

Loading

zhwesky2010 commented Nov 11, 2024 •

edited

Loading