-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
【Hackathon 6th Fundable Projects 1 1-1】Add _typing
module to paddle
#63604
Conversation
你的PR提交成功,感谢你对开源项目的贡献! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
注意一下与 paddlepaddle-stubs 不同的是,这里使用 .py
定义,所定义出来的是新的类型,是可能和运行时的类型混淆的(运行时真的有两个不同的 class),需要保证用户使用原有的类型时也是可以的,以及,用户一旦不小心使用了这里的 class 构造了运行时可访问的对象怎么办呢?
比如这里定义了 paddle._typing.dtype
,如何保证它和 paddle.dtype
(paddle.base.core.DataType
)是一样的呢?最佳方式是为 paddle.base.core.DataType
提供 stub file,使其从根本上保持一致
Tensor 等 pybind 数据结构一样,都可以这样考虑
@megemini 觉得呢?
对!大原则参考 RFC 里面说的:
可以这么想,
我们无法直接标注 pybind 的接口,并且,目前的检查工具没法提示这类东西的 docstring,因此,需要我们补充 stub ~ 而 至于这个 @SigureMo @Asthestarsfalll 帮忙看看这个思路有没有问题? |
@Asthestarsfalll 另外,这个任务需要补充 |
这个我没有太理解,stub file 更主要提供的是类型信息,docstring 则是次要的,为什么没有 docstring 的需要就可以不用 stub file 了呢? 其实根据 typeshed 的规范1,stub file 不应该写 docstring 的,当然,这个在社区里也有争议,但这里以是否有 docstring 来评判是否加 stub file 我觉得是颠倒主次关系的 其余感觉没啥问题的~ Footnotes |
我的表述有问题 ~~~ 是说这个
再使用 mypy 检查: import paddle
a = paddle.dtype.FP16
reveal_type(a) mypy 的输出: note: Revealed type is "paddle.framework.dtype.dtype"
这里有两个问题:
pybind 的接口如果用了 stub,好像必须得把 docstring 加进去,不然 vscode 我试了,貌似 docstring 的提示就没了? |
这里同意
不加stub的话,我这里类型提示会为unknown
这里我明天看下numpy的类型 |
想先确定好具体实现,后续再添加 |
python/paddle/__init__.pyi
Outdated
|
||
class Tensor: ... | ||
|
||
def to_tensor(data, dtype=None, place=None, stop_gradient=True) -> Tensor: ... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
不在pyi里加的话,mypy会找不到,我本地的lsp也找不到,但是我搜索到的资料是优先从pyi中找,其次是py。但这样看起来直接不从py找了
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
不过,_typing 模块,貌似需要在 init.py 中引入
import paddle._typing as _typing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
一般使用 from . import _typing as _typing
Sorry to inform you that dd9e04e's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
综合 review ,有几个问题:
-
python/paddle/__init__.pyi
感觉应该暂时不需要 ~这里面的 Tensor 会在 【Type Hints】Paddle 中引入 Tensor stub 文件 #63953 这里增加,而
MLUPlace NPUPlace
目前 paddle 没有暴露出去,个人感觉不需要单独加这个 stub ~ -
pyi
文件要打包到 wheel 里面,似乎要修改一下 setup.py 文件我在 [Typing] Paddle 的 CI 中引入 mypy 对于 API 中 docstring 的示例代码的类型检查 #63901 里面修改了两个地方
- python/setup.py.in
- setup.py
具体怎么改,还需要确认一下 ~
-
test_type_hints.py
CI 中没有测试到,应该也需要配置一下 ~ -
test/type_hints 里面缺少 fail 和 reveal 的测试用例 ~
-
mypy
的配置文件,写在pyproject.toml
里面 ~
具体涉及到 _typing
模块里面的东西,目前看应该够用,后面随着具体标注任务的进行,再修改也来得及 ~ 包括 _typing 模块的单测,可以放到后面再增加 ~
@Asthestarsfalll 辛苦!:)
@SigureMo 请帮忙看看上面的问题,尤其是 setup 和 test_type_hints 的配置,我也吃不准 ~
感谢二位!!!
python/paddle/__init__.pyi
Outdated
|
||
class Tensor: ... | ||
|
||
def to_tensor(data, dtype=None, place=None, stop_gradient=True) -> Tensor: ... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
python/paddle/__init__.pyi
Outdated
|
||
class Tensor: ... | ||
|
||
def to_tensor(data, dtype=None, place=None, stop_gradient=True) -> Tensor: ... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
不过,_typing 模块,貌似需要在 init.py 中引入
import paddle._typing as _typing
python/paddle/_typing/device_like.py
Outdated
class _Place(Protocol): | ||
def __init__(self, id: int) -> None: | ||
... | ||
|
||
|
||
NPUPlace = _Place | ||
MLUPlace = _Place |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
xxx = _xxx
这样写有点问题,比如:
from __future__ import annotations
class _Place:
def __init__(self, id: int) -> None:
...
NPUPlace = _Place
MLUPlace = _Place
def test(place: NPUPlace) -> None:
return
a = MLUPlace(0)
test(a)
这里面 a
类型实际是错的,但是 mypy 检查 pass ~
直接使用 =
会使两个类型没啥区别 ~
如果要写的话,还是单独列一个类 ~
不过,建议 NPUPlace MLUPlace
暂时不要了,因为 paddle 目前还没有暴露出来这俩个东西,只在 pyi 里面写可能会有异议 ~ 尽量不要增加 _typing 之外属于 api 的东西吧 ~
PlaceLike
支持 str
应该目前用足够 ~ 这样的话,init.py 和 init.pyi 也不需要这两个东西 ~
@SigureMo 看看行不?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
protocol 应该继承,(或者说是实现,这是组合的概念),而不是直接 assign,NPU 和 MLU 很早之前就在框架内退场了吧大概一年前左右的开源任务
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
不需要了是吧?~
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
对的,没有就删掉好了
Tuple[Union[int, Tensor, None], ...], | ||
List[Union[int, Tensor, None]], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
使用 tuple
和 list
吧 ~
另外,需要 from __future__ import annotations
这里手误写错了,不过我在本地测试过,这些代码都没问题,3.8.0可以使用list作为类型提示,现在添加了mypy进行测试
python 3.8 应该不能直接用 tuple
和 list
,我这里是 3.8.17
----> 1 a:list[int] = [1,2]
TypeError: 'type' object is not subscriptable
In [2]: from __future__ import annotations
In [3]: a:list[int] = [1,2]
你那边测试可能是 mypy 指定版本有问题?
python/unittest_py/requirements.txt
Outdated
@@ -19,3 +19,4 @@ wandb>=0.13 ; python_version<"3.12" | |||
xlsxwriter==3.0.9 | |||
xdoctest==1.1.1 | |||
ubelt==1.3.3 # just for xdoctest | |||
mypy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
统一一下版本号吧 mypy==1.10.0
test/type_hints/test_type_hints.py
Outdated
|
||
class TestTypeHints(unittest.TestCase): | ||
def check_with_mypy(self, file_path): | ||
stdout, stderr, exitcode = api.run([file_path]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
搞一个配置文件吧 ~
后面统一使用一个配置文件,如果你这里加了,后面我那里可以去掉~
需要配置 因此本 PR 就不涉及 mypy 的添加什么的吧,相关内容不要做重了,mypy 在 #63901 考虑
|
可以不用添加这个吧?
有两个地方要改,
|
我来改一下这个 PR,尽快推动本 PR 合入,以免阻塞整个项目进度 |
还有 #63953 没 review,是不是漏掉了 😂 |
没有哈,现在注意力在这俩上 |
_typing
module to paddle
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
…PaddlePaddle#63604) --------- Co-authored-by: SigureMo <sigure.qaq@gmail.com>
PR Category
Others
PR Types
New features
Description
大部分般自paddlepaddle-stubs
添加一些常用的以及可能会用到的,如
Size1
Shape1D
,名称可能还需要再考虑一下;后面还需要调研一下paddle内比较常用的类型提示有哪些测试结果:
Related links