Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify core avx and core_noavx to libpaddle #46095

Merged
merged 11 commits into from
Sep 16, 2022

Conversation

chenwhql
Copy link
Contributor

@chenwhql chenwhql commented Sep 15, 2022

PR types

Function optimization

PR changes

Others

Describe

Unify core avx and core_noavx to libpaddle

之前paddle为了使一个whl包同时支持avx和noavx,采用了一种比较trick的方式,编译两遍paddle,分别编译avx和noavx,最后打包到一起,#17889

这种方式虽然解决了问题,但也引入了新的问题:

  1. 编译效率差
  2. paddle whl包体积大,由于体积过大的问题,之前在linux和windows已经重新拆分为两个不同的包,由用户进行选择下载安装
  3. copy修改文件名导致soname不一致,外部无法正确链接,之前通过patchelf修复了此问题,但是patchelf本身也有一些使用限制,比如限制包的大小,目前在gcc5.4+cuda10.1、gcc9+cuda11上都会触发包大小上限无法正常使用,因此还是需要从根本上解决此问题,使动态库名称一致且规范,Fix core so name mismatch error #43977

本PR清理了一些历史代码,将动态库名称统一改为libpaddle,参考linux shared library的命名公约,https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

同时需要在PaddleCustomDevice中对这两个命名的使用增加一个if分支:https://github.com/PaddlePaddle/PaddleCustomDevice/search?q=core_avx

PaddlePaddle/PaddleCustomDevice#112

@paddle-bot
Copy link

paddle-bot bot commented Sep 15, 2022

你的PR提交成功,感谢你对开源项目的贡献!
请关注后续CI自动化测试结果,详情请参考Paddle-CI手册
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

pangyoki
pangyoki previously approved these changes Sep 16, 2022
paddle/scripts/paddle_build.sh Show resolved Hide resolved
python/paddle/fluid/core.py Show resolved Hide resolved
Copy link
Contributor

@pangyoki pangyoki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jzhang533
Copy link
Contributor

之前paddle为了使一个whl包同时支持avx和noavx,采用了一种比较trick的方式,编译两遍paddle,分别编译avx和noavx,最后打包到一起,#17889

才知道这个历史原因。
我们还是要解决在kernel实现上能够做到屏蔽CPU扩展指令,才能根本解决这个问题。

@chenwhql
Copy link
Contributor Author

之前paddle为了使一个whl包同时支持avx和noavx,采用了一种比较trick的方式,编译两遍paddle,分别编译avx和noavx,最后打包到一起,#17889

才知道这个历史原因。 我们还是要解决在kernel实现上能够做到屏蔽CPU扩展指令,才能根本解决这个问题。

是的,需要从底层解决

@chenwhql chenwhql merged commit 267d71a into PaddlePaddle:develop Sep 16, 2022
chenwhql added a commit to chenwhql/Paddle that referenced this pull request Sep 16, 2022
* unify  core_avx and core_noavx

* fix except error

* revert mac compile logic

* revert dylib to so

* add core_noavx branch

* remove core_noavx

* replace paddle_core by lib paddle

* polish var name

* replace paddle_core by libpaddle

* update custom device commit

* polish code by comments
phlrain pushed a commit that referenced this pull request Sep 19, 2022
* unify  core_avx and core_noavx

* fix except error

* revert mac compile logic

* revert dylib to so

* add core_noavx branch

* remove core_noavx

* replace paddle_core by lib paddle

* polish var name

* replace paddle_core by libpaddle

* update custom device commit

* polish code by comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants