Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inference's naive_executor use InterperterCore #57920

Merged

Conversation

yuanlehome
Copy link
Contributor

@yuanlehome yuanlehome commented Oct 8, 2023

PR types

New features

PR changes

Others

Description

  • 新执行器部分功能打平旧执行器,如hook、nvtx等
  • 由于新执行器的多线程执行会导致Paddle-TRT执行异常,并且某个业务模型由于执行次序导致了精度异常,因此推理默认采用新执行器的单线程模式。
  • 新增 config.EnableNewExecutor (c++) 和 config.enable_new_executor (python) 来控制,一方面是方便测试,另一方面是给推理完全切换到新执行器一个缓冲期。

备注:新增临时接口不建议外部用户使用,仅短期内内部测试使用。

测试结论:

  • 显存收益明显,大小模型上有几十兆到数G的收益
  • 推理性能下降5%-10%左右,分析的原因包括但不限于:新执行器调度开销比较大,旧执行器(几百纳秒)vs 新执行器(几微秒)

TODO:

  • 新执行器(调度等)性能优化
  • 新旧执行器功能打平,如predictor clone等

Others

Pcard-71500

@paddle-bot
Copy link

paddle-bot bot commented Oct 8, 2023

你的PR提交成功,感谢你对开源项目的贡献!
请关注后续CI自动化测试结果,详情请参考Paddle-CI手册
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@yuanlehome yuanlehome force-pushed the predictor_with_new_executor branch 7 times, most recently from b550b3e to 3ce71f1 Compare October 23, 2023 13:51
Copy link
Contributor

@sunzhongkai588 sunzhongkai588 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@From00 From00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@XiaoguangHu01 XiaoguangHu01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yuanlehome yuanlehome merged commit f092821 into PaddlePaddle:develop Oct 25, 2023
28 checks passed
danleifeng pushed a commit to danleifeng/Paddle that referenced this pull request Nov 14, 2023
…dle#57920)

* inference use InterperterCore

* delete some unused member in AnalysisPredictor

* fix compile

* add ScopedFlushDenormal

* set new_executor_serial_run default to true

* default true

* new executor support hook

* default to false

* add nvtx

* fix comile on win

* fix comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants