Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AutoParallel] add VPP pass #59751

Merged
merged 18 commits into from
Dec 12, 2023
Merged

[AutoParallel] add VPP pass #59751

merged 18 commits into from
Dec 12, 2023

Conversation

zhaoyinglia
Copy link
Contributor

@zhaoyinglia zhaoyinglia commented Dec 6, 2023 •
edited
Loading

PR types

New features

PR changes

Others

Description

  • set chunk_id attr for sharding, gradient_merge, recompute, sequence parallel and reshard
  • add VPP scheduler pass and get sub_program fuc
  • usage
    • when accumulate_steps == pp_degree, vpp schedule mode is based on FThenB
    • when accumulate_steps > pp_degree, vpp schedule mode is based on 1F1B
strategy = auto.Strategy()
pipeline = strategy.pipeline
pipeline.enable = True
pipeline.micro_batch_size = 1
pipeline.accumulate_steps = 4
pipeline.schedule_mode = "VPP"
  • TODO: Remove shard_op(..., chunk_id=...) to annotation chunk when schedule_mode is VPP

Others

Pcard-76459

@paddle-bot Paddle-bot
Copy link

paddle-bot bot commented Dec 6, 2023

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

XieYunshen
XieYunshen approved these changes Dec 12, 2023
View reviewed changes
Copy link
Contributor

@XieYunshen XieYunshen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Caozhou1995
Caozhou1995 approved these changes Dec 12, 2023
View reviewed changes
Copy link
Contributor

@Caozhou1995 Caozhou1995 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

JZ-LIANG
JZ-LIANG approved these changes Dec 12, 2023
View reviewed changes
Copy link
Contributor

@JZ-LIANG JZ-LIANG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JZ-LIANG JZ-LIANG merged commit 75e105c into PaddlePaddle:develop Dec 12, 2023
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Caozhou1995 Caozhou1995 Caozhou1995 approved these changes

@JZ-LIANG JZ-LIANG JZ-LIANG approved these changes

@XieYunshen XieYunshen XieYunshen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zhaoyinglia @XieYunshen @JZ-LIANG @Caozhou1995