feat(ai): AI增加多机多卡 分布式训练 #1261
Merged
feat(ai): AI增加多机多卡 分布式训练 #1261
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: 17ae366 The changes in this PR will be included in the next version bump. This PR includes changesets to release 15 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
OYX-1
force-pushed
the
feat-multi-pods-train-ai
branch
from
f01c99e to
b28579f
Compare
OYX-1
force-pushed
the
feat-multi-pods-train-ai
branch
from
7eac59c to
c311d9f
Compare
ddadaal
requested changes
Jul 12, 2024
| @@ -5,7 +5,7 @@ | |||
| "main": "build/index.js", | |||
| "private": true, | |||
| "scripts": { | |||
| "generate": "rimraf generated && buf generate --template buf.gen.yaml https://github.com/PKUHPC/scow-scheduler-adapter-interface.git#branch=ai-multi-pod-train", | |||
| "generate": "rimraf generated && buf generate --template buf.gen.yaml https://github.com/PKUHPC/scow-scheduler-adapter-interface.git#branch=feat-ai-release", | |||
There was a problem hiding this comment.
这个功能准备好了吗？如果准备好了，适配器接口需要发布一个新的版本，并且这个功能需要检查集群的适配器的版本是否是已经实现了新的接口的新版本。SCOW主分支代码不能使用未正式发布的适配器接口。
There was a problem hiding this comment.
现在scow的master上就是feat-ai-release这个分支，不是版本号。
要在adapter-interface仓库里把feat-ai-release合到master中才能继续之前的检查适配器版本操作。
当时好像是说ai模块还是beta版就先不合
There was a problem hiding this comment.
#1361 区分了SCOW AI和SCOW所使用的接口版本。此功能以及其他SCOW部分准备好的接口应该合并到master，并在v1.5.0的基础上发一个新的版本v1.6.0，之后在此PR中使用v1.6.0的接口版本
There was a problem hiding this comment.
#1361 已经合并了，这个PR的proto修改是只对AI的吧？那就
- 在接口项目中，发起一个PR，把这个功能所用到的修改合并到ai的分支里
- 当接口合并到AI的分支后，SCOW项目的本分支合并master，然后在ai项目用的proto项目里用ai的分支
pkuhpc-review-bot
bot
added
Code-ChangeRequested
and removed
Code-ReviewRequested
OYX-1
approved these changes
Jul 18, 2024
pkuhpc-review-bot
bot
added
Code-ReviewRequested
ddadaal
requested changes
Jul 18, 2024
pkuhpc-review-bot
bot
added
Code-ChangeRequested
and removed
Code-ReviewRequested
pkuhpc-review-bot
bot
added
Code-ReviewRequested
pkuhpc-review-bot
bot
added
Code-ChangeRequested
and removed
Code-ReviewRequested
pkuhpc-review-bot
bot
added
Code-ReviewRequested
ddadaal
approved these changes
Jul 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
改动
集群信息新增gpuType和vramMb 参数，同时训练时需要将gpuType传回给适配器
新增多机多卡功能，当用户在AI 训练时，如果选择多个节点（pod），则需要指定对应的算法框架，目前支持 tensorflow、pytorch和mindspore（华为特有），来进行对应的分布式训练
AI应用配置文件新增tag,并且新增三个接口：listApp, listTags,以及根据appId 来获取对应可创建该应用的集群信息 listClusters，这部分接口主要是为了后续 AI作业模块重构准备。
gpu分区提交作业和应用必传gpuType告知适配器是是什么gpu类型
若是华为gpu卡，不管是不是分布式训练，提交应用和作业都需要指定框架