[LLM] Support gpt3 fine grained dybatch v1 #7080
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
Codecov Report
@@ Coverage Diff @@
## develop #7080 +/- ##
===========================================
- Coverage 59.91% 59.78% -0.13%
===========================================
Files 556 558 +2
Lines 82037 82217 +180
===========================================
+ Hits 49149 49152 +3
- Misses 32888 33065 +177
|
yuanlehome
force-pushed
the
support_gpt3
branch
from
bac5c29
to
bd27eee
Compare
yuanlehome
force-pushed
the
support_gpt3
branch
from
bd27eee
to
c7ec8bf
Compare
| def set_state_dict(self, state_dict): | ||
| dtype = paddle.get_default_dtype() | ||
|
|
||
| for k, v in state_dict.items(): |
There was a problem hiding this comment.
这里的if代码有点多啊？能改的和llama里的一样吗？
There was a problem hiding this comment.
建议先这样写，因为gpt的模型来源比较复杂，参数名比较乱，这种if写法已经尽可能多的兼容各种命名的模型了
wj-Mcat
requested changes
Sep 20, 2023
| cls, pretrained_model_name_or_path, from_hf_hub: bool = False, subfolder: str | None = None, *args, **kwargs | ||
| ): | ||
| # TODO: Support safetensors loading. | ||
| kwargs["use_safetensors"] = False |
There was a problem hiding this comment.
Suggested change
| kwargs["use_safetensors"] = False | |
| kwargs["use_safetensors"] = kwargs.get("use_safetensors", False) |
建议使用这个，因为单分片 safetensors 是可以支持 inferencemodel 加载的。
| position_ids = tgt_pos | ||
| attention_mask = (tgt_generation_mask - 1) * 1e4 | ||
| else: | ||
| attention_mask = (attention_mask - 1) * 1e4 |
There was a problem hiding this comment.
这里建议使用：paddle.finfo(attention_mask.dtype).min 的方式来转化 attention_mask 的值。
在 bf16 和 fp16 下面不同值域不一样，建议用这个来得到该 dtype 下的最小值。
上面的 tgt_attention_mask 也是需要调整一下。
可以的，我想补充单测和comment指出的问题 放在下个PR一块做吧，这个PR先合一版？ |
wj-Mcat
approved these changes
Sep 20, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Description
Support gpt3 fine grained dybatch v1.