[LLM] Add Yuan model #8654
[LLM] Add Yuan model #8654
Conversation
|
Thanks for your contribution! |
|
Lint问题可以参考link进行修复 |
|
我看lint的日志中引起black、isort、copyright_checker failed的文件已经被修改了，请问我还需要修改吗？或者这个错误具体指什么？我没看到引起错误的源文件？ |
具体是格式错误问题，PR中的文件需要满足要求格式。可以本地使用
辛苦拉一下最近代码即可，最新commit已经修复。 |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #8654 +/- ##
===========================================
- Coverage 55.74% 55.42% -0.32%
===========================================
Files 623 626 +3
Lines 97456 98057 +601
===========================================
+ Hits 54323 54351 +28
- Misses 43133 43706 +573 ☔ View full report in Codecov by Sentry. |
|
请问目前的三个failed应该如何修改？ |
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """ Yuan model tools""" |
There was a problem hiding this comment.
这串代码建议封装成函数，避免出现2，24这样的魔鬼数字，明文模型路径等；
建议搞成参数传入；
There was a problem hiding this comment.
已修改。请问codecov中的warning应该如何修改？
There was a problem hiding this comment.
这个应该不影响代码合入，找commiter帮忙合入就可以了
There was a problem hiding this comment.
@DesmonDay 如果没有什么问题，麻烦帮忙合入，谢谢
.pre-commit-config.yaml
Outdated
| @@ -1,21 +1,21 @@ | |||
| exclude: 'model_zoo/gpt-3' | |||
| repos: | |||
| # For Python files | |||
| - repo: https://github.com/psf/black.git | |||
| - repo: https://gitee.com/wygfzren/black.git | |||
There was a problem hiding this comment.
此文件请修改回原版本，会影响其他贡献者对代码进行格式化
paddlenlp/transformers/__init__.py
Outdated
| @@ -296,3 +296,5 @@ | |||
| from .deberta_v2.configuration import * | |||
| from .qwen2 import * | |||
| from .qwen2_moe import * | |||
| from .yuan.modeling import * | |||
There was a problem hiding this comment.
请使用 from .yuan import * 进行导入，并在yuan文件夹下的__init__.py中import 导入modeling和configuration
| from paddle.distributed import fleet | ||
| from paddle.nn import CrossEntropyLoss | ||
|
|
||
| from paddlenlp.transformers.conversion_utils import ( |
There was a problem hiding this comment.
paddlenlp库内部函数和类，推荐使用相对路径import
| return q_embed, k_embed | ||
|
|
||
|
|
||
| class YuanPreTrainedModel(PretrainedModel): |
There was a problem hiding this comment.
此处需要修改模型名称为YuanPretrainedModel，否则在auto import时会产生报错。当前PaddleNLP导入规则为模型自定义名称（如Qwen2， Yuan） + 固定类型（如PretrainedModel, ForCausalLM）
| model_mappings.extend(layer_mappings) | ||
|
|
||
| init_name_mappings(mappings=model_mappings) | ||
| # base-model prefix "LlamaModel" |
| if module._padding_idx is not None: | ||
| module.weight.data[module._padding_idx].zero_() | ||
|
|
||
| def _set_gradient_checkpointing(self, module, value=False): |
There was a problem hiding this comment.
PaddleNLP中使用recompute来控制重计算，可参考llama相关重计算设置
|
|
||
| hidden_states = inputs_embeds | ||
|
|
||
| if self.gradient_checkpointing and self.training: |
There was a problem hiding this comment.
此处判断参数应为recompute，实现细节可参考llama代码
|
请问，test测试不通过应该如何修改？ |
非常感谢您的贡献。
|
1、已修改 |
PR types
New features
PR changes
Models
Description
添加了源2.0的模型结构、配置等相关文件