Add new mistral #7425
Add new mistral #7425
Conversation
|
Thanks for your contribution! |
…nto add-new-mistral
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #7425 +/- ##
===========================================
+ Coverage 55.61% 55.74% +0.12%
===========================================
Files 620 623 +3
Lines 96965 97450 +485
===========================================
+ Hits 53930 54322 +392
- Misses 43035 43128 +93 ☔ View full report in Codecov by Sentry. |
| def set_input_embeddings(self, value): | ||
| self.embed_tokens = value | ||
|
|
||
| def _prepare_decoder_attention_mask( |
There was a problem hiding this comment.
确认一下目前attention_mask符合我们的标准，支持2d和4d， 1和0的含义一致，可以支持intokens策略
|
@wtmlon 不需要了吗？ |
|
This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动，被标记为stale。 |
llm/data.py
Outdated
| # Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. |
llm/data.py
Outdated
| from paddlenlp.peft import LoRAModel, PrefixModelForCausalLM | ||
|
|
||
|
|
||
| def get_convert_example(model): |
There was a problem hiding this comment.
mistral有chat_template吗。确认支持吗？
There was a problem hiding this comment.
llm/data.py
Outdated
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at |
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| from .configuration import MistralConfig |
There was a problem hiding this comment.
官方的 mistral 直接用的 LlamaTokenizer：https://huggingface.co/mistralai/Mistral-7B-v0.3/blob/b67d6a03ca097c5122fa65904fce0413500bf8c8/tokenizer_config.json#L6183
llm/mistral/lora_argument.json
Outdated
| @@ -0,0 +1,30 @@ | |||
| { | |||
There was a problem hiding this comment.
放到config文件里面，config目录里新增mistral包含json和readme。顺带更新llm目录下的readme
|
支持zero_padding策略吗，同时新增支持dpo |
都已支持 |
PR types
PR changes
Description
新增mistral模型