[NPU][LLM] add README & reformat llama scripts #8642
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
SylarTiaNII
force-pushed
the
reformat_npu_llama
branch
from
c511f7c
to
5419849
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #8642 +/- ##
===========================================
+ Coverage 55.80% 55.81% +0.01%
===========================================
Files 620 620
Lines 96642 96599 -43
===========================================
- Hits 53928 53917 -11
+ Misses 42714 42682 -32 ☔ View full report in Codecov by Sentry. |
|
LGTM |
ZHUI
reviewed
Jun 21, 2024
llm/llama/npu/README.md
Outdated
| @@ -0,0 +1,198 @@ | |||
| ## 🚣♂️ 使用PaddleNLP在NPU下跑通llama2-13b模型 🚣 | |||
| PaddleNLP在昇腾NPU（[了解昇腾](https://www.hiascend.com/zh/ecosystem/industry)）上对llama2-13B模型进行了深度适配和优化，该套件实现了昇腾NPU和GPU的训推入口基本统一，达到了『无缝切换』的效果。 | |||
There was a problem hiding this comment.
https://github.com/PaddlePaddle/PaddleNLP/tree/develop/llm/npu/llama
文件放到 llm/npu 下面，我们重构的pr已经合入了
ronny1996
suggested changes
Jun 21, 2024
llm/llama/npu/README.md
Outdated
| } | ||
| ``` | ||
| 为了保障极致压缩的推理成本，我们使用了静态图实现。因此需要从训练产出的动态图模型中导出静态图模型，执行如下命令进行导出： | ||
| ``` |
There was a problem hiding this comment.
改为
cd PaddleNLP/llm
python predict/export_model.py --model_name_or_path merged_model --inference_model --output_path ./inference --dtype float16 --device npu --block_attn
当前看必须在 PaddleNLP/llm 才跑得通
llm/llama/npu/README.md
Outdated
| ``` | ||
| 最终，我们通过静态图的模型执行推理： | ||
| ``` | ||
| # 执行推理代码 |
There was a problem hiding this comment.
改为
# 执行推理代码
python predict/predictor.py --model_name_or_path ./inference --inference_model --dtype "float16" --mode "static" --block_attn --device npu
|
https://github.com/PaddlePaddle/PaddleNLP/blob/develop/llm/predict/export_model.py#L104 这里帮忙改成 |
SylarTiaNII
force-pushed
the
reformat_npu_llama
branch
2 times, most recently
from
43c883d
to
f9f424c
Compare
ZHUI
requested changes
Jun 22, 2024
llm/export_npu.sh
Outdated
| source /usr/local/Ascend/ascend-toolkit/set_env.sh | ||
| source /usr/local/Ascend/atb/set_env.sh | ||
|
|
||
| export PYTHONPATH=../:$PYTHONPATH |
llm/predict_npu.sh
Outdated
|
|
||
| model_path=${1:-"./inference"} | ||
|
|
||
| source /usr/local/Ascend/ascend-toolkit/set_env.sh |
SylarTiaNII
force-pushed
the
reformat_npu_llama
branch
from
f9f424c
to
7ab4a98
Compare
ZHUI
approved these changes
Jun 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Description
add README for llama training/inference on NPU(910)