[New Features] add llm pretrain & lora & sft & prefix_tuning testing scripts #7056
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #7056 +/- ##
===========================================
- Coverage 59.91% 59.84% -0.07%
===========================================
Files 556 557 +1
Lines 82035 82150 +115
===========================================
+ Hits 49148 49161 +13
- Misses 32887 32989 +102 |
wj-Mcat
changed the title
llm/chatglm/sft_argument.json
Outdated
| "dataset_name_or_path": "./data", | ||
| "output_dir": "./checkpoints/chatglm_sft_ckpts", | ||
| "per_device_train_batch_size": 4, | ||
| "gradient_accumulation_steps": 4, | ||
| "per_device_eval_batch_size": 8, | ||
| "eval_accumulation_steps":16, | ||
| "num_train_epochs": 3, | ||
| "learning_rate": 3e-05, | ||
| "warmup_steps": 30, | ||
| "logging_steps": 1, | ||
| "evaluation_strategy": "epoch", | ||
| "save_strategy": "epoch", | ||
| "src_length": 1024, | ||
| "max_length": 2048, | ||
| "fp16": true, | ||
| "fp16_opt_level": "O2", | ||
| "do_train": true, | ||
| "do_eval": true, | ||
| "disable_tqdm": true, | ||
| "load_best_model_at_end": true, | ||
| "eval_with_do_generation": false, | ||
| "metric_for_best_model": "accuracy", | ||
| "recompute": true, | ||
| "save_total_limit": 1, | ||
| "tensor_parallel_degree": 4, | ||
| "pipeline_parallel_degree": 1 | ||
| } |
There was a problem hiding this comment.
这个是 pre-commit 给 format 的
| llama: | ||
| model_name_or_path: __internal_testing__/tiny-random-llama |
| llama: | ||
| model_name_or_path: __internal_testing__/tiny-random-llama | ||
| chatglm: | ||
| model_name_or_path: __internal_testing__/tiny-fused-chatglm |
There was a problem hiding this comment.
tiny-fused-chatglm这里的fused指的是什么
There was a problem hiding this comment.
因为 fused 模型中的 head_dim 必须是在：10,26,32,64，128 .... 中的一个，所以为了不影响之前的单测，就直接新创建了一个tiny-random 的模型专门用来做 非 fused & fuse 相关的单测
| pipeline_parallel_degree: 1 | ||
| default: | ||
| llama: | ||
| model_name_or_path: __internal_testing__/tiny-random-llama |
There was a problem hiding this comment.
因为 fused 模型中的 head_dim 必须是在：10,26,32,64，128 .... 中的一个，所以为了不影响之前的单测，就直接新创建了一个tiny-random 的模型专门用来做 非 fused & fuse 相关的单测
| model_type: llama | ||
| model_name_or_path: __internal_testing__/tiny-random-llama | ||
| chatglm: | ||
| model_type: chatglm |
There was a problem hiding this comment.
这里是没有的，然后 fine-tune 这里也没有配置 chatglm，我可以把它先删掉。
tests/llm/test_finetune.py
Outdated
| LLMTest.tearDown(self) | ||
| shutil.rmtree(self.data_dir) | ||
|
|
||
| def test_pretrain(self): |
There was a problem hiding this comment.
copy 过来的还没有改，我来调整一下。
tests/llm/test_finetune.py
Outdated
| self.data_dir = tempfile.mkdtemp() | ||
| sys.path.insert(0, self.model_dir) | ||
|
|
||
| # Run pretrain |
| use_few_examples("dev.json") | ||
| use_few_examples("validation.json") | ||
|
|
||
| def tearDown(self) -> None: |
There was a problem hiding this comment.
因为是 from_pretrained 的，所以会缓存在 .paddlenlp/models 目录下，应该也没必要删除吧。
tests/llm/test_lora.py
Outdated
| self.data_dir = os.path.join(self.data_dir, "data") | ||
| self.use_small_datasets() | ||
|
|
||
| def use_small_datasets(self): |
tests/llm/test_prefix_tuning.py
Outdated
| self.data_dir = os.path.join(self.data_dir, "data") | ||
| self.use_small_datasets() | ||
|
|
||
| def use_small_datasets(self): |
|
|
||
| merge() | ||
|
|
||
| if self.model_dir not in ["chatglm2"]: |
There was a problem hiding this comment.
这里是说chatglm2没有办法动转静然后预测是吗？这里记个TODO吧
There was a problem hiding this comment.
chatglm2 还没有支持 细粒度 op，所以这里需要给移除掉
PR types
New features
PR changes
Others
Description
添加 LLM 目录下的finetuning 单测脚本