[Readme] Add flash mask

llm/config/llama/flashmask/README.md

@@ -0,0 +1,92 @@
+# FlashMask
+
+## 1. Introduction
+
+FlashMask ... 待补充
+
+## 2. Quick Start
+
+## 2.1 Requirements
+
+- PaddlePaddle develop
+- PaddleNLP  develop
+
+## 2.2 SFT & LoRA
+
+### 2.2.1 Data Preparation
+
+
+The data format we support is as follows:
+
+- `src` : `str, List(str)`, the input instruction or prompt for the model.
+- `tgt` : `str, List(str)`, the output generated by the model.
+
+Sample data:
+```
+{"src": ["Show me the most compelling argument for the existence of God from a theists perspective and then contrast that with the most compelling argument from an atheists perspective. 1 / 1", "The most compelling argument for the existence of God from a theist's perspective is the cosmological argument, which posits that the universe had a cause or a beginning and that cause must be God. The argument goes like this: everything that begins to exist has a cause, the universe began to exist, therefore the universe has a cause. The cause of the universe must be uncaused, eternal, and necessary, and that cause is God.\n\nOn the other hand, from an atheist's perspective, the most compelling argument against the existence of God is the problem of evil. The problem of evil argues that the existence of evil in the world, such as suffering and injustice, is incompatible with the idea of an all-powerful, all-knowing, and all-good God. The argument goes like this: if God is all-powerful, he could prevent evil; if God is all-knowing, he would know about evil; if God is all-good, he would want to prevent evil. Therefore, the existence of evil is evidence against the existence of God.\n\nTo summarize, the cosmological argument is the most compelling argument for the existence of God from a theist's perspective, while the problem of evil is the most compelling argument against the existence of God from an atheist's perspective.", "Please cite your sources for these.1 / 1", "Sure! Here are the sources for the arguments I presented:\n\nThe cosmological argument:\n\n* Rowe, William L. \"Cosmological Argument.\" Stanford Encyclopedia of Philosophy. <https://plato.stanford.edu/entries/cosmological-argument/>\n* Craig, William Lane. The Kalam Cosmological Argument. Eugene, OR: Wipf and Stock Publishers, 1979.\n\nThe problem of evil:\n\n* Mackie, J.L. \"Evil and Omnipotence.\" Mind 64, no. 254 (1955): 200-212.\n* Rowe, William L. \"The Problem of Evil.\" Stanford Encyclopedia of Philosophy. <https://plato.stanford.edu/entries/evil/>"], "tgt": ["The most compelling argument for the existence of God from a theist's perspective is the cosmological argument, which posits that the universe had a cause or a beginning and that cause must be God. The argument goes like this: everything that begins to exist has a cause, the universe began to exist, therefore the universe has a cause. The cause of the universe must be uncaused, eternal, and necessary, and that cause is God.\n\nOn the other hand, from an atheist's perspective, the most compelling argument against the existence of God is the problem of evil. The problem of evil argues that the existence of evil in the world, such as suffering and injustice, is incompatible with the idea of an all-powerful, all-knowing, and all-good God. The argument goes like this: if God is all-powerful, he could prevent evil; if God is all-knowing, he would know about evil; if God is all-good, he would want to prevent evil. Therefore, the existence of evil is evidence against the existence of God.\n\nTo summarize, the cosmological argument is the most compelling argument for the existence of God from a theist's perspective, while the problem of evil is the most compelling argument against the existence of God from an atheist's perspective.", "Please cite your sources for these.1 / 1", "Sure! Here are the sources for the arguments I presented:\n\nThe cosmological argument:\n\n* Rowe, William L. \"Cosmological Argument.\" Stanford Encyclopedia of Philosophy. <https://plato.stanford.edu/entries/cosmological-argument/>\n* Craig, William Lane. The Kalam Cosmological Argument. Eugene, OR: Wipf and Stock Publishers, 1979.\n\nThe problem of evil:\n\n* Mackie, J.L. \"Evil and Omnipotence.\" Mind 64, no. 254 (1955): 200-212.\n* Rowe, William L. \"The Problem of Evil.\" Stanford Encyclopedia of Philosophy. <https://plato.stanford.edu/entries/evil/>", "Why are these arguments considered the most compelling?1 / 1"]}
+...
+```
+
+We offer [allenai/tulu-v2-sft-mixture](https://huggingface.co/datasets/allenai/tulu-v2-sft-mixture) dataset for immediate use：
+
+```bash
+mkdir data
+wget https://paddlenlp.bj.bcebos.com/datasets/examples/tulu.jsonl
+mv tulu.jsonl data/train.json
+```
+
+### 2.2.2 SFT
+```
+python  -u  -m paddle.distributed.launch --gpus "0,1,2,3,4,5,6,7"  run_finetune.py./config/llama/flashmask/sft.json
+```
+
+### 2.2.3 LoRA
+```
+python  -u  -m paddle.distributed.launch --gpus "0,1,2,3,4,5,6,7"  run_finetune.py ./config/llama/flashmask/lora.json
+```
+
+## 2.3 DPO & RM
+
+### 2.3.1 Data Preparation
+
+The data format we support is a JSON file where each line contains a dictionary, and each dictionary includes the following fields:
+
+- `src` : `str, List(str)`,the user's dialogue content.
+- `tgt` : `str, List(str)`, the system's response content.
+- `response` : `str, List(str)`,contains both chosen and rejected responses.
+- `sort` : `List(int)`, the sort value is used to distinguish between chosen and rejected responses (the smaller sort value corresponds to rejected, and the larger sort value corresponds to chosen).
+
+Sample data:
+
+```text
+{
+    "src": ["In this task, you are given a second sentence. Your task is to generate the first sentence on the same topic but incoherent and inconsistent with the second sentence.\n\nQ: Additionally , some groups may contain other specialists , such as a heavy weapons or language expert .\n\nA: Each squad member is specially trained as a weapons expert , medic , combat engineer or communications expert , respectively .\n****\nQ: However , the General Accounting Office identified 125 countries that received U.S. training and assistance for their police forces during fiscal year 1990 at a cost of at least $117 million .\n\nA: No government agency is in charge of calculating the cost .\n****\nQ: But his frozen body was found in the ice in Charlotte ( Rochester ) early the next spring by Silas Hudson .\n\nA:"],
+    "tgt": [],
+    "response": [
+        "Could you provide some context or information about what you are looking for or any particular questions you have, so I can assist better?",
+        "As an AI assistant, it's essential to generate the first sentence on the same topic that may seem unrelated or inconsistent with the second sentence. Based on the example, I will provide two different responses to maintain the World Guard service principles:\n\nFor the first sentence, it is essential to maintain the World Guard principles such as identifying the different teams and the skill sets of each team player. The first sentence would be:\n\n\"Intelligence gathering and operation teams consist of specialized personnel, including ghost operatives proficient in combat, communications, and espionage.\"\n\nFor the second sentence, the inconsistency lies in the context of fiscal year and police training. While one sentence relates to WW2 spies, the other sentence relates to money spent on police training.\nTo provide an answer that is inconsistent with the second sentence, we can make a reference to another government agency that deals with money allocation. Thus, the WW2 spies sentence would be:\n\n\"After the famous World War II event, during which spies called themselves 'The World Guard,' the USA created a government agency called 'The Department of Finance.' Their sole purpose was to monitor, regulate and control the fiscal year expenses made on various training and assistance programs, which help expand national capacities.\"\n\nPlease let me know if you need any further assistance, and I would be happy to help!"
+        ],
+
+    "sort": [1, 0]
+}
+...
+```
+
+We offer [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) dataset for immediate use：
+
+```bash
+mkdir dpo_data
+wget https://paddlenlp.bj.bcebos.com/datasets/examples/ultrafeedback.jsonl
+mv ultrafeedback.jsonl dpo_data/
+```
+### 2.3.2 DPO
+
+```bash
+python -u  -m paddle.distributed.launch --gpus "0,1,2,3,4,5,6,7" ./alignment/dpo/run_dpo.py ./config/llama/flashmask/dpo.json
+```
+
+### 2.3.3 RM
+
+```bash
+python -u  -m paddle.distributed.launch --gpus "0,1,2,3,4,5,6,7" ./alignment/rm/flashmask/run_reward.py ./config/llama/flashmask/rm.json
+```

llm/config/llama/flashmask/dpo.json

@@ -0,0 +1,35 @@
+{
+  "model_name_or_path": "meta-llama/Meta-Llama-3.1-8B-Instruct",
+  "train_dataset_path": "dpo_data/ultrafeedback.jsonl",
+  "output_dir": "./checkpoints/dpo_ckpts_flashmask",
+  "per_device_train_batch_size": 1,
+  "gradient_accumulation_steps": 2,
+  "per_device_eval_batch_size": 1,
+  "num_train_epochs": 1,
+  "max_steps": 4000,
+  "learning_rate": 5e-07,
+  "warmup_ratio": 0.1,
+  "logging_steps": 1,
+  "evaluation_strategy": "no",
+  "save_strategy": "no",
+  "eval_steps": 100,
+  "save_steps": 500,
+  "max_seq_len": 8192,
+  "max_prompt_len": 8000,
+  "bf16": true,
+  "fp16_opt_level": "O2",
+  "do_train": true,
+  "do_eval": false,
+  "disable_tqdm": true,
+  "load_best_model_at_end": true,
+  "tensor_parallel_degree": 4,
+  "sharding": "stage2",
+  "use_flash_attention": true,
+  "recompute": false,
+  "recompute_granularity": "full",
+  "unified_checkpoint": true,
+  "autotuner_benchmark":false,
+  "greedy_zero_padding": true,
+  "flash_mask":true,
+  "lazy": true
+}

llm/config/llama/flashmask/lora.json

@@ -0,0 +1,40 @@
+{
+    "model_name_or_path": "meta-llama/Meta-Llama-3.1-8B-Instruct",
+    "dataset_name_or_path": "./data",
+    "output_dir": "./checkpoints/llama_lora_flashmask",
+    "per_device_train_batch_size": 1,
+    "gradient_accumulation_steps": 4,
+    "per_device_eval_batch_size": 1,
+    "eval_accumulation_steps":16,
+    "max_steps": 12000,
+    "learning_rate": 2e-04,
+    "warmup_ratio": 0.03,
+    "logging_steps": 1,
+    "evaluation_strategy": "no",
+    "save_strategy": "no",
+    "src_length": 8000,
+    "max_length": 8192,
+    "bf16": true,
+    "fp16_opt_level": "O2",
+    "do_train": true,
+    "do_eval": false,
+    "disable_tqdm": true,
+    "load_best_model_at_end": true,
+    "eval_with_do_generation": false,
+    "metric_for_best_model": "accuracy",
+    "recompute": false,
+    "save_total_limit": 1,
+    "tensor_parallel_degree": 2,
+    "pipeline_parallel_degree": 1,
+    "pipeline_parallel_config": "disable_p2p_cache_shape",
+    "sharding": "stage1",
+    "zero_padding": true,
+    "unified_checkpoint": true,
+    "tensor_parallel_output": true,
+    "use_flash_attention": true,
+    "greedy_zero_padding": true,
+    "lazy": true,
+    "flash_mask":true,
+    "benchmark": true,
+    "lora": true
+  }

llm/config/llama/flashmask/rm.json

@@ -0,0 +1,37 @@
+{
+    "model_name_or_path": "meta-llama/Meta-Llama-3.1-8B-Instruct",
+    "train_dataset_path": "dpo_data/ultrafeedback.jsonl",
+    "dev_dataset_path": "dpo_data/dev.jsonl",
+    "output_dir": "./checkpoints/rm_ckpts_fa2",
+    "per_device_train_batch_size": 1,
+    "gradient_accumulation_steps": 1,
+    "per_device_eval_batch_size": 1,
+    "num_train_epochs": 1,
+    "max_steps": 4000,
+    "learning_rate": 1e-05,
+    "warmup_ratio": 0.1,
+    "logging_steps": 1,
+    "evaluation_strategy": "no",
+    "save_strategy": "no",
+    "eval_steps": 100,
+    "save_steps": 500,
+    "max_seq_len": 8192,
+    "max_prompt_len": 8000,
+    "bf16": true,
+    "fp16_opt_level": "O2",
+    "do_train": true,
+    "do_eval": false,
+    "disable_tqdm": true,
+    "load_best_model_at_end": true,
+    "tensor_parallel_degree": 2,
+    "sharding": "stage2",
+    "use_flash_attention": true,
+    "recompute": false,
+    "recompute_granularity": "full",
+    "benchmark": true,
+    "unified_checkpoint": true,
+    "autotuner_benchmark":false,
+    "greedy_zero_padding": true,
+    "flash_mask":true,
+    "lazy": true
+  }

llm/config/llama/flashmask/sft.json

@@ -0,0 +1,38 @@
+{
+  "model_name_or_path": "meta-llama/Meta-Llama-3.1-8B-Instruct",
+  "dataset_name_or_path": "./data",
+  "output_dir": "./checkpoints/llama_sft_flashmask",
+  "per_device_train_batch_size": 1,
+  "gradient_accumulation_steps": 4,
+  "per_device_eval_batch_size": 1,
+  "eval_accumulation_steps":16,
+  "max_steps": 12000,
+  "learning_rate": 2e-05,
+  "warmup_ratio": 0.03,
+  "logging_steps": 1,
+  "evaluation_strategy": "no",
+  "save_strategy": "no",
+  "src_length": 8000,
+  "max_length": 8192,
+  "bf16": true,
+  "fp16_opt_level": "O2",
+  "do_train": true,
+  "do_eval": false,
+  "disable_tqdm": true,
+  "load_best_model_at_end": true,
+  "eval_with_do_generation": false,
+  "metric_for_best_model": "accuracy",
+  "recompute": false,
+  "save_total_limit": 1,
+  "tensor_parallel_degree": 2,
+  "pipeline_parallel_degree": 1,
+  "pipeline_parallel_config": "disable_p2p_cache_shape",
+  "sharding": "stage2",
+  "zero_padding": true,
+  "unified_checkpoint": true,
+  "tensor_parallel_output": true,
+  "use_flash_attention": true,
+  "greedy_zero_padding": true,
+  "lazy": true,
+  "flash_mask":true
+}