Reproducing your example leads to a model that replies in English

[DavidFarago]

I LoRA-fine-tuned https://huggingface.co/decapoda-research/llama-7b-hf with your dataset vigogne_data_cleaned.json, but my fine-tuned model replies in English. What did I do wrong?

---

Details:

I conducted the following setup (on a paperspace core default Ampere A6000 GPU with 48GiB GPU memory upgraded to 250GB block storage):

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp/
make
cd ..

wget https://repo.anaconda.com/archive/Anaconda3-2022.05-Linux-x86_64.sh
bash Anaconda3-2022.05-Linux-x86_64.sh
bash
conda create --name vigogne310 python=3.10
conda activate vigogne310

curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git clone https://huggingface.co/decapoda-research/llama-7b-hf

git clone https://github.com/bofenghuang/vigogne.git
cd vigogne
pip install -r requirements.txt
cd data
python ../scripts/export_state_dict_checkpoint.py --base_model_name_or_path ~/llama-7b-hf --lora_model_name_or_path "bofenghuang/vigogne-lora-7b" --output_dir ./models/7B --base_model_size "7B"
mv models ..
cd ..
wget -P ./models https://huggingface.co/bofenghuang/vigogne-lora-7b/resolve/main/tokenizer.model
cd ../llama.cpp
python convert-pth-to-ggml.py ../vigogne/models/7B/ 1
./quantize ../vigogne/models/7B/ggml-model-f16.bin ../vigogne/models/7B/ggml-model-q4_0.bin 2
./main -m ../vigogne/models/7B/ggml-model-q4_0.bin --color -ins -c 2048 --temp 0.1 -n 256

Which performs correct inference on English, but not on French.

Then I continue for fine-tuning:

pip install wandb
wandb login
python finetune.py --model_name_or_path ../llama-7b-hf/ --train_file "data/vigogne_data_cleaned.json" --output_dir "outputs/llama-7b-ft-vigogne-lora" --run_name "llama-7b-ft-vigogne-lora" --overwrite_output_dir --model_max_length_percentile 95 --preprocessing_num_workers 2 --dataloader_num_workers 1 --lora_r 8 --lora_alpha 16 --lora_dropout 0.05 --target_modules "q_proj" "v_proj" --per_device_train_batch_size 16 --per_device_eval_batch_size 8 --gradient_accumulation_steps 8 --num_train_epochs 3 --learning_rate 3e-4 --warmup_steps 100 --logging_steps 25 --save_strategy "steps" --save_steps 200 --save_total_limit 3 --report_to "wandb"

The fine-tuning seems to be successful, so I prepare the new model for inference:

cd vigogne
python scripts/export_state_dict_checkpoint.py --base_model_name_or_path ~/llama-7b-hf --lora_model_name_or_path "outputs/llama-7b-ft-vigogne-lora/" --output_dir ./models/7B_french --base_model_size "7B"
cd ../llama.cpp
python convert-pth-to-ggml.py ../vigogne/models/7B_french/ 1
./quantize ../vigogne/models/7B_french/ggml-model-f16.bin ../vigogne/models/7B_french/ggml-model-q4_0.bin 2
./main -m ../vigogne/models/7B_french/ggml-model-q4_0.bin --color -ins -c 2048 --temp 0.1 -n 256

When I enter

Répondez à la question suivante : Les pratiques artistiques transforment-elles le monde ?

I get the following response in English:

Artistic practices transform the world.

The artistic practice is a way to express oneself, to create and to share with others. It is also an opportunity to reflect on our society and its evolutions. Artists are able to question themselves about their environment and to propose new ways of thinking. They can also be involved in social issues such as the environment or the economy.

### Exemple:

* 

"The artistic practice is a way to express oneself, to create and to share with others."
* 

"It is also an opportunity to reflect on our society and its evolutions."
* 

"Artists are able to question themselves about their environment and to propose new ways of thinking. They can also be involved in social issues such as the environment or the economy."

[bofenghuang]

Hi @DavidFarago,

Thank you for providing such detailed steps, they will be useful to others!

Regarding your issue with the fine-tuned model replying in English, it's possible that tuning the inference parameters could help.

However, it's important to note that this model is far from perfect and there is a big room for improvement. One possible solution could be to re-finetune the model with different hyperparameters, particularly those related to LoRA. Another option would be to collect more French instruction-following data. Additionally, continuing the pre-training of the LLaMA model to better suit the French language may also be beneficial.

I'm currently working on some new features that could probably resolve this issue. Stay tuned!