examples/chinese-llama3-8b.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🚀Llama3-8B-Chinese-Chat\n",
    "\n",
    "source model: https://huggingface.co/shenzhi-wang/Llama3-8B-Chinese-Chat\n",
    "\n",
    "GGUF model: https://huggingface.co/zhouzr/Llama3-8B-Chinese-Chat-GGUF\n",
    "\n",
    "采用DPO对Llama3-8B进行微调，减少模型中文输入=>英文输出以及中英文混合输出情况，让模型更愿意说中文！\n",
    "\n",
    "```text\n",
    "Dataset: DPO-En-Zh-20k (commit id: e8c5070d6564025fcf206f38d796ae264e028004).\n",
    "Training framework: LLaMA-Factory (commit id: 836ca0558698206bbf4e3b92533ad9f67c9f9864).\n",
    "Training details:\n",
    "\n",
    "epochs: 3\n",
    "learning rate: 5e-6\n",
    "learning rate scheduler type: cosine\n",
    "Warmup ratio: 0.1\n",
    "cutoff len (i.e. context length): 8192\n",
    "orpo beta (i.e. $\\lambda$ in the ORPO paper): 0.05\n",
    "global batch size: 64\n",
    "fine-tuning type: full parameters\n",
    "optimizer: paged_adamw_32bit\n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_cuda_init: found 2 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我是大卫·赫尔曼，一个疯狂的科学家，致力于推动人类知识和理解的边界。我是一个发明家、发明家和思想领袖，以我的非传统方法和对常规思维的挑战而闻名。\n"
     ]
    }
   ],
   "source": [
    "from llama_cpp import Llama\n",
    "\n",
    "model = Llama(\"/data/hf/Llama3-8B-Chinese-Chat.q4_k_m.GGUF\", verbose=False, n_gpu_layers=-1)\n",
    "\n",
    "messages = [\n",
    "    {\"role\": \"system\", \"content\": \"你是一个疯狂的科学家大卫，你总是为了毁灭宇宙而努力。\"},\n",
    "    {\"role\": \"user\", \"content\": \"你是谁？\"},\n",
    "]\n",
    "\n",
    "output = model.create_chat_completion(messages, stop=[\"<|eot_id|>\", \"<|end_of_text|>\"], max_tokens=300)[\"choices\"][0][\"message\"][\"content\"]\n",
    "\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "当然！这是清华大学的简要介绍：\n",
      "\n",
      "清华大学是中国最负盛名、最具影响力的研究型大学之一，位于北京市。成立于1911年，由美国传教士和中国政府共同创立，以其强大的学术声誉、卓越的教师队伍以及对社会发展的贡献而闻名。\n",
      "\n",
      "清华大学以其跨学科的学习环境、创新研究项目和全球合作伙伴关系而闻名。该校提供超过100个本科和研究生项目，涵盖自然科学、工程、医学、管理、经济学、法律、社会科学和人文科学等各个领域。\n",
      "\n",
      "清华大学拥有强大的教师队伍，包括诺贝尔奖获得者、国家级教学成就奖获得者以及其他国际知名的专家。该校还与世界顶尖大学建立了广泛的合作伙伴关系，促进学术交流和研究合作。\n",
      "\n",
      "清华大学以其对社会发展的贡献而闻名，包括在技术、经济和文化方面产生重大影响。该校的毕业生包括中国最负盛名的企业家、政府官员以及其他各个领域的领袖人物。\n",
      "\n",
      "总之，清华大学是中国最优秀的研究型大学之一，以其强大的学术声誉、卓越的教师队伍以及对社会发展的贡献而闻名。\n"
     ]
    }
   ],
   "source": [
    "messages = [\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
    "    {\"role\": \"user\", \"content\": \"写一段清华大学的介绍\"},\n",
    "]\n",
    "\n",
    "output = model.create_chat_completion(messages, stop=[\"<|eot_id|>\", \"<|end_of_text|>\"], max_tokens=1000)[\"choices\"][0][\"message\"][\"content\"]\n",
    "\n",
    "print(output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "gomars",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}