自然语言处理（NLP）基础知识分享系列（10）.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5ad35e0e",
   "metadata": {},
   "source": [
    "## `训练深度学习网络的代码`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67d432bb",
   "metadata": {},
   "source": [
    "首先，我们把需要导入的包集中放在最前面。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "570c61ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "import pickle \n",
    "import torch \n",
    "import torch.nn as nn \n",
    "\n",
    "from torchtext.vocab import build_vocab_from_iterator \n",
    "\n",
    "from torchtext.data.functional import simple_space_split \n",
    "from torchtext.data.functional import numericalize_tokens_from_iterator \n",
    "\n",
    "from torchtext.functional import truncate \n",
    "from torchtext.functional import to_tensor \n",
    "\n",
    "from torch.utils.data import Dataset \n",
    "from torch.utils.data import DataLoader \n",
    "from torch.utils.data import random_split "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc293482",
   "metadata": {},
   "source": [
    "### 将目标变量进行One-hot Encoding\n",
    "\n",
    "为了便于后续处理，这里需要修改一下用来处理训练数据中的label字段的自定义函数。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f5a67323",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_to_index = { 'unrelated' : 0 , 'agreed' : 1 , 'disagreed' : 2 } \n",
    "label_pipeline = lambda x : np.eye(3)[label_to_index [ x ]] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "99caaaf1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 1.0, 0.0]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_pipeline('agreed').tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5109e92",
   "metadata": {},
   "source": [
    "你可以看到现在每个label 都从1 个数字变成一个3 维的向量（Vector）。 \n",
    "每1 维度则对应到1 个分类： \n",
    "•\t[1, 0, 0]代表label 为unrelated \n",
    "•\t[0, 1, 0]代表label 为agreed \n",
    "•\t[0, 0, 1]代表label 为disagreed \n",
    " \n",
    "用这样的方式表达label 的好处是我们可以把分类结果想成概率分布。比方说[0.7, 0.2, 0.1],此预测结果代表模型认为这两个新闻标题的关系有70 % 的概率为unrelated、20 % 的概率是 agreed 而10 % 为disagreed。 \n",
    "这样我们会比较好计算预测结果跟正确解答之间差距，模型就会自动修正学习方向，想尽办法拉近这个差距。 \n",
    "\n",
    "定义一些作为模型超参数的全局变量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "03250960",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = 20 \n",
    "TEST_SPLIT = 0.1 \n",
    "\n",
    "BATCH_SIZE = 32 \n",
    "EMBEDDING_DIM = 64 \n",
    "\n",
    "LSTM_UNITS = 100 \n",
    "DROP_OUT = 0.2 \n",
    "\n",
    "LEARNING_RATE = 0.001 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4647518",
   "metadata": {},
   "source": [
    "### 输入数据预处理\n",
    "\n",
    "为了便于后续处理，**myDataset** 子类做了以下修改：\n",
    "\n",
    "\\# 对于词典外的生词用0作为index <br>\n",
    "\\# 对lable的One-hot处理需要做一下修改 <br> \n",
    "\\# 增加了一个自定义方法get_vocab_size用来返回词典规模 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "17b91bae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自定义一个DataSet 对输入数据进行预处理 \n",
    "class myDataset(Dataset): \n",
    "    def __init__(self, picked_file , max_len=20, transform=None): \n",
    "        super().__init__() \n",
    "        pkl_file_rb = open(picked_file, 'rb') \n",
    "        train =pickle.load(pkl_file_rb) \n",
    "\n",
    "        corpus = pd.concat([train . title1_tokenized, train . title2_tokenized]) \n",
    "        corpus = [c for c in corpus] \n",
    "\n",
    "        vocab = build_vocab_from_iterator(simple_space_split(corpus),\\\n",
    "                                          min_freq=2, specials=[\"<unk>\"]) \n",
    "        vocab.set_default_index(0) # 对于词典外的生词用0作为index \n",
    "        self.vocab_size = vocab.__len__() # 词典大小规模  \n",
    "        \n",
    "        y_train = train.label.apply(label_pipeline) \n",
    "        \n",
    "        # 数字化处理成对的新闻标题A和B \n",
    "        tensor_x = {} \n",
    "        for i in range(2): \n",
    "            x = train.title1_tokenized if i==0 else train.title2_tokenized \n",
    "            tmp_x = [] \n",
    "            ids_iter_x = numericalize_tokens_from_iterator\\\n",
    "            (vocab,simple_space_split([c for c in x])) \n",
    "            for ids in ids_iter_x: \n",
    "                tmp_x.append( truncate([num for num in ids],MAX_LEN )) \n",
    "            tensor_x[i] = to_tensor(tmp_x, padding_value=0) \n",
    "        \n",
    "        # 新闻标题A和B的数据拼接  \n",
    "        self.x = torch.stack([tensor_x[0], tensor_x[1]], 1) \n",
    "        \n",
    "        # 对lable的One-hot处理需要做一下修改 \n",
    "        y_train_list = [y.tolist() for y in y_train.values]\n",
    "        self.y = torch.from_numpy(np.asarray(y_train_list)) \n",
    "        \n",
    "        self.transform = transform \n",
    "\n",
    "    def __len__(self): \n",
    "        return len(self.y)  # 数据集长度 \n",
    "    \n",
    "    def __getitem__(self, index): \n",
    "        x = self.x[index]  # tensor类型 \n",
    "        y = self.y[index] \n",
    "        if self.transform is not None: \n",
    "            x = self.transform(x)  # 对输入进行某些变换 \n",
    "        \n",
    "        return x, y \n",
    "    \n",
    "    def get_vocab_size(self): \n",
    "        return self.vocab_size # 词典规模 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a04c556c",
   "metadata": {},
   "source": [
    "下面用Pytorch 函数**random_split** 分割训练集和验证集。<br> \n",
    "如果你还要问这个save_file文件哪里来的？ 请参看一下本系列的第3期。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4cf46e30",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\hp\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\torch\\_jit_internal.py:1138: UserWarning: The inner type of a container is lost when calling torch.jit.isinstance in eager mode. For example, List[int] would become list and therefore falsely return True for List[float] or List[str].\n",
      "  warnings.warn(\"The inner type of a container is lost when \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "288497 32055\n"
     ]
    }
   ],
   "source": [
    "full_dataset = myDataset(r'./save_file',max_len=MAX_LEN) \n",
    "dataset_size = len(full_dataset) \n",
    "test_size = int(TEST_SPLIT * dataset_size)\n",
    "train_size = dataset_size - test_size\n",
    "print(train_size,test_size)\n",
    "\n",
    "train_dataset,test_dataset = random_split(full_dataset,[train_size, test_size]) \n",
    "vocab_size = full_dataset.get_vocab_size() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8711d39c",
   "metadata": {},
   "source": [
    "### 定义我们的深度学习网络\n",
    "\n",
    "我们通过子类化**nn.Module** 定义我们的神经网络**Net**，并在**__init__** 方法中初始化神经网络层，它主要包括以下结构： \n",
    "\n",
    "一个用来生成词向量（word Vector）的embedding层； \n",
    "一个默认2层的共享LSTM，这里为了灵活性增加了一个bidirectional参数从，用来区分是单向还是双向。所谓双向LSTM（Bi-LSTM），简单理解就是一个文档除了从左向右处理一遍，再倒过来从右向左处理一遍，这样会形成不同的记忆； \n",
    "一个组合分类器，串联了Dropout+Linear+ReLU+Linear层。 \n",
    "\n",
    "在方法**forward** 中实现对输入数据的操作，代码详细注释了tensor维数的变换过程。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "05e0920f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Net(nn.Module): \n",
    "    def __init__(self,num_embeddings=vocab_size, embedding_dim=EMBEDDING_DIM, padding_idx=0, max_norm=True,\\\n",
    "                 hidden_size=LSTM_UNITS,num_layers=2, bidirectional=False, batch_first=True,drop_out=DROP_OUT): \n",
    "        super(Net, self).__init__() \n",
    "        self.embedding = nn.Embedding(num_embeddings=num_embeddings, embedding_dim=embedding_dim,\\\n",
    "                                      padding_idx=padding_idx, max_norm=max_norm) \n",
    "        self.shared_lstm = nn.LSTM(input_size=embedding_dim, hidden_size=hidden_size,\\\n",
    "                                   num_layers=num_layers, bidirectional=bidirectional, batch_first=batch_first) \n",
    "        num_direction = 2 if bidirectional else 1   # 单向/双向LSTM可选 \n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Dropout(drop_out),\n",
    "            nn.Linear(2 * num_layers * hidden_size * num_direction, 84),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(84, 3)\n",
    "        )\n",
    "        \n",
    "    def forward(self, x): \n",
    "        #top_ 和 bm_ 分别对应来自新闻标题A和B的数据 \n",
    "        top_embedded = self.embedding(x)[:,0,:,:] \n",
    "        bm_embedded = self.embedding(x)[:,1,:,:] \n",
    "        #print('top_embedded.size():',top_embedded.size()) \n",
    "        #[BATCH_SIZE, MAX_LEN, EMBEDDING_DIM] \n",
    "        \n",
    "        output, (hn, cn) = self.shared_lstm(top_embedded) \n",
    "        #top_output = output \n",
    "        #print('hn.size(): ',hn.size()) \n",
    "        #[num_layers * num_direction, BATCH_SIZE, hidden_size] \n",
    "\n",
    "        hn = torch.transpose(hn, 0, 1).contiguous() #转换第0维和第1维，将BATCH_SIZE放到最前 \n",
    "        #print('hn.size(): ',hn.size()) \n",
    "        #[BATCH_SIZE, num_layers * num_direction, hidden_size] \n",
    "        \n",
    "        top_output = hn.view(hn.shape[0], -1) #简化处理，使用LSTM隐层输出 \n",
    "        #print('top_output.size(): ',top_output.size()) \n",
    "        #[BATCH_SIZE, num_layers * num_direction * hidden_size] \n",
    "\n",
    "        output, (hn, cn) = self.shared_lstm(bm_embedded) \n",
    "        #bm_output = output \n",
    "        \n",
    "        hn = torch.transpose(hn, 0, 1).contiguous() \n",
    "        bm_output = hn.view(hn.shape[0], -1) #简化处理，使用LSTM隐层输出 \n",
    "\n",
    "        merged = torch.cat((top_output,bm_output),dim=1) \n",
    "        #是否还记得上回说的「孪生网络」？我们需要把来自新闻标题A和B的成对数据做个拼接 \n",
    "        #print('merged.size(): ',merged.size()) \n",
    "        #[BATCH_SIZE, 2 * num_layers * num_direction * hidden_size] \n",
    "        \n",
    "        logits = self.classifier(merged)\n",
    "        \n",
    "        return logits "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef77a4e9",
   "metadata": {},
   "source": [
    "下面定义训练函数和验证函数。 \n",
    "在训练循环中，参数优化步骤如下（后面我们会对损失、梯度和优化器再做详细解释）： \n",
    "\n",
    "1、调用optimizer.zero_grad()以重置模型参数的梯度。默认情况下梯度渐变会累加；为了防止重复计算，在每次迭代时明确地将它们归零。 \n",
    "2、调用net(inputs)执行forward，基于当下训练参数计算输出值，通过criterion(outputs, labels)计算损失loss。 \n",
    "3、调用来反向传播预测损失loss.backward()。PyTorch 存储每个参数的损失梯度。 \n",
    "4、一旦有了梯度，调用optimizer.step()通过在反向传递中收集的梯度来调整参数。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "94841f55",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(net, trainloader, optimizer, device, criterion): \n",
    "    for epoch in range(2):  # 在数据集上循环多次  \n",
    "\n",
    "        running_loss = 0.0 \n",
    "        for i, data in enumerate(trainloader, 0): \n",
    "            # 获取输入数据  \n",
    "            inputs, labels = data \n",
    "            # inputs: [BATCH_SIZE, 2, MAX_LEN] labels: [BATCH_SIZE, 3] \n",
    "\n",
    "            # 将参数梯度归零  \n",
    "            optimizer.zero_grad() \n",
    "\n",
    "            # forward + backward + optimize \n",
    "            outputs = net(inputs) \n",
    "            loss = criterion(outputs, labels) \n",
    "            loss.backward() \n",
    "            optimizer.step() \n",
    "\n",
    "            # 打印统计数据  \n",
    "            running_loss += loss.item() \n",
    "            if i % 1000 == 999:    # 每1000个小批量打印一次 \n",
    "                print('[%d, %5d] loss: %.3f' %(epoch + 1, i + 1, running_loss / 1000)) \n",
    "                running_loss = 0.0 \n",
    "\n",
    "    print('Finished Training') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "68db3f63",
   "metadata": {},
   "outputs": [],
   "source": [
    "def valid(net, testloader): \n",
    "    correct = 0 \n",
    "    total = 0 \n",
    "    with torch.no_grad(): \n",
    "        for data in testloader: \n",
    "            inputs, labels = data \n",
    "            outputs = net(inputs) \n",
    "            #print(outputs.data) \n",
    "            #print(labels.argmax(1)) \n",
    "            #print(labels.size(0)) \n",
    "            _, predicted = torch.max(outputs.data, 1) \n",
    "            # 按维度dim 返回最大值以及最大值的索引 \n",
    "            #print(predicted) \n",
    "            total += labels.size(0) \n",
    "            correct += (predicted == labels.argmax(1)).sum().item() \n",
    "\n",
    "    print('Accuracy of the network on the valid dataset: %.1f %%' % (100 * correct / total)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cf11f131",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cpu device\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\") \n",
    "print(f\"Using {device} device\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "244c8d74",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model structure: Net(\n",
      "  (embedding): Embedding(61841, 64, padding_idx=0, max_norm=True)\n",
      "  (shared_lstm): LSTM(64, 100, num_layers=2, batch_first=True)\n",
      "  (classifier): Sequential(\n",
      "    (0): Dropout(p=0.2, inplace=False)\n",
      "    (1): Linear(in_features=400, out_features=84, bias=True)\n",
      "    (2): ReLU()\n",
      "    (3): Linear(in_features=84, out_features=3, bias=True)\n",
      "  )\n",
      ")\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model = Net(bidirectional=False).to(device) \n",
    "print(f\"Model structure: {model}\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85933264",
   "metadata": {},
   "source": [
    "### 关于损失、梯度和优化器\n",
    "\n",
    "**损失函数** \n",
    "在上面One-hot编码时，我们提到过需要计算预测结果跟正确解答之间差距，而损失函数正是用来衡量得到的结果与目标值的相异程度。 \n",
    "我们在训练时要最小化的损失函数。为了计算损失，我们使用给定数据样本的输入进行预测，并将其与真实数据标签值进行比较。 \n",
    "\n",
    "我们的任务使用的是交叉熵损失**nn.CrossEntropyLoss**，它结合**nn.LogSoftmax** 和**nn.NLLLoss**，将对 logits 进行归一化并计算预测误差。\n",
    "\n",
    "**梯度** \n",
    "为了优化神经网络中参数的权重，我们需要计算我们的损失函数对参数的导数，也就是所谓的梯度。 \n",
    "当参数w 有不同值时，损失函数的值也有所不同。训练模型时会持续修正参数w 以期最小化损失函数，沿梯度下降的方向修正w 是一个常用方法。 \n",
    "为了计算梯度，我们调用**loss.backward()**。 \n",
    "\n",
    "**优化器** \n",
    "优化是在每个训练步骤中调整模型参数以减少模型误差的过程。优化算法定义了如何执行这个过程，所有优化逻辑都封装在**optimizer**对象中。在这里，我们使用**Adam** 优化器，需要通过注册模型需要训练的参数并传入学习率超参数来初始化优化器。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0096144d",
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss() \n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b32ace66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,  1000] loss: 0.629\n",
      "[1,  2000] loss: 0.487\n",
      "[1,  3000] loss: 0.455\n",
      "[1,  4000] loss: 0.448\n",
      "[1,  5000] loss: 0.424\n",
      "[1,  6000] loss: 0.421\n",
      "[1,  7000] loss: 0.417\n",
      "[1,  8000] loss: 0.406\n",
      "[1,  9000] loss: 0.408\n",
      "[2,  1000] loss: 0.376\n",
      "[2,  2000] loss: 0.367\n",
      "[2,  3000] loss: 0.362\n",
      "[2,  4000] loss: 0.367\n",
      "[2,  5000] loss: 0.355\n",
      "[2,  6000] loss: 0.355\n",
      "[2,  7000] loss: 0.351\n",
      "[2,  8000] loss: 0.345\n",
      "[2,  9000] loss: 0.347\n",
      "Finished Training\n"
     ]
    }
   ],
   "source": [
    "train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=False) \n",
    "train(net=model, trainloader=train_loader, optimizer=optimizer, device=device, criterion=criterion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "431b788a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the network on the valid dataset: 83.1 %\n"
     ]
    }
   ],
   "source": [
    "test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False) \n",
    "valid(net=model, testloader=test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95fca2a0",
   "metadata": {},
   "source": [
    "结果是<font color=\"#dd0000\">83.1%</font>，比之前最好的TF-IDF未压缩的<font color=\"#dd0000\">77.4%</font>有所提升！ \n",
    "这个是单向LSTM的结果，如果改成双向的Bi-LSTM还会稍好一点，可以修改模型参数bidirectional=**True** 自行尝试。 \n",
    "\n",
    "---\n",
    "好了，就到这儿吧。 \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}