自然语言处理（NLP）基础知识分享系列（7）.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5ad35e0e",
   "metadata": {},
   "source": [
    "## `从『词袋』到『词序列』`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e59550",
   "metadata": {},
   "source": [
    "我们之前对于文档的数字化表示，采用的是向量空间模型（例如TF-IDF），又被形象地称为『词袋』模型（Bag-of-words model）。就像是把文档里的词汇，放入到以词典作为标签的袋子里。 \n",
    "我们可以看到，基于词袋模型的文档表示方法，虽然考虑了词的重要程度，但它只是根据词的统计特性表示一个文档，而没有考虑到词在文中的次序。\n",
    "比方说有这样两句话： \n",
    "•“熊二/的/哥哥/是/熊大。” \n",
    "•“熊二/是/熊大/的/哥哥。” \n",
    "这两句话的TF-IDF数据表示形式是一样的，但它们的语义是截然相反的。 \n",
    "\n",
    "究其原因，是因为词袋模型丢失了文档中的上下文信息。 \n",
    "\n",
    "这样就有了一个新的思路：将文档表示成词编码的一个序列，这样词在文档上下文关系信息就能够保留下来。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09673bdb",
   "metadata": {},
   "source": [
    "### 建立词典并将文本转成数字序列\n",
    "\n",
    "接下来我们尝试将文档转换成一个数字序列，方便电脑处理。 \n",
    "首先我们要根据语料库中出现的词建立一个词典，建立数字索引（Index），分别对应到特定的词汇。实际上，这个建立词典的步骤在之前的TF-IDF模型训练函数TfidfVectorizer中已经隐含地做过。但现在我们不再使用TF-IDF，因此需要把这一过程单独拿出来显性实现一下。 \n",
    "\n",
    "在李孟博士的原博客里，后续演示使用的是TensorFlow.Keras相关包，我们改为全部使用Pytorch演示相关功能的实现。 \n",
    "\n",
    "首先，把Jieba分词结果和语料库调入内存。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6eca9a36",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import pickle \n",
    " \n",
    "pkl_file_rb = open(r'./save_file', 'rb') \n",
    "train =pickle.load(pkl_file_rb) \n",
    " \n",
    "corpus = pd.concat([train . title1_tokenized, train . title2_tokenized]) \n",
    "corpus = [c for c in corpus] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "25f482e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(train[0:1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "57aa1718",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['男人', '在', '机舱', '口', '跪下', '原来', '一切', '都', '只', '因为', '爱']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[16].strip().split() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee286b37",
   "metadata": {},
   "source": [
    "定义一个迭代器对语料库中的分词数据进行预处理： "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "55f00216",
   "metadata": {},
   "outputs": [],
   "source": [
    "def yield_tokens(corpus): \n",
    "    for i in range(len(corpus)): \n",
    "        yield corpus[i].strip().split() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77315e80",
   "metadata": {},
   "source": [
    "使用 **torchtext** 的 **build_vocab_from_iterator** 函数建立词典。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ab3cec68",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.vocab import build_vocab_from_iterator \n",
    " \n",
    "vocab = build_vocab_from_iterator(yield_tokens(corpus), min_freq=2, specials=[\"<unk>\"]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a60f1b8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab.set_default_index(vocab['<unk>']) #在词典里设置一个默认词，用来对于OOV的生词做默认编码 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "523a51eb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 326, 0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab['熊二'], vocab['爱'],vocab['<unk>'] #'熊二'在语料库里没有出现过 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a49c41d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "61841"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.__len__()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "332bae13",
   "metadata": {},
   "source": [
    "经过这一步骤，我们就可以将一个新闻标题文档表示为一个词典序编码的数字序列，如下所示： "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "20ea40db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[306, 16, 5905, 1724, 5474, 181, 1843, 14, 99, 487, 326]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab(['男人', '在', '机舱', '口', '跪下', '原来', '一切', '都', '只', '因为', '爱']) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3692b335",
   "metadata": {},
   "source": [
    "### 用PyTorch的DataLoader 预处理数据\n",
    "\n",
    "在后续建立深度学习模型之前，我们将采用PyTorch建议使用的 **torch.utils.data.DataLoader** 方法对数据进行预处理。 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc293482",
   "metadata": {},
   "source": [
    "首先我们定义2个匿名函数，用来对训练数据中的label和titleX_tokenized字段做个简单的处理。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f5a67323",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_to_index = { 'unrelated' : 0 , 'agreed' : 1 , 'disagreed' : 2 } \n",
    "label_pipeline = lambda x : label_to_index [ x ] \n",
    "text_pipeline = lambda x: vocab(x.strip().split()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "03250960",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "import numpy as np \n",
    "from torch.utils.data import Dataset \n",
    "from torch.utils.data import DataLoader,TensorDataset \n",
    "\n",
    "MAX_LEN = 30 #定义一个最大长度，用于对过长的标题进行截断 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "373136a7",
   "metadata": {},
   "source": [
    "下面我们定义一个**myDataSet** 类，用于对之前的**DataFrame** train 进行预处理。 \n",
    "这里稍微有点复杂，对于程序比较生疏的我来说有些许困难，花了半天时间学习和试验才调通。 \n",
    "\n",
    "主要的逻辑是：\n",
    "从train中提取title1_tokenized和title2_tokenized字段，转为词典编码序列，并根据最大长度MAX_LEN进行截断，作为将来模型输入的tensor； \n",
    "提取label字段转换为0、1、2作为预测目标，作为将来模型输出tensor。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "17b91bae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自定义一个myDataSet类 对输入数据进行预处理 \n",
    "class myDataset(Dataset): \n",
    "    def __init__(self, train_data, max_len=20, transform=None): \n",
    "        super().__init__() \n",
    "        x1_train = train_data.title1_tokenized.apply(text_pipeline) \n",
    "        x2_train = train_data.title2_tokenized.apply(text_pipeline) \n",
    "        y_train = train.label.apply(label_pipeline) \n",
    "        \n",
    "        # 以下根据max_len参数，对长短不一的数字化标题序列进行截断和补齐 \n",
    "        tmp = [] \n",
    "        for x in (x1_train.values,x2_train.values): \n",
    "            tmp_x = [] \n",
    "            for i in range(len(x)): \n",
    "                temp_x1 = [] \n",
    "                for j in range(max_len): \n",
    "                    temp_x1.append(x[i][j] if j < len(x[i]) else 0) \n",
    "                tmp_x.append(temp_x1) \n",
    "            tmp.append(tmp_x) \n",
    "            \n",
    "        self.x = torch.from_numpy(np.transpose(np.asarray(tmp),[1,0,2])) \n",
    "        self.y = torch.from_numpy(np.asarray(y_train.values)) \n",
    "        self.transform = transform \n",
    "\n",
    "    def __len__(self): \n",
    "        return len(self.y)  # 数据集长度 \n",
    "    \n",
    "    def __getitem__(self, index): \n",
    "        x = self.x[index]  # tensor类型 \n",
    "        y = self.y[index] \n",
    "        if self.transform is not None: \n",
    "            x = self.transform(x)  # 对输入进行某些变换 \n",
    "        \n",
    "        return x, y \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f116ec1",
   "metadata": {},
   "source": [
    "实例化一个**MyDataset** 类的对象train_data，作为参数传入**DataLoader** 进行数据预处理。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "77ee8b6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\") \n",
    "train_data = myDataset(train,max_len=MAX_LEN) \n",
    "dataloader = DataLoader(train_data, batch_size=8, shuffle=False) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "076f2dc8",
   "metadata": {},
   "source": [
    "看一下处理后的数据样例和维度形式是否如我们所愿： "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5e0cc688",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[  218,  1269,    32,  1178,  5971,    24,   488,  2875,   116,  5568,\n",
      "              4,  1847,     2,    13,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0],\n",
      "         [  150,     8, 12895,  6168,  6345,   529,    44,    64,   740,    12,\n",
      "            488,   286, 13213,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0]],\n",
      "\n",
      "        [[    4,    10,    47,   677,  2561,     4,   165,    34,    17,    47,\n",
      "           5153,    62,    15,   677,  4509,  3208,    23,   284,  1185,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0],\n",
      "         [  677,  3208,  1141,   284,  1185,   677, 23975,     8,   784,  4515,\n",
      "             16, 12858,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0]],\n",
      "\n",
      "        [[    4,    10,    47,   677,  2561,     4,   165,    34,    17,    47,\n",
      "           5153,    62,    15,   677,  4509,  3208,    23,   284,  1185,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0],\n",
      "         [ 3208,  1141,   284,  1185,   677,   390,    22,   953,  9816,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0]],\n",
      "\n",
      "        [[    4,    10,    47,   677,  2561,     4,   165,    34,    17,    47,\n",
      "           5153,    62,    15,   677,  4509,  3208,    23,   284,  1185,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0],\n",
      "         [ 3299,   677,  3208,  1141,   284,  1185,   677, 23975,     8,    22,\n",
      "            953, 45212,   120,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0]],\n",
      "\n",
      "        [[   31,   319,  3373,  3057,     1,    94,    97,  3373,  3057,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0],\n",
      "         [    7,     2,   208,    15, 14291,   174,    50,  1627,   319,   122,\n",
      "           3373,  3057,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0]],\n",
      "\n",
      "        [[    4,    10,    47,   677,  2561,     4,   165,    34,    17,    47,\n",
      "           5153,    62,    15,   677,  4509,  3208,    23,   284,  1185,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0],\n",
      "         [  677,  3208,  1141,   284,  1185, 23975,     8,   381,   284,   242,\n",
      "           4515,  1670, 12858,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0]],\n",
      "\n",
      "        [[    7,  2220,     1,  2071,     7, 18165,  2188,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0],\n",
      "         [ 2220,    78,    20,   101,   125,     7,    46,  1552,   641,     7,\n",
      "           3055,  4705,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0]],\n",
      "\n",
      "        [[    4,    10,    47,   677,  2561,     4,   165,    34,    17,    47,\n",
      "           5153,    62,    15,   677,  4509,  3208,    23,   284,  1185,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0],\n",
      "         [  677,  3208,  1141,   284,  1185,     8,   381,   284,   242,  4515,\n",
      "           1670, 12858,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0]]],\n",
      "       dtype=torch.int32) tensor([0, 0, 0, 0, 1, 0, 0, 0])\n",
      "********************************************************************************\n",
      "torch.Size([8, 2, 30]) torch.Size([8])\n"
     ]
    }
   ],
   "source": [
    "for x,y in dataloader: \n",
    "    print(x,y) \n",
    "    print(\"*\"*80) \n",
    "    print(x.shape,y.shape) \n",
    "    break "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95fca2a0",
   "metadata": {},
   "source": [
    "---\n",
    "好了，就到这儿吧。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}