EXPLORATION/EXPLORATION 21. 트랜스포머로 만드는 대화형 챗봇.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EXPLORATION 21. 트랜스포머로 만드는 대화형 챗봇"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "chatbot_data = pd.read_csv('./data/ChatbotData .csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12시 땡!</td>\n",
       "      <td>하루가 또 가네요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1지망 학교 떨어졌어</td>\n",
       "      <td>위로해 드립니다.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3박4일 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3박4일 정도 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PPL 심하네</td>\n",
       "      <td>눈살이 찌푸려지죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11818</th>\n",
       "      <td>훔쳐보는 것도 눈치 보임.</td>\n",
       "      <td>티가 나니까 눈치가 보이는 거죠!</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11819</th>\n",
       "      <td>훔쳐보는 것도 눈치 보임.</td>\n",
       "      <td>훔쳐보는 거 티나나봐요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11820</th>\n",
       "      <td>흑기사 해주는 짝남.</td>\n",
       "      <td>설렜겠어요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11821</th>\n",
       "      <td>힘든 연애 좋은 연애라는게 무슨 차이일까?</td>\n",
       "      <td>잘 헤어질 수 있는 사이 여부인 거 같아요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11822</th>\n",
       "      <td>힘들어서 결혼할까봐</td>\n",
       "      <td>도피성 결혼은 하지 않길 바라요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11823 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Q                         A  label\n",
       "0                       12시 땡!                하루가 또 가네요.      0\n",
       "1                  1지망 학교 떨어졌어                 위로해 드립니다.      0\n",
       "2                 3박4일 놀러가고 싶다               여행은 언제나 좋죠.      0\n",
       "3              3박4일 정도 놀러가고 싶다               여행은 언제나 좋죠.      0\n",
       "4                      PPL 심하네                눈살이 찌푸려지죠.      0\n",
       "...                        ...                       ...    ...\n",
       "11818           훔쳐보는 것도 눈치 보임.        티가 나니까 눈치가 보이는 거죠!      2\n",
       "11819           훔쳐보는 것도 눈치 보임.             훔쳐보는 거 티나나봐요.      2\n",
       "11820              흑기사 해주는 짝남.                    설렜겠어요.      2\n",
       "11821  힘든 연애 좋은 연애라는게 무슨 차이일까?  잘 헤어질 수 있는 사이 여부인 거 같아요.      2\n",
       "11822               힘들어서 결혼할까봐        도피성 결혼은 하지 않길 바라요.      2\n",
       "\n",
       "[11823 rows x 3 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chatbot_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50000\n"
     ]
    }
   ],
   "source": [
    "MAX_SAMPLES = 50000\n",
    "print(MAX_SAMPLES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_sentence(sentence):\n",
    "    sentence = sentence.lower().strip()\n",
    "    sentence = re.sub(r\"([?.!,])\", r\" \\1 \", sentence)\n",
    "    sentence = re.sub(r'[\" \"]+', \" \", sentence)\n",
    "    sentence = sentence.strip()\n",
    "\n",
    "    return sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_conversations(questions, answers):\n",
    "    inputs, outputs = [], []\n",
    "    \n",
    "    for question, answer in zip(questions, answers):\n",
    "        inputs.append(preprocess_sentence(question))\n",
    "        outputs.append(preprocess_sentence(answer))\n",
    "        \n",
    "    return inputs, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플 수 : 11823\n",
      "전체 샘플 수 : 11823\n"
     ]
    }
   ],
   "source": [
    "questions, answers = load_conversations(chatbot_data['Q'], chatbot_data['A'])\n",
    "print('전체 샘플 수 :', len(questions))\n",
    "print('전체 샘플 수 :', len(answers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전처리 후의 22번째 질문 샘플: 가스비 장난 아님\n",
      "전처리 후의 22번째 답변 샘플: 다음 달에는 더 절약해봐요 .\n"
     ]
    }
   ],
   "source": [
    "print('전처리 후의 22번째 질문 샘플: {}'.format(questions[21]))\n",
    "print('전처리 후의 22번째 답변 샘플: {}'.format(answers[21]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SubwordTextEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
    "    questions + answers, target_vocab_size=2**13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "START_TOKEN, END_TOKEN = [tokenizer.vocab_size], [tokenizer.vocab_size + 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "START_TOKEN의 번호 : [8173]\n",
      "END_TOKEN의 번호 : [8174]\n"
     ]
    }
   ],
   "source": [
    "print('START_TOKEN의 번호 :', [tokenizer.vocab_size])\n",
    "print('END_TOKEN의 번호 :', [tokenizer.vocab_size + 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8175\n"
     ]
    }
   ],
   "source": [
    "VOCAB_SIZE = tokenizer.vocab_size + 2\n",
    "print(VOCAB_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "정수 인코딩 후의 21번째 질문 샘플: [5763, 610, 2492, 4164]\n",
      "정수 인코딩 후의 21번째 답변 샘플: [2356, 7513, 7, 6276, 97, 1]\n"
     ]
    }
   ],
   "source": [
    "print('정수 인코딩 후의 21번째 질문 샘플: {}'.format(tokenizer.encode(questions[21])))\n",
    "print('정수 인코딩 후의 21번째 답변 샘플: {}'.format(tokenizer.encode(answers[21])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "40\n"
     ]
    }
   ],
   "source": [
    "MAX_LENGTH = 40\n",
    "print(MAX_LENGTH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_and_filter(inputs, outputs):\n",
    "    tokenized_inputs, tokenized_outputs = [], []\n",
    "\n",
    "    for (sentence1, sentence2) in zip(inputs, outputs):\n",
    "        sentence1 = START_TOKEN + tokenizer.encode(sentence1) + END_TOKEN\n",
    "        sentence2 = START_TOKEN + tokenizer.encode(sentence2) + END_TOKEN\n",
    "\n",
    "        if len(sentence1) <= MAX_LENGTH and len(sentence2) <= MAX_LENGTH:\n",
    "            tokenized_inputs.append(sentence1)\n",
    "            tokenized_outputs.append(sentence2)\n",
    "\n",
    "    tokenized_inputs = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_inputs, maxlen=MAX_LENGTH, padding='post')\n",
    "    tokenized_outputs = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_outputs, maxlen=MAX_LENGTH, padding='post')\n",
    "\n",
    "    return tokenized_inputs, tokenized_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "단어장의 크기 : 8175\n",
      "필터링 후의 샘플 개수: 11823\n",
      "필터링 후의 샘플 개수: 11823\n"
     ]
    }
   ],
   "source": [
    "questions, answers = tokenize_and_filter(questions, answers)\n",
    "print('단어장의 크기 :', (VOCAB_SIZE))\n",
    "print('필터링 후의 샘플 개수: {}'.format(len(questions)))\n",
    "print('필터링 후의 샘플 개수: {}'.format(len(answers)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PositionalEncoding\n",
    "class PositionalEncoding(tf.keras.layers.Layer):\n",
    "\n",
    "    def __init__(self, position, d_model):\n",
    "        super(PositionalEncoding, self).__init__()\n",
    "        self.pos_encoding = self.positional_encoding(position, d_model)\n",
    "\n",
    "    def get_angles(self, position, i, d_model):\n",
    "        angles = 1 / tf.pow(10000, (2 * (i // 2)) /\n",
    "                            tf.cast(d_model, tf.float32))\n",
    "        return position * angles\n",
    "\n",
    "    def positional_encoding(self, position, d_model):\n",
    "        angle_rads = self.get_angles(\n",
    "            position=tf.range(position, dtype=tf.float32)[:, tf.newaxis],\n",
    "            i=tf.range(d_model, dtype=tf.float32)[tf.newaxis, :],\n",
    "            d_model=d_model)\n",
    "\n",
    "        sines = tf.math.sin(angle_rads[:, 0::2])\n",
    "        cosines = tf.math.cos(angle_rads[:, 1::2])\n",
    "\n",
    "        pos_encoding = tf.concat([sines, cosines], axis=-1)\n",
    "        pos_encoding = pos_encoding[tf.newaxis, ...]\n",
    "        return tf.cast(pos_encoding, tf.float32)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        return inputs + self.pos_encoding[:, :tf.shape(inputs)[1], :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scaled_dot_product_attention\n",
    "def scaled_dot_product_attention(query, key, value, mask):\n",
    "    matmul_qk = tf.matmul(query, key, transpose_b=True)\n",
    "\n",
    "    depth = tf.cast(tf.shape(key)[-1], tf.float32)\n",
    "    logits = matmul_qk / tf.math.sqrt(depth)\n",
    "\n",
    "    if mask is not None:\n",
    "        logits += (mask * -1e9)\n",
    "\n",
    "    attention_weights = tf.nn.softmax(logits, axis=-1)\n",
    "\n",
    "    output = tf.matmul(attention_weights, value)\n",
    "\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(tf.keras.layers.Layer):\n",
    "\n",
    "    def __init__(self, d_model, num_heads, name=\"multi_head_attention\"):\n",
    "        super(MultiHeadAttention, self).__init__(name=name)\n",
    "        self.num_heads = num_heads\n",
    "        self.d_model = d_model\n",
    "\n",
    "        assert d_model % self.num_heads == 0\n",
    "\n",
    "        self.depth = d_model // self.num_heads\n",
    "\n",
    "        self.query_dense = tf.keras.layers.Dense(units=d_model)\n",
    "        self.key_dense = tf.keras.layers.Dense(units=d_model)\n",
    "        self.value_dense = tf.keras.layers.Dense(units=d_model)\n",
    "\n",
    "        self.dense = tf.keras.layers.Dense(units=d_model)\n",
    "\n",
    "    def split_heads(self, inputs, batch_size):\n",
    "        inputs = tf.reshape(\n",
    "            inputs, shape=(batch_size, -1, self.num_heads, self.depth))\n",
    "        return tf.transpose(inputs, perm=[0, 2, 1, 3])\n",
    "\n",
    "    def call(self, inputs):\n",
    "        query, key, value, mask = inputs['query'], inputs['key'], inputs[\n",
    "            'value'], inputs['mask']\n",
    "        batch_size = tf.shape(query)[0]\n",
    "\n",
    "        query = self.query_dense(query)\n",
    "        key = self.key_dense(key)\n",
    "        value = self.value_dense(value)\n",
    "\n",
    "        query = self.split_heads(query, batch_size)\n",
    "        key = self.split_heads(key, batch_size)\n",
    "        value = self.split_heads(value, batch_size)\n",
    "\n",
    "        scaled_attention = scaled_dot_product_attention(\n",
    "            query, key, value, mask)\n",
    "        scaled_attention = tf.transpose(scaled_attention, perm=[0, 2, 1, 3])\n",
    "        concat_attention = tf.reshape(scaled_attention,\n",
    "                                      (batch_size, -1, self.d_model))\n",
    "\n",
    "        outputs = self.dense(concat_attention)\n",
    "\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_padding_mask(x):\n",
    "    mask = tf.cast(tf.math.equal(x, 0), tf.float32)\n",
    "    return mask[:, tf.newaxis, tf.newaxis, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_look_ahead_mask(x):\n",
    "    seq_len = tf.shape(x)[1]\n",
    "    look_ahead_mask = 1 - \\\n",
    "        tf.linalg.band_part(tf.ones((seq_len, seq_len)), -1, 0)\n",
    "    padding_mask = create_padding_mask(x)\n",
    "    return tf.maximum(look_ahead_mask, padding_mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encoder_layer(units, d_model, num_heads, dropout, name=\"encoder_layer\"):\n",
    "    inputs = tf.keras.Input(shape=(None, d_model), name=\"inputs\")\n",
    "\n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name=\"padding_mask\")\n",
    "\n",
    "    attention = MultiHeadAttention(\n",
    "        d_model, num_heads, name=\"attention\")({\n",
    "            'query': inputs,\n",
    "            'key': inputs,\n",
    "            'value': inputs,\n",
    "            'mask': padding_mask\n",
    "        })\n",
    "\n",
    "    attention = tf.keras.layers.Dropout(rate=dropout)(attention)\n",
    "    attention = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(inputs + attention)\n",
    "\n",
    "    outputs = tf.keras.layers.Dense(units=units, activation='relu')(attention)\n",
    "    outputs = tf.keras.layers.Dense(units=d_model)(outputs)\n",
    "\n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(outputs)\n",
    "    outputs = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(attention + outputs)\n",
    "\n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, padding_mask], outputs=outputs, name=name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encoder(vocab_size,\n",
    "            num_layers,\n",
    "            units,\n",
    "            d_model,\n",
    "            num_heads,\n",
    "            dropout,\n",
    "            name=\"encoder\"):\n",
    "    inputs = tf.keras.Input(shape=(None,), name=\"inputs\")\n",
    "\n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name=\"padding_mask\")\n",
    "\n",
    "    embeddings = tf.keras.layers.Embedding(vocab_size, d_model)(inputs)\n",
    "    embeddings *= tf.math.sqrt(tf.cast(d_model, tf.float32))\n",
    "\n",
    "    embeddings = PositionalEncoding(vocab_size, d_model)(embeddings)\n",
    "\n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(embeddings)\n",
    "\n",
    "    for i in range(num_layers):\n",
    "        outputs = encoder_layer(\n",
    "            units=units,\n",
    "            d_model=d_model,\n",
    "            num_heads=num_heads,\n",
    "            dropout=dropout,\n",
    "            name=\"encoder_layer_{}\".format(i),\n",
    "        )([outputs, padding_mask])\n",
    "\n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, padding_mask], outputs=outputs, name=name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder_layer(units, d_model, num_heads, dropout, name=\"decoder_layer\"):\n",
    "    inputs = tf.keras.Input(shape=(None, d_model), name=\"inputs\")\n",
    "    enc_outputs = tf.keras.Input(shape=(None, d_model), name=\"encoder_outputs\")\n",
    "    look_ahead_mask = tf.keras.Input(\n",
    "        shape=(1, None, None), name=\"look_ahead_mask\")\n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name='padding_mask')\n",
    "\n",
    "    attention1 = MultiHeadAttention(\n",
    "        d_model, num_heads, name=\"attention_1\")(inputs={\n",
    "            'query': inputs,\n",
    "            'key': inputs,\n",
    "            'value': inputs,\n",
    "            'mask': look_ahead_mask\n",
    "        })\n",
    "\n",
    "    attention1 = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(attention1 + inputs)\n",
    "\n",
    "    attention2 = MultiHeadAttention(\n",
    "        d_model, num_heads, name=\"attention_2\")(inputs={\n",
    "            'query': attention1,\n",
    "            'key': enc_outputs,\n",
    "            'value': enc_outputs,\n",
    "            'mask': padding_mask\n",
    "        })\n",
    "\n",
    "    attention2 = tf.keras.layers.Dropout(rate=dropout)(attention2)\n",
    "    attention2 = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(attention2 + attention1)\n",
    "\n",
    "    outputs = tf.keras.layers.Dense(units=units, activation='relu')(attention2)\n",
    "    outputs = tf.keras.layers.Dense(units=d_model)(outputs)\n",
    "\n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(outputs)\n",
    "    outputs = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(outputs + attention2)\n",
    "\n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, enc_outputs, look_ahead_mask, padding_mask],\n",
    "        outputs=outputs,\n",
    "        name=name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder(vocab_size,\n",
    "            num_layers,\n",
    "            units,\n",
    "            d_model,\n",
    "            num_heads,\n",
    "            dropout,\n",
    "            name='decoder'):\n",
    "    inputs = tf.keras.Input(shape=(None,), name='inputs')\n",
    "    enc_outputs = tf.keras.Input(shape=(None, d_model), name='encoder_outputs')\n",
    "    look_ahead_mask = tf.keras.Input(\n",
    "        shape=(1, None, None), name='look_ahead_mask')\n",
    "\n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name='padding_mask')\n",
    "\n",
    "    embeddings = tf.keras.layers.Embedding(vocab_size, d_model)(inputs)\n",
    "    embeddings *= tf.math.sqrt(tf.cast(d_model, tf.float32))\n",
    "\n",
    "    embeddings = PositionalEncoding(vocab_size, d_model)(embeddings)\n",
    "\n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(embeddings)\n",
    "\n",
    "    for i in range(num_layers):\n",
    "        outputs = decoder_layer(\n",
    "            units=units,\n",
    "            d_model=d_model,\n",
    "            num_heads=num_heads,\n",
    "            dropout=dropout,\n",
    "            name='decoder_layer_{}'.format(i),\n",
    "        )(inputs=[outputs, enc_outputs, look_ahead_mask, padding_mask])\n",
    "\n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, enc_outputs, look_ahead_mask, padding_mask],\n",
    "        outputs=outputs,\n",
    "        name=name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "BUFFER_SIZE = 20000\n",
    "\n",
    "dataset = tf.data.Dataset.from_tensor_slices((\n",
    "    {\n",
    "        'inputs': questions,\n",
    "        'dec_inputs': answers[:, :-1]\n",
    "    },\n",
    "    {\n",
    "        'outputs': answers[:, 1:]\n",
    "    },\n",
    "))\n",
    "\n",
    "dataset = dataset.cache()\n",
    "dataset = dataset.shuffle(BUFFER_SIZE)\n",
    "dataset = dataset.batch(BATCH_SIZE)\n",
    "dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transformer(vocab_size,\n",
    "                num_layers,\n",
    "                units,\n",
    "                d_model,\n",
    "                num_heads,\n",
    "                dropout,\n",
    "                name=\"transformer\"):\n",
    "    inputs = tf.keras.Input(shape=(None,), name=\"inputs\")\n",
    "    dec_inputs = tf.keras.Input(shape=(None,), name=\"dec_inputs\")\n",
    "\n",
    "    enc_padding_mask = tf.keras.layers.Lambda(\n",
    "        create_padding_mask, output_shape=(1, 1, None),\n",
    "        name='enc_padding_mask')(inputs)\n",
    "\n",
    "    look_ahead_mask = tf.keras.layers.Lambda(\n",
    "        create_look_ahead_mask,\n",
    "        output_shape=(1, None, None),\n",
    "        name='look_ahead_mask')(dec_inputs)\n",
    "\n",
    "    dec_padding_mask = tf.keras.layers.Lambda(\n",
    "        create_padding_mask, output_shape=(1, 1, None),\n",
    "        name='dec_padding_mask')(inputs)\n",
    "\n",
    "    # encoder\n",
    "    enc_outputs = encoder(\n",
    "        vocab_size=vocab_size,\n",
    "        num_layers=num_layers,\n",
    "        units=units,\n",
    "        d_model=d_model,\n",
    "        num_heads=num_heads,\n",
    "        dropout=dropout,\n",
    "    )(inputs=[inputs, enc_padding_mask])\n",
    "\n",
    "    # decoder\n",
    "    dec_outputs = decoder(\n",
    "        vocab_size=vocab_size,\n",
    "        num_layers=num_layers,\n",
    "        units=units,\n",
    "        d_model=d_model,\n",
    "        num_heads=num_heads,\n",
    "        dropout=dropout,\n",
    "    )(inputs=[dec_inputs, enc_outputs, look_ahead_mask, dec_padding_mask])\n",
    "\n",
    "    outputs = tf.keras.layers.Dense(\n",
    "        units=vocab_size, name=\"outputs\")(dec_outputs)\n",
    "\n",
    "    return tf.keras.Model(inputs=[inputs, dec_inputs], outputs=outputs, name=name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"transformer\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "inputs (InputLayer)             [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dec_inputs (InputLayer)         [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "enc_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "encoder (Functional)            (None, None, 512)    13653504    inputs[0][0]                     \n",
      "                                                                 enc_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "look_ahead_mask (Lambda)        (None, 1, None, None 0           dec_inputs[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dec_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "decoder (Functional)            (None, None, 512)    19963392    dec_inputs[0][0]                 \n",
      "                                                                 encoder[0][0]                    \n",
      "                                                                 look_ahead_mask[0][0]            \n",
      "                                                                 dec_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "outputs (Dense)                 (None, None, 8175)   4193775     decoder[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 37,810,671\n",
      "Trainable params: 37,810,671\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "\n",
    "NUM_LAYERS = 6  # 인코더와 디코더의 층의 개수\n",
    "D_MODEL = 512  # 인코더와 디코더 내부의 입, 출력의 고정 차원\n",
    "NUM_HEADS = 8  # 멀티 헤드 어텐션에서의 헤드 수\n",
    "UNITS = 512  # 피드 포워드 신경망의 은닉층의 크기\n",
    "DROPOUT = 0.1  # 드롭아웃의 비율\n",
    "\n",
    "model = transformer(\n",
    "    vocab_size=VOCAB_SIZE,\n",
    "    num_layers=NUM_LAYERS,\n",
    "    units=UNITS,\n",
    "    d_model=D_MODEL,\n",
    "    num_heads=NUM_HEADS,\n",
    "    dropout=DROPOUT)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_function(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAX_LENGTH - 1))\n",
    "\n",
    "    loss = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "        from_logits=True, reduction='none')(y_true, y_pred)\n",
    "\n",
    "    mask = tf.cast(tf.not_equal(y_true, 0), tf.float32)\n",
    "    loss = tf.multiply(loss, mask)\n",
    "\n",
    "    return tf.reduce_mean(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "\n",
    "    def __init__(self, d_model, warmup_steps=4000):\n",
    "        super(CustomSchedule, self).__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.d_model = tf.cast(self.d_model, tf.float32)\n",
    "\n",
    "        self.warmup_steps = warmup_steps\n",
    "\n",
    "    def __call__(self, step):\n",
    "        arg1 = tf.math.rsqrt(step)\n",
    "        arg2 = step * (self.warmup_steps**-1.5)\n",
    "\n",
    "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = CustomSchedule(D_MODEL)\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(\n",
    "    learning_rate, beta_1=0.9, beta_2=0.98, epsilon=1e-9)\n",
    "\n",
    "\n",
    "def accuracy(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAX_LENGTH - 1))\n",
    "    return tf.keras.metrics.sparse_categorical_accuracy(y_true, y_pred)\n",
    "\n",
    "\n",
    "model.compile(optimizer=optimizer, loss=loss_function, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "185/185 [==============================] - 44s 238ms/step - loss: 1.3384 - accuracy: 0.0233\n",
      "Epoch 2/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 1.0752 - accuracy: 0.0490\n",
      "Epoch 3/50\n",
      "185/185 [==============================] - 45s 241ms/step - loss: 0.9841 - accuracy: 0.0506\n",
      "Epoch 4/50\n",
      "185/185 [==============================] - 45s 243ms/step - loss: 0.9411 - accuracy: 0.0525\n",
      "Epoch 5/50\n",
      "185/185 [==============================] - 44s 240ms/step - loss: 0.9096 - accuracy: 0.0546\n",
      "Epoch 6/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.8746 - accuracy: 0.0564\n",
      "Epoch 7/50\n",
      "185/185 [==============================] - 44s 239ms/step - loss: 0.8391 - accuracy: 0.0582\n",
      "Epoch 8/50\n",
      "185/185 [==============================] - 44s 238ms/step - loss: 0.7975 - accuracy: 0.0605\n",
      "Epoch 9/50\n",
      "185/185 [==============================] - 45s 243ms/step - loss: 0.7512 - accuracy: 0.0635\n",
      "Epoch 10/50\n",
      "185/185 [==============================] - 45s 243ms/step - loss: 0.7004 - accuracy: 0.0672\n",
      "Epoch 11/50\n",
      "185/185 [==============================] - 46s 249ms/step - loss: 0.6461 - accuracy: 0.0723\n",
      "Epoch 12/50\n",
      "185/185 [==============================] - 44s 238ms/step - loss: 0.5898 - accuracy: 0.0780\n",
      "Epoch 13/50\n",
      "185/185 [==============================] - 44s 238ms/step - loss: 0.5335 - accuracy: 0.0850\n",
      "Epoch 14/50\n",
      "185/185 [==============================] - 44s 240ms/step - loss: 0.4817 - accuracy: 0.0912\n",
      "Epoch 15/50\n",
      "185/185 [==============================] - 45s 244ms/step - loss: 0.4324 - accuracy: 0.0978\n",
      "Epoch 16/50\n",
      "185/185 [==============================] - 44s 239ms/step - loss: 0.3914 - accuracy: 0.1036\n",
      "Epoch 17/50\n",
      "185/185 [==============================] - 44s 237ms/step - loss: 0.3558 - accuracy: 0.1085\n",
      "Epoch 18/50\n",
      "185/185 [==============================] - 44s 237ms/step - loss: 0.3266 - accuracy: 0.1121\n",
      "Epoch 19/50\n",
      "185/185 [==============================] - 44s 237ms/step - loss: 0.3035 - accuracy: 0.1152\n",
      "Epoch 20/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.2853 - accuracy: 0.1177\n",
      "Epoch 21/50\n",
      "185/185 [==============================] - 44s 238ms/step - loss: 0.2729 - accuracy: 0.1193\n",
      "Epoch 22/50\n",
      "185/185 [==============================] - 44s 239ms/step - loss: 0.2595 - accuracy: 0.1219\n",
      "Epoch 23/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.2474 - accuracy: 0.1237\n",
      "Epoch 24/50\n",
      "185/185 [==============================] - 45s 241ms/step - loss: 0.2296 - accuracy: 0.1268\n",
      "Epoch 25/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.2146 - accuracy: 0.1291\n",
      "Epoch 26/50\n",
      "185/185 [==============================] - 45s 243ms/step - loss: 0.2016 - accuracy: 0.1311\n",
      "Epoch 27/50\n",
      "185/185 [==============================] - 45s 243ms/step - loss: 0.1906 - accuracy: 0.1330\n",
      "Epoch 28/50\n",
      "185/185 [==============================] - 45s 243ms/step - loss: 0.1811 - accuracy: 0.1346\n",
      "Epoch 29/50\n",
      "185/185 [==============================] - 45s 243ms/step - loss: 0.1727 - accuracy: 0.1361\n",
      "Epoch 30/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.1633 - accuracy: 0.1376\n",
      "Epoch 31/50\n",
      "185/185 [==============================] - 45s 241ms/step - loss: 0.1599 - accuracy: 0.1384\n",
      "Epoch 32/50\n",
      "185/185 [==============================] - 44s 239ms/step - loss: 0.1497 - accuracy: 0.1401\n",
      "Epoch 33/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.1437 - accuracy: 0.1410\n",
      "Epoch 34/50\n",
      "185/185 [==============================] - 46s 250ms/step - loss: 0.1384 - accuracy: 0.1419\n",
      "Epoch 35/50\n",
      "185/185 [==============================] - 45s 245ms/step - loss: 0.1321 - accuracy: 0.1432\n",
      "Epoch 36/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.1278 - accuracy: 0.1437\n",
      "Epoch 37/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.1223 - accuracy: 0.1451\n",
      "Epoch 38/50\n",
      "185/185 [==============================] - 45s 241ms/step - loss: 0.1177 - accuracy: 0.1458\n",
      "Epoch 39/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.1123 - accuracy: 0.1469\n",
      "Epoch 40/50\n",
      "185/185 [==============================] - 45s 241ms/step - loss: 0.1089 - accuracy: 0.1475\n",
      "Epoch 41/50\n",
      "185/185 [==============================] - 45s 243ms/step - loss: 0.1057 - accuracy: 0.1482\n",
      "Epoch 42/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.1024 - accuracy: 0.1486\n",
      "Epoch 43/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.0985 - accuracy: 0.1497\n",
      "Epoch 44/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.0935 - accuracy: 0.1507\n",
      "Epoch 45/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.0913 - accuracy: 0.1510\n",
      "Epoch 46/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.0898 - accuracy: 0.1513\n",
      "Epoch 47/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.0853 - accuracy: 0.1525\n",
      "Epoch 48/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.0836 - accuracy: 0.1528\n",
      "Epoch 49/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.0804 - accuracy: 0.1534\n",
      "Epoch 50/50\n",
      "185/185 [==============================] - 45s 242ms/step - loss: 0.0774 - accuracy: 0.1541\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fc698273610>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EPOCHS = 50\n",
    "model.fit(dataset, epochs=EPOCHS, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder_inference(sentence):\n",
    "    sentence = preprocess_sentence(sentence)\n",
    "\n",
    "    sentence = tf.expand_dims(\n",
    "        START_TOKEN + tokenizer.encode(sentence) + END_TOKEN, axis=0)\n",
    "\n",
    "    output_sequence = tf.expand_dims(START_TOKEN, 0)\n",
    "\n",
    "    for i in range(MAX_LENGTH):\n",
    "        predictions = model(inputs=[sentence, output_sequence], training=False)\n",
    "        predictions = predictions[:, -1:, :]\n",
    "\n",
    "        predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n",
    "\n",
    "        if tf.equal(predicted_id, END_TOKEN[0]):\n",
    "            break\n",
    "\n",
    "        output_sequence = tf.concat([output_sequence, predicted_id], axis=-1)\n",
    "\n",
    "    return tf.squeeze(output_sequence, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence_generation(sentence):\n",
    "    prediction = decoder_inference(sentence)\n",
    "\n",
    "    predicted_sentence = tokenizer.decode(\n",
    "        [i for i in prediction if i < tokenizer.vocab_size])\n",
    "\n",
    "    print('입력 : {}'.format(sentence))\n",
    "    print('출력 : {}'.format(predicted_sentence))\n",
    "\n",
    "    return predicted_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 가장 확실한 건 뭘까?\n",
      "출력 : 가장 확실한 시간은 오늘이에요 . 어제와 내일을 놓고 고민하느라 시간을 낭비하지 마세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'가장 확실한 시간은 오늘이에요 . 어제와 내일을 놓고 고민하느라 시간을 낭비하지 마세요 .'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation(\"가장 확실한 건 뭘까?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transformer를 이용하여 한국어 대화용 챗봇을 만들었습니다.  \n",
    "learning rate scheduling의 새로운 방법에 대해서 알게되어 더 공부해볼 필요가 있다고 생각합니다.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "aiffel",
   "language": "python",
   "name": "aiffel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}