C3_W4_lecture_notebook_accuracy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate a Siamese model: Ungraded Lecture Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens_length=568 inputs_length=512 targets_length=114 noise_density=0.15 mean_noise_span_length=3.0 \n"
     ]
    }
   ],
   "source": [
    "import trax.fastmath.numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspecting the necessary elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lecture notebook you will learn how to evaluate a Siamese model using the accuracy metric. Because there are many steps before evaluating a Siamese network (as you will see in this week's assignment) the necessary elements and variables are replicated here using real data from the assignment:\n",
    "\n",
    "   - `q1`: vector with dimension `(batch_size X max_length)` containing first questions to compare in the test set.\n",
    "   - `q2`: vector with dimension `(batch_size X max_length)` containing second questions to compare in the test set.\n",
    "   \n",
    "   **Notice that for each pair of vectors within a batch $([q1_1, q1_2, q1_3, ...]$, $[q2_1, q2_2,q2_3, ...])$  $q1_i$ is associated to $q2_k$.**\n",
    "        \n",
    "        \n",
    "   - `y_test`: 1 if  $q1_i$ and $q2_k$ are duplicates, 0 otherwise.\n",
    "   \n",
    "   - `v1`: output vector from the model's prediction associated with the first questions.\n",
    "   - `v2`: output vector from the model's prediction associated with the second questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can inspect each one of these variables by running the following cells:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q1 has shape: (512, 64) \n",
      "\n",
      "And it looks like this: \n",
      "\n",
      " [[ 32  38   4 ...   1   1   1]\n",
      " [ 30 156  78 ...   1   1   1]\n",
      " [ 32  38   4 ...   1   1   1]\n",
      " ...\n",
      " [ 32  33   4 ...   1   1   1]\n",
      " [ 30 156 317 ...   1   1   1]\n",
      " [ 30 156   6 ...   1   1   1]]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "q1 = np.load('q1.npy')\n",
    "print(f'q1 has shape: {q1.shape} \\n\\nAnd it looks like this: \\n\\n {q1}\\n\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice those 1s on the right-hand side?  \n",
    "\n",
    "Hope you remember that the value of `1` was used for padding. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q2 has shape: (512, 64) \n",
      "\n",
      "And looks like this: \n",
      "\n",
      " [[   30   156    78 ...     1     1     1]\n",
      " [  283   156    78 ...     1     1     1]\n",
      " [   32    38     4 ...     1     1     1]\n",
      " ...\n",
      " [   32    33     4 ...     1     1     1]\n",
      " [   30   156    78 ...     1     1     1]\n",
      " [   30   156 10596 ...     1     1     1]]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "q2 = np.load('q2.npy')\n",
    "print(f'q2 has shape: {q2.shape} \\n\\nAnd looks like this: \\n\\n {q2}\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_test has shape: (512,) \n",
      "\n",
      "And looks like this: \n",
      "\n",
      " [0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 0 0 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0\n",
      " 0 0 0 1 0 0 0 1 0 0 0 0 1 0 1 1 1 1 0 1 0 1 0 0 0 1 0 1 1 1 0 0 0 1 0 1 0\n",
      " 0 0 0 1 0 0 1 1 0 0 0 1 0 1 1 0 1 0 0 0 1 0 1 0 0 0 0 1 1 1 0 1 0 1 1 0 0\n",
      " 0 1 0 0 1 1 0 0 1 0 1 0 0 1 1 0 1 0 0 1 1 0 1 1 1 0 1 0 0 0 0 0 0 0 0 0 0\n",
      " 1 0 1 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 1 0 1 0 1 1 0 1 1 1\n",
      " 1 0 1 1 0 0 0 0 1 1 0 0 0 0 0 1 1 0 1 0 0 1 1 0 0 0 1 0 1 0 0 0 0 1 0 0 1\n",
      " 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0\n",
      " 1 0 0 0 0 1 0 0 0 0 1 0 1 0 1 1 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 1 0 0 0 1\n",
      " 1 0 1 1 0 0 0 1 0 1 0 1 1 0 0 0 1 0 0 0 0 1 0 0 0 0 1 1 1 0 1 0 1 1 1 0 0\n",
      " 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 0 0 1 0 0 1 0 1 0 0 1 0 0 0 0\n",
      " 0 0 1 0 1 1 0 0 0 0 1 1 0 0 0 0 0 0 0 1 1 1 0 0 1 1 1 0 1 1 0 1 0 1 1 1 0\n",
      " 1 1 0 1 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 1 1 0 1 1 1 0 0 0 1 0 1 1 1\n",
      " 0 1 0 0 0 0 0 0 0 1 1 1 0 0 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 1\n",
      " 1 0 1 0 0 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_test = np.load('y_test.npy')\n",
    "print(f'y_test has shape: {y_test.shape} \\n\\nAnd looks like this: \\n\\n {y_test}\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "v1 has shape: (512, 128) \n",
      "\n",
      "And looks like this: \n",
      "\n",
      " [[ 0.01273625 -0.1496373  -0.01982759 ...  0.02205012 -0.00169148\n",
      "  -0.01598107]\n",
      " [-0.05592084  0.05792497 -0.02226785 ...  0.08156938 -0.02570007\n",
      "  -0.00503111]\n",
      " [ 0.05686752  0.0294889   0.04522024 ...  0.03141788 -0.08459651\n",
      "  -0.00968536]\n",
      " ...\n",
      " [ 0.15115018  0.17791134  0.02200656 ... -0.00851707  0.00571415\n",
      "  -0.00431194]\n",
      " [ 0.06995274  0.13110274  0.0202337  ... -0.00902792 -0.01221745\n",
      "   0.00505962]\n",
      " [-0.16043712 -0.11899089 -0.15950686 ...  0.06544471 -0.01208312\n",
      "  -0.01183368]]\n",
      "\n",
      "\n",
      "v2 has shape: (512, 128) \n",
      "\n",
      "And looks like this: \n",
      "\n",
      " [[ 0.07437647  0.02804951 -0.02974014 ...  0.02378932 -0.01696189\n",
      "  -0.01897198]\n",
      " [ 0.03270066  0.15122835 -0.02175895 ...  0.00517202 -0.14617395\n",
      "   0.00204823]\n",
      " [ 0.05635608  0.05454165  0.042222   ...  0.03831453 -0.05387777\n",
      "  -0.01447786]\n",
      " ...\n",
      " [ 0.04727105 -0.06748016  0.04194937 ...  0.07600753 -0.03072828\n",
      "   0.00400715]\n",
      " [ 0.00269269  0.15222628  0.01714724 ...  0.01482705 -0.0197884\n",
      "   0.01389528]\n",
      " [-0.15475044 -0.15718803 -0.14732707 ...  0.04299919 -0.01070975\n",
      "  -0.01318042]]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "v1 = np.load('v1.npy')\n",
    "print(f'v1 has shape: {v1.shape} \\n\\nAnd looks like this: \\n\\n {v1}\\n\\n')\n",
    "v2 = np.load('v2.npy')\n",
    "print(f'v2 has shape: {v2.shape} \\n\\nAnd looks like this: \\n\\n {v2}\\n\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating the accuracy\n",
    "\n",
    "You will calculate the accuracy by iterating over the test set and checking if the model predicts right or wrong.\n",
    "\n",
    "The first step is to set the accuracy to zero:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will also need the `batch size` and the `threshold` that determines if two questions are the same or not. \n",
    "\n",
    "**Note :A higher threshold means that only very similar questions will be considered as the same question.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 512 # Note: The max it can be is y_test.shape[0] i.e all the samples in test data\n",
    "threshold = 0.7 # You can play around with threshold and then see the change in accuracy.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the assignment you will iterate over multiple batches of data but since this is a simplified version only one batch is provided. \n",
    "\n",
    "**Note: Be careful with the indices when slicing the test data in the assignment!**\n",
    "\n",
    "The process is pretty straightforward:\n",
    "   - Iterate over each one of the elements in the batch\n",
    "   - Compute the cosine similarity between the predictions\n",
    "       - For computing the cosine similarity, the two output vectors should have been normalized using L2 normalization meaning their magnitude will be 1. This has been taken care off by the Siamese network you will build in the assignment. Hence the cosine similarity here is just dot product between two vectors. You can check by implementing the usual cosine similarity formula and check if this holds or not.\n",
    "   - Determine if this value is greater than the threshold (If it is, consider the two questions as the same and return 1 else 0)\n",
    "   - Compare against the actual target and if the prediction matches, add 1 to the accuracy (increment the correct prediction counter)\n",
    "   - Divide the accuracy by the number of processed elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "for j in range(batch_size):        # Iterate over each one of the elements in the batch\n",
    "    \n",
    "    d = np.dot(v1[j],v2[j])        # Compute the cosine similarity between the predictions as l2 normalized, ||v1[j]||==||v2[j]||==1 so only dot product is needed\n",
    "    res = d > threshold            # Determine if this value is greater than the threshold (if it is consider the two questions as the same)\n",
    "    accuracy += (y_test[j] == res) # Compare against the actual target and if the prediction matches, add 1 to the accuracy\n",
    "\n",
    "accuracy = accuracy / batch_size   # Divide the accuracy by the number of processed elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy of the model is: 0.7421875\n"
     ]
    }
   ],
   "source": [
    "print(f'The accuracy of the model is: {accuracy}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations on finishing this lecture notebook!** \n",
    "\n",
    "Now you should have a clearer understanding of how to evaluate your Siamese language models using the accuracy metric. \n",
    "\n",
    "**Keep it up!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}