2019101019.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topics in Applied Optimization (IIIT, Hyderabad, India)\n",
    "# Jupyter Notebook Assignment-3 \n",
    "### Instructor: Dr. Pawan Kumar (IIIT, H) (https://faculty.iiit.ac.in/~pawan.kumar/)\n",
    "If you are not familiar with Jupyter notebook, before proceeding further, please go and watch this video:\n",
    "\n",
    "https://www.youtube.com/watch?v=HW29067qVWk\n",
    "\n",
    "## Regarding assignments\n",
    "### Deadline: 13 October 2019 (or 20 Oct with 15% penalty)\n",
    "- all the assignment is to be done in this notbook itself\n",
    "- any proof etc can be done on paper, and image is to be inserted in this notebook\n",
    "- save this notebook with your roll number and upload it in moodle\n",
    "- basic familiarity with python is required, brush up if necessary\n",
    "- you are not allowed to use any existing library for gradient methods; this defeats the purpose of this assignment\n",
    "- sample output is in the zip file of assignment\n",
    "- if there are any doubt, then raise it in course moodle site, it may help others\n",
    "- if you find typo, raise this issue in moodle promptly!\n",
    "- please avoid copying from others, there may be oral exam to test your knowledge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation of Recommender System Using Latent Factor Model in Python\n",
    "\n",
    "Please refer to the class slides for more detail. Also, for full detail, please refer the following main reference:\n",
    "\n",
    "**[1] Y. Koren, R. Bell, and C. Volinsky, Matrix Factorization Techniques for Recommender Systems, Computer Archive, Volume 42, Issue 8, 2009** \n",
    "\n",
    "We first load the necessary libraries: \n",
    "1. matplotlib: needed for plotting figures and \n",
    "2. numpy needed for doing math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now we write the main functions\n",
    "\n",
    "We write everything inside a class LF, which stands for latent factor. We need the following functions:\n",
    "1. main\n",
    "2. train\n",
    "3. mse\n",
    "4. sgd\n",
    "5. predict\n",
    "6. full_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Short Tutorial on Recommender Systems\n",
    "Please see lecture slides for more detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Latent Factor Model\n",
    "In a recommender systems, we are given an incomplete ratings matrix say R. The $(i,j)$ entry of this matrix is denoted by $r_{ij}.$ \n",
    "- We believe that there are dependencies in the matrix, i.e., the matrix is low rank, hence it can be written as a product of low rank matrices. \n",
    "- Even if the matrix is not strictly low rank, we believe that certain features are more important than others. In other words, if the ratings matrix were full, then after doing SVD, we get \n",
    "$$R = U S V^T$$\n",
    "some of the singular values were very small, hence the matrix can be approximated well with truncated SVD. Let us consider the first $k$ largest singular vectors, then the truncated SVD is \n",
    "$$R \\approx U_k S_k V_k.$$\n",
    "- See IPSC notes for more detail on how to compute SVD or truncated SVD\n",
    "- For a recomendation problem, for example, movie recomendation or news recommendation or product recommendation, etc, we believe that certain latent features are most important and play a major role in ratings.\n",
    "- For movie recommendation, for example, latent factors could be comedy, adventure, horror, etc. These features are called latent, because the given data, which is ratings matrix does not explicitely tells about these.\n",
    "- For recommendation systems, we cannot do SVD, because the matrix $R$ is incomplete! Obviously, we cant treat the missing entries to be zero! Hence, instead of doing SVD, we will consider an error function that consider only the ratings that are given. Let us assume that the ratings matrix can be modelled as a latent factors, i.e., suppose that $$R = PQ, \\quad P \\in \\mathbb{R}^{n \\times k}, Q \\in \\mathbb{R}^{k \\times n},$$ then the given rating $$r_{ij} = p_{i*}q_{*j},$$ where $p_{i*}$ denotes the $i$th row of $P,$ and $q_{*j}$ denotes the $j$th column of $Q.$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization Model\n",
    "- As mentioned above, we cannot directly do SVD, however, we can indeed create a **loss function** as follows:\n",
    "$$\\mathcal{L}(p_{1*}, p_{2*}, \\dots p_{n*}, q_{*1}, q_{*2}, \\dots, q_{*n}) = \\sum_{(i,j) \\in K} (r_{ij} - p_{i*}q_{*j}).$$\n",
    "- Note above that we see loss function as a function the variables $p_{1*}, p_{2*}, \\dots, p_{*n},$ which are row vectors of $P,$ and as the column vectors of $Q,$ which are $q_{*1}, q_{*2}, \\dots, q_{*n}.$ This allows us to **vectorize**. Why vectorize?\n",
    "- In machine learning the weights and their combinations with the given data creates a model. In this case, the model is $PQ.$ Since the $k,$ is a hyperparameter, it is likely that one may choose $P$ and $Q$ to have too many columns, i.e., too many weights are used, this will lead to a large model, and hence leading to **overfitting**. \n",
    "If you are not familiar with overfitting, then pause here, and see the video here:\n",
    "\n",
    "https://www.youtube.com/watch?v=u73PU6Qwl1I\n",
    "\n",
    "    - What did you learn from video? \n",
    "    - What is termed as model? \n",
    "    - What is the meaning of model being large? \n",
    "    - What is the difference between linear and logistic regression? \n",
    "    - What is underfitting? \n",
    "    - What is a bias? \n",
    "    - How do you characterize bias and variance using terms overfitting or underfitting?\n",
    "\n",
    "- A well known way to avoid overfitting is to do regularizations, by penalizing **large model** size. Let us modify our loss function as follows:\n",
    "$$\\mathcal{L}(\\cdot) = \\sum_{(i,j) \\in K} (r_{ij} - p_{i*}q_{*j})^2 + \\gamma/2 (\\| P \\|_F^2 + \\| Q \\|_F^2)$$\n",
    "    - We have added the term $\\gamma/2(\\| P \\|_F^2 + \\| Q \\|_F^2).$ It is called **regularization term**\n",
    "    - The parameter $\\gamma$ is called the **regularization parameter**\n",
    "    - High $\\gamma$ would mean that we don't want big model size; this may help prevent overfitting\n",
    "- Let $e_{ij}$ denote the error corresponding to each $(ij)$ term, \n",
    "$$e_{ij} = r_{ij} - \\sum_{s=1}^k p_{is}q_{sj}$$\n",
    "then let us define the new error term $\\tilde{e}$\n",
    "$$\\tilde{e}_{ij} = e_{ij}^2 + \\gamma/2 (\\|P\\|_F^2 + \\| Q \\|_F^2),$$ \n",
    "that is,\n",
    "$$\\tilde{e}_{ij} = (r_{ij} - \\sum_{s=1}^k p_{is}q_{sj} )^2 + \\gamma/2 (\\|P\\|_F^2 + \\| Q \\|_F^2)$$\n",
    "- Note that $$\\mathcal{L}(\\cdot) = \\sum_{(i,j) \\in \\mathcal{K}} \\tilde{e}_{ij},$$\n",
    "where $\\mathcal{K}$ is the set of all indices for which ratings $r_{ij}$ are available.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Method Versus Stochastic Gradient Methods\n",
    "Following notes are taken from \n",
    "https://en.wikipedia.org/wiki/Stochastic_gradient_descent\n",
    "\n",
    "For large dataset, $k$ and $n$ tend to be large, and computing gradient fully becomes too demanding. In machine learning or statistical estimation, we are usually required to find minimum of the loss functions of the form \n",
    "$$Q(w) = 1/n \\sum_{j=1}^n Q_j (w)$$\n",
    "That is loss function as \"additive\" decomposition. In other words, the loss function is a sum of the loss function where the sum is over the data samples. If $n$ is large, then computing full gradient is computationally demanding. However, if we indeed decide to run the gradient method. Then we recall the following steps:\n",
    "1. Take initial random weight (possibly random and normally distributed between 0 and 1): $w^0$\n",
    "2. $w^{i+1} = w^i - \\alpha \\nabla Q(w^i), \\quad i=0, \\dots$\n",
    "\n",
    "Here the gradient $\\nabla Q$ is the full gradient in the sense that\n",
    "\\begin{align}\n",
    "\\nabla Q(w) = \\sum_{i=1}^n \\nabla_{w}Q_i(w). \\label{grad} \\tag{1}\n",
    "\\end{align}\n",
    "\n",
    "Here\n",
    "\\begin{align}\n",
    "\\nabla_w Q_i (w) = \\begin{bmatrix}\n",
    "\\dfrac{\\partial Q_i (w)}{\\partial w_1} \\\\\n",
    "\\dfrac{\\partial Q_i (w)}{\\partial w_1} \\\\\n",
    "\\vdots \\\\\n",
    "\\dfrac{\\partial Q_i (w)}{\\partial w_n} \\\\\n",
    "\\end{bmatrix}  \\label{grad2} \\tag{2}\n",
    "\\end{align}\n",
    "\n",
    "### Stochastic gradient  descent method\n",
    "Stochastic gradient merthod, in short, SGD is a method (not necessarily descent due to stochasticity, but in expectation is usually is a descent), where the gradient is computed partially. For example, instead of summing for all $i=1...n,$ we can take a random sample, say $j_1,$ and do the update only for this sample as follows:\n",
    "$$\\nabla Q(w) = \\nabla_{w}Q_{j_1}(w),$$\n",
    "that is we have picked just one of the term i.e., $j_1$th term from \\eqref{grad}.\n",
    "Then the SGD update is:\n",
    "$$w^{i+1} = w^i - \\alpha \\nabla Q_{j_1}(w^i), \\quad i=0, \\dots$$\n",
    "Note that we could have taken few more terms in the sum, and would have updated for a batch. This is called, batched stochastic gradient descent.\n",
    "It sounds incorrect, isnt it? Why such method may converge? With some basic assumption on the loss function, we can prove some convergence in expectance. More on convergence theory in class!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-0:}}$ Compute the full gradient of the loss function for recommendation system\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$  \n",
    "## Please find the answer in attached pdf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following, we derive a stochastic gradient for the loss function for recommender loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Method for Optimization Problem\n",
    "- We want to use gradient method to minimize the error. An update in gradient method is given as follows\n",
    "$$w^{i+1} = w^i - \\nabla \\mathcal{L}(w^i)$$\n",
    "- To use gradient method, we need to compute gradient. If we see $\\mathcal{L}$ as a function of $p_{ik}'$s and $q_{ki}'$s, then \n",
    "$$\\dfrac{\\partial \\mathcal{L}}{\\partial p_{is}} = 2e_{is} \\dfrac{\\partial e_{is}}{\\partial p_{is}} + \\gamma p_{is} = -2e_{is} q_{sj} + \\gamma p_{is}$$\n",
    "Similarly,\n",
    "$$\\dfrac{\\partial \\mathcal{L}}{\\partial q_{sj}} = 2e_{sj} \\dfrac{\\partial e_{sj}}{\\partial q_{sj}}+\\gamma q_{sj} = -2e_{sj}p_{js} + \\gamma q_{sj},$$\n",
    "- The gradient vector $\\nabla \\mathcal{L}$ is given by\n",
    "$$\\nabla \\mathcal{L} = \\left(\\dfrac{\\partial \\mathcal{L}}{\\partial p_{11}}, \\dots, \\dfrac{\\partial \\mathcal{L}}{\\partial p_{nk}}, \\dfrac{\\partial \\mathcal{L}}{\\partial q_{11}}, \\dots, \\dfrac{\\partial \\mathcal{L}}{\\partial p_{kn}} \\right)$$\n",
    "- The weights $p_{is}$ can be updated as\n",
    "$$p_{is} = p_{is} + \\alpha (2e_{ij} q_{sj} - \\gamma p_{is}) $$\n",
    "- The weights $p_{sj}$ can be updated as\n",
    "$$q_{sj} = q_{sj} + \\alpha (2e_{ij} p_{js} - \\gamma q_{sj}) $$\n",
    "- We can vectorize $s.$ Finally, the vectorized form of the update looks like:\n",
    "$$(p_{i+1*}, q_{*j+1})^T = (p_{i*}, q_{*j})^T + \\alpha ((2e_{ij} q_{*j} - \\gamma p_{i*}), (2e_{ij} p_{j*} - \\gamma q_{*j}))^T  $$\n",
    "- The error at $(p_{i*}, q_{*j})^T$ nedded above can be computed by $$e_{ij} =  r_{ij} - \\text{prediction at}~ (i,j) $$\n",
    "Here prediction at $(i,j)$ can be computed by making a call to predict() function below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-1:}}$ Why the error is computed this way?\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Either use latex (preferred) or write it on paper, take pic, and insert it here. See the video on jupyter above on how to paste images.\n",
    "\n",
    "##  ** please find the answer in attached pdf**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-2:}}$ Are the updates for $p_{i*}$ and $q_{*j}$ stochastic? Justify.\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ \n",
    "\n",
    "Yes the updates for $p_{i*}$ and $q_{*j}$ are stochastic as they are getting updated after every<br> training sample. As the rating in one cell is given by a product of a whole row from P , i.e.$P_{i*}$ and a whole column<br> from Q, i.e. $Q_{*j}$, the update of one sample updates the weights of whole row and column in P and Q respectively. The gradient of loss function of a single term , i.e. $e_{ij}$ updates the entire row and column in P and Q. \n",
    "##  ** please find the rest of the answer in attached pdf **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding Bias\n",
    "It is often observed in practice that adding a bias term helps. Let us add the bias term $b$ to the prediction function. The bias term is computed as the mean of the ratings matrix. See the predict function below to see how bias is added."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-2b:}}$ Read the main reference paper and justify why bias is added. Note in this notebook we add global bias. Which other bias term was suggested in the paper?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Answer:}}$\n",
    "\n",
    "1. The purpose of adding bias is to keep a balance between overfitting and underfitting. The model's variance can be<br> changed by adding bias. The purpose for adding a bias value is to fit the data better. There can be a possibility that model would hyper personalise the rating for the user and lead to overfiiting. So, it would only provide correct results for people in the training set and would lead to inaccurate results in new data. variance van't be decreased without increasing the bias. So, in order to make the model more general such that it works for all kind of data, bias needs to be added.\n",
    "\n",
    "2. The other bias terms that were suggested in the paper were:\n",
    "\n",
    " The observed\n",
    "variation in rating values is due to effects associated with\n",
    "either users or items, known as biases or intercepts, independent of any interactions. For example, typical collaborative\n",
    "filtering data exhibits large systematic tendencies for some\n",
    "users to give higher ratings than others, and for some items\n",
    "to receive higher ratings than others. After all, some products\n",
    "are widely perceived as better (or worse) than others. The other bias terms are $ b_i $ and $ b_u $ which are item and user bias respectively apart from the bias already defined which is , u, i.e. global average.\n",
    "\n",
    "<center> So, $ r_{ui}  = b_i + b_u + q_i^Tp_u + u $ </center>\n",
    "\n",
    "<center> $ b_i $ denotes the item bias which is the value by which average rating of an item differs from other items </center>\n",
    "<center> $ b_u $ denotes the user bias which is the value by which average rating of a user differs from other users </center>\n",
    "<center> u is the average rating from the entire matrix </center>\n",
    "<center> $ r_{ui} $ is the predicted rating. </center>\n",
    "\n",
    "    \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LF(): \n",
    "    \n",
    "    def __init__(self, R, K, alpha, gamma, iterations):\n",
    "        \"\"\"\n",
    "        Parameters:\n",
    "        - R (ndarray)   : user-item rating matrix\n",
    "        - K (int)       : number of latent dimensions\n",
    "        - alpha (float) : learning rate\n",
    "        - gamma (float)  : regularization parameter\n",
    "        \"\"\"\n",
    "        \n",
    "        self.R = R\n",
    "        self.num_users, self.num_items = R.shape\n",
    "        print(self.num_items)\n",
    "        self.K = K\n",
    "        self.alpha = alpha\n",
    "        self.gamma = gamma\n",
    "        self.iterations = iterations\n",
    "#         j=1\n",
    "#         b_i=np.mean(self.R.T[j][np.where(self.R.T[j]!=0)])\n",
    "#         b_u=np.mean(self.R[i][np.where(self.R[i] != 0)])\n",
    "#         print(self.R[np.where(self.R[:,j]!=0)\n",
    "#         print(b_i)\n",
    "\n",
    "    def train(self):\n",
    "        # Initialize user and item latent feature matrice\n",
    "        self.P = np.random.normal(scale=1./self.K, size=(self.num_users, self.K))\n",
    "        self.Q = np.random.normal(scale=1./self.K, size=(self.num_items, self.K))\n",
    "        \n",
    "        # Compute the global bias\n",
    "        # For Question-6, you may want to initialize additional bias terms here\n",
    "        self.b = np.mean(self.R[np.where(self.R != 0)])\n",
    "#         self.b=0\n",
    "        #self.b_i = TODO \n",
    "        #self.b_u = TODO\n",
    "        \n",
    "        # Create a list of training samples\n",
    "        self.samples = [\n",
    "            (i, j, self.R[i, j])\n",
    "            for i in range(self.num_users)\n",
    "            for j in range(self.num_items)\n",
    "            if self.R[i, j] > 0\n",
    "        ]\n",
    "#         print(self.samples)\n",
    "        # Perform stochastic gradient descent for number of iterations\n",
    "        training_process = []\n",
    "        for i in range(self.iterations):\n",
    "            np.random.shuffle(self.samples)\n",
    "            self.sgd()\n",
    "            mse = self.mse()\n",
    "            training_process.append((i, mse))\n",
    "            if (i+1) % 10 == 0:\n",
    "                print(\"SGD Iteration: %d ; error = %.4f\" % (i+1, mse))\n",
    "        \n",
    "        return training_process\n",
    "\n",
    "    def mse(self):\n",
    "        \"\"\"\n",
    "        MSE: Compute Mean Square Error\n",
    "        \"\"\"\n",
    "        xs, ys = self.R.nonzero()\n",
    "        predicted = self.full_matrix()\n",
    "        error = 0\n",
    "        for x, y in zip(xs, ys):\n",
    "            error += pow(self.R[x, y] - predicted[x, y], 2)\n",
    "            error+=((self.gamma)/2)*(np.sum(np.square(self.P[x]))+np.sum(np.square(self.Q[y])))\n",
    "#         print(error)\n",
    "        return np.sqrt(error)\n",
    "\n",
    "    def sgd(self):\n",
    "        \"\"\"\n",
    "        SGD: Stochastic Graident Descent\n",
    "        \"\"\"\n",
    "        for i, j, r in self.samples:\n",
    "            # Compute prediction and error\n",
    "            prediction = self.predict(i,j)\n",
    "            e = self.R[i,j]-prediction\n",
    "            \n",
    "            # Keep a copy of row of P. It will be needed for update\n",
    "            P_i = self.P[i, :]\n",
    "            Q_j= self.Q[j,:]\n",
    "            self.P[i, :] = np.add(P_i,self.alpha*2*e*Q_j-self.alpha*self.gamma*P_i)\n",
    "            self.Q[j, :] = np.add(Q_j,self.alpha*2*e*P_i-self.alpha*self.gamma*Q_j)\n",
    "            \n",
    "            \n",
    "    def gd(self):\n",
    "        \"\"\"\n",
    "        GD: Graident Descent\n",
    "        Replace self.sample below to full batch for gradient descent\n",
    "        Feel free to remove everything, and implement from scratch\n",
    "        \"\"\"\n",
    "        change_p=np.zeros((self.num_users,self.K))\n",
    "        change_q=np.zeros((self.num_items,self.K))\n",
    "        for i, j, r in self.samples:\n",
    "            # Compute prediction and error\n",
    "            prediction = self.predict(i,j)\n",
    "            e = self.R[i,j]-prediction\n",
    "            \n",
    "            # Create copy of row of P since we need to update it but use older values for update on Q\n",
    "            P_i = self.P[i, :]\n",
    "            Q_j= self.Q[j,:]\n",
    "            \n",
    "            # Update user and item latent feature matrices\n",
    "            change_p[i, :] = np.add(change_p[i,:],self.alpha*2*e*Q_j-self.alpha*self.gamma*P_i)\n",
    "            change_q[j, :] = np.add(change_q[j,:],self.alpha*2*e*P_i-self.alpha*self.gamma*Q_j)\n",
    "        np.multiply(change_p,(1/len(self.samples)))\n",
    "        np.multiply(change_q,(1/len(self.samples)))\n",
    "        self.P=np.add(self.P,change_p)\n",
    "        self.Q=np.add(self.Q,change_q)\n",
    "    \n",
    "            \n",
    "\n",
    "    def predict(self, i, j):\n",
    "        \"\"\"\n",
    "        Prediction: Predicted the rating of user i and item j\n",
    "        \"\"\"\n",
    "        prediction = self.b + self.P[i, :].dot(self.Q[j, :].T)\n",
    "\n",
    "        return prediction\n",
    "    \n",
    "    def full_matrix(self):\n",
    "        \"\"\"\n",
    "        The rating matrix using the biases P and Q\n",
    "        \"\"\"\n",
    "        return self.b + self.P.dot(self.Q.T)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "SGD Iteration: 10 ; error = 4.2185\n",
      "SGD Iteration: 20 ; error = 2.1467\n",
      "SGD Iteration: 30 ; error = 1.5079\n",
      "SGD Iteration: 40 ; error = 1.2384\n",
      "SGD Iteration: 50 ; error = 1.0365\n",
      "\n",
      "P x Q:\n",
      "[[4.83515828 3.40481059 1.48761974 1.13283659]\n",
      " [4.16447545 4.04156284 1.30044502 1.32705947]\n",
      " [0.8006628  1.12910623 4.73063718 4.74203847]\n",
      " [1.00405832 2.61564184 3.58560119 4.01171616]\n",
      " [0.90934326 1.20189342 4.63505998 4.64022357]]\n",
      "\n",
      "Global bias:\n",
      "2.769230769230769\n",
      "\n"
     ]
    }
   ],
   "source": [
    "R = np.array([\n",
    "    [5, 3, 0, 1],\n",
    "    [4, 0, 0, 1],\n",
    "    [1, 1, 0, 5],\n",
    "    [1, 0, 0, 4],\n",
    "    [0, 1, 5, 4],\n",
    "])\n",
    "lf = LF(R, K=2, alpha=0.01, gamma=0.001, iterations=50)\n",
    "training_process = lf.train()\n",
    "print()\n",
    "print(\"P x Q:\")\n",
    "print(lf.full_matrix())\n",
    "print()\n",
    "print(\"Global bias:\")\n",
    "print(lf.b)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-3:}}$ Implement the full gradient descent method\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Complete gd function above\n",
    "\n",
    "## completed in the code above "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-4:}}$ Vectorize the updates of p and q, by vectorizing s (detailed above)\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Fill the TODO in code above\n",
    "## completed in the code above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the code below, answer the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6YAAAEGCAYAAAB2AYgsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA+1ElEQVR4nO3dd3ic1Zn///dRlyxbcpWMu8HGYNNNbyaQUJcUNm13k2ySDekJqb/Npmyy2Xyzm06ym0LKJiGFkJAEUmgBDCbBgE0xBtvYuODeLdvq5fz+mJEsG81oRsWj8n5d11yado7ukW8/o4+eZ84TYoxIkiRJkpQrebkuQJIkSZI0vBlMJUmSJEk5ZTCVJEmSJOWUwVSSJEmSlFMGU0mSJElSThXkuoDOxo0bF6dPn57rMiRJkiRJfWzp0qW7Yozju3psQAXT6dOns2TJklyXIUmSJEnqYyGEDake69dDeUMIlSGE34QQVoYQVoQQzu3P7ydJkiRJGnz6e4/pjcBdMca/DyEUAWX9/P0kSZIkSYNMvwXTEMIo4CLgnwFijE1AU399P0mSJEnS4NSfe0xnAjuB/wshnAIsBT4YY6zt/KQQwvXA9QBVVVUsXLiwH0uSJEmSJA00IcbYPxOHMB9YDJwfY3w0hHAjsD/G+OlUY+bPnx9d/EiSJEmShp4QwtIY4/yuHuvPxY82AZtijI8mb/8GOL0fv58kSZIkaRDqt2AaY9wGbAwhHJ+861Lguf76fpIkSZKkwam/V+V9P/Dz5Iq8a4G39vP36zf3r9zOul11XDmvmmMqS3NdjiRJkiQNGf0aTGOMTwFdHkM82Ny3Ygc/f/RFPv/H5zhtaiVXzZvIFfOqmTLGM+BIkiRJUm/02+JHPTHQFz9au/Mgdy7fxp+f2cqzW/YDcMrkCq48aSJXzqtm2tgROa5QkiRJkgamdIsfGUx7aMPuWu5cvo07n9nK05tqAJh7zCiuSobUmePLc1yhJEmSJA0cBtN+tnFPHXct38afl2/lyRf3ATCneiRXnTSRq06q5rgJI3NboCRJkiTlmMH0KNqyr567lm/jzuVbWbJhLzHCrAnlyZA6kdlV5YQQcl2mJEmSJB1VBtMc2b6/IbEn9ZmtPLZ+T0dIfeWpx3DtKZOYOtaFkyRJkiQNDwbTAWDHgQbuXr6NPzydCKkAp02t5JWnHMPVJx/D+JHFOa5QkiRJkvqPwXSA2byvnj88vYXbn9rCiq37yQtw/nHjeOWpk7h8bhUjSwpzXaIkSZIk9SmD6QD2/PYD3PHUFm5/ejMb99RTVJDHZSdM4NpTJrHg+PGUFObnukRJkiRJ6jWD6SAQY+TJjfu446kt/HHZFnYdbGJkSQFXzqvmladO4pyZY8nPc9EkSZIkSYOTwXSQaWlt428v7Ob2p7Zw97PbONjYwviRxfzdycfwylOP4eTJFa7sK0mSJGlQMZgOYg3Nrdy/cge3P7WZB1bupKm1jdlV5bz2jCm86rRJLpokSZIkaVAwmA4RNfXN/GnZVn69dCNPvriPgrzAJXMm8NozJnPJnAkU5uflukRJkiRJ6pLBdAhavf0Av1m6idue2Myug42MKy/i1adN4rXzpzC7amSuy5MkSZKkwxhMh7Dm1jYeXLWTXy/dyH0rdtDSFjllSiWvPWMyf3fKMVSUeuoZSZIkSblnMB0mdh1s5PdPbubXSzaxavsBigvyuGJeNa89YwrnHTuWPFf1lSRJkpQjBtNhJsbIM5tr+PWSTdz+1Gb2N7QwqbKU686YzGvPmMyUMWW5LlGSJEnSMGMwHcYamlu557nt/HrJRh5es4sY4YLjxvFP50zjshMmUOCCSZIkSZKOAoOpANi8r57blm7il4+9yNaaBiZWlPDGs6byhjOnMGFUSa7LkyRJkjSEGUx1mJbWNu5fuYObF29g0epdFOQFLp9XzZvOmcbZM8YQgp9FlSRJktS30gXTgqNdjHKvID+PV8yt5hVzq1m3q5afL97ArUs28qdlW5k1oZw3nTuNV582iZElrugrSZIkqf+5x1QA1De18odlW7j5kQ08s7mGEUX5vOq0Sbzp3GnMqR6V6/IkSZIkDXIeyqusPL1xHzcv3sAdT2+hqaWNM6eP5p/OmcaV8yZSVOBiSZIkSZKyZzBVj+ytbeLXSzfys8Uv8uKeOsaVF/H6M6fwD2dPY1Jlaa7LkyRJkjSIGEzVK21tkYdW7+Rnizdw38odBODKkybyjgtncuqUylyXJ0mSJGkQcPEj9UpeXmDB8RNYcPwENu6p42eLN/CLR1/kT8u2cub00bzjwplcdkIVeXmu5itJkiQpe+4xVY8cbGzhV49v5EcPr2PzvnpmjBvB2y+YwXWnT6a0KD/X5UmSJEkaYDyUV/2mpbWNO5dv4/uL1rJsUw2jywp507nTefO50xhXXpzr8iRJkiQNEAZT9bsYI4+t28P3F63lLyt2UFSQx3WnT+LtF8zkuAnluS5PkiRJUo75GVP1uxACZ88cy9kzx7Jmx0F++PA6bntiE798bCOXzpnAv1w4k3NmjiEEP4cqSZIk6XDuMVW/2XWwkZsf2cDNizewp7aJeZNG8Y4LZ3LVSRMpzPd8qJIkSdJw4qG8yqmG5lZue2ITP1y0jrW7ajmmooS3XziTfzhrqgslSZIkScOEwVQDQltb5P6VO7jpobU8tn4P48qLeMeFM/mnc6YxotijyiVJkqShLGfBNISwHjgAtAItqYpoZzAdPh5bt4dv3b+aRat3MbqskLdfMIM3nzedUSWFuS5NkiRJUj/IdTCdH2PclcnzDabDzxMv7uV/7l/D/St3MKqkgLeeP4O3nT+DijIDqiRJkjSUGEw14D2zqYZv3b+ae57bTnlxAW85bxpvv2AmY0YU5bo0SZIkSX0gl8F0HbAXiMD3Yow3dfGc64HrAaqqqs645ZZb+q0eDXwbD7RxxwtNLNnWSlE+XDKlkCtnFFJR7GlmJEmSpMHskksuyVkwPSbGuCWEMAG4F3h/jPGhVM93j6nard5+gP99YA13PL2Fwvw8/uHsqbzzomOprijJdWmSJEmSemBArMobQvgscDDG+JVUzzGY6kjrdtXy7QfW8NsnN5MfAq87czLvXnAckypLc12aJEmSpCzkJJiGEEYAeTHGA8nr9wL/EWO8K9UYg6lS2binjm8vfIHfLN0IwHWnT+Z9LzuOyaPLclyZJEmSpEzkKpjOBH6XvFkA/CLG+IV0Ywym6s6WffV898EXuOXxjRDhH86eyvtedhzjyotzXZokSZKkNAbEobyZMJgqU1v21fPN+1bz66WbKC7I4+0XzOAdF830PKiSJEnSAGUw1ZC1dudBvnbv8/xx2VYqywp598XH8pbzplNSmJ/r0iRJkiR1YjDVkLd8cw1fvnsVDz6/k6pRxXzg0lm8bv4UCvPzcl2aJEmSJNIHU39r15Awb1IFP3nbWfzq+nOYPLqMT/5uOS//2oPc/tRm2toGzh9fJEmSJL2UwVRDytkzx/Kbd53Lj/55PiWF+Xzwlqe46puLuG/FdgbS0QGSJEmSDjGYasgJIfCyOVX8+QMXcuMbTqW+uZW3/2QJr/3uIzy6dneuy5MkSZJ0BIOphqy8vMArT53EXz58MV949Tw27q3j9Tct5i0/eozlm2tyXZ4kSZKkJBc/0rDR0NzKT/62nm8vfIGa+mauOXki/98Vc5gypizXpUmSJElDnqvySp3sb2jm+w+t5fuL1tLWBm+9YDrvveQ4z4EqSZIk9SNX5ZU6GVVSyEdecTwPfHQBf3fKMXzvwbUs+PJCbl68gZbWtlyXJ0mSJA07BlMNWxMrSvnq607hD++7gFkTyvn075dzxY2LeGDlDlfwlSRJko4ig6mGvZMmV3DL9edw05vOoLUt8tYfP86bf/QYK7ftz3VpkiRJ0rBgMJVInGLmFXOrufuGi/jMNSeybFMNV924iE/8dhk7DjTkujxJkiRpSEsbTEMI+SGEvxytYqRcKyrI420XzODBjy3grefP4NdLNnHJlxfyP/evpqG5NdflSZIkSUNS2mAaY2wF6kIIFUepHmlAqCwr4tPXnMi9H76YC2aN4yv3PM/LvrKQ3z25ibY2P38qSZIk9aVuTxcTQrgVOAe4F6htvz/G+IG+LsbTxWigWrx2N//5p+dYvnk/J0+u4FNXn8hZM8bkuixJkiRp0OjVeUxDCG/p6v4Y40/6oLbDGEw1kLW1RX7/1Ga+dNcqtu1v4Iq51fzbVScwdWxZrkuTJEmSBrxeBdPkBEXA7OTNVTHG5j6sr4PBVINBfVMr31+0lu8++AItbZF3XjST9yw4jtKi/FyXJkmSJA1Y6YJpt6vyhhAWAKuB/wW+DTwfQrioLwuUBpPSonw+cOks7v/IAq6YW8237l/DpV9dyJ+WbfX8p5IkSVIPZHK6mK8Cr4gxXhxjvAi4HPh6/5YlDXzVFSV8842nces7z6WirIj3/uIJ/vEHj/L89gO5Lk2SJEkaVDIJpoUxxlXtN2KMzwOF/VeSNLicNWMMf3jf+Xz+lXN5dst+rrxxEZ/7w7PU1PfLEe+SJEnSkJNJMF0aQvhhCGFB8vJ9YGl/FyYNJgX5ebzp3Ok88NEFvP7MKfz4b+u59KsLufXxjZ5eRpIkSepGJqvyFgPvBS4AAvAQ8O0YY2NfF+PiRxoqlm+u4d/veJalG/ZyypRKPnftXE6dUpnrsiRJkqSc6fGqvCGEPGBZjHFefxXXmcFUQ0mMkd89uZkv3rmSnQcaed38yXz8ijmMKy/OdWmSJEnSUdfjVXljjG3A0yGEqf1SmTSEhRB4zemTuf8jF3P9RTP57RObueQrC/nRw+tobm3LdXmSJEnSgJHJobz3A2cCjwG17ffHGK/t62LcY6qhbM2Og3zuD8+yaPUuZleV89lr53LeseNyXZYkSZJ0VPT4UN7k4Iu7uj/G+GAf1HYYg6mGuhgj9zy3nc//8Tk27a3n6pMm8qlrTmBiRWmuS5MkSZL6lZ8xlQaYhuZWvvfgWr69cA35eYEPXTabfz5/OoX5mSyULUmSJA0+fsZUGmBKCvP54GWz+MuHL+acmWP5wp9XcM03H+bx9XtyXZokSZJ01GWye2Yi8GwI4b4Qwh3tl/4uTBoOpowp44dvmc9NbzqDg40tvPa7j/DRXz/N7oN9fjYmSZIkacAqyOA5n+v3KqRhLITAK+ZWc8GscXzr/jV8/6G13Pvcdj5+xfG84cyp5OeFXJcoSZIk9auUnzENIcyJMa5MXi+OMTZ2euycGOPivi7Gz5hKsHr7AT59+3IWr93DKVMq+c9XzuOkyRW5LkuSJEnqlZ5+xvQXna4/csRj3+51VZK6NKtqJL98xzl84/WnsnlvPa/834f5zO3LqalvznVpkiRJUr9IF0xDiutd3U49SQj5IYQnQwh/zKoyaRgLIfCq0yZx30cu5s3nTudnizdw6VcX8rsnN9HdKZ4kSZKkwSZdMI0prnd1O50PAiuyeL6kpIrSQj577VzueN8FTBpdxod+9TRvuGkxq7cfyHVpkiRJUp9JF0wnhxC+GUL4Vqfr7bcnZTJ5CGEycDXwgz6oVRq25k2q4HfvPo//9+qTWLntAFfeuIj/unMldU0tuS5NkiRJ6rV0q/J+rNP1I1ckynSFom8AHwdGpnpCCOF64HqAqqoqFi5cmOHU0vBzDPD5cwv59arIdx98gVsfXcs/nlDE6RPyCcHVeyVJkjQ4pVyVt9cTh3ANcFWM8T0hhAXAR2OM16Qb46q8UuaWrN/Dp36/nJXbDnDZCRP47LVzmTy6LNdlSZIkSV3q6aq8vXU+cG0IYT1wC/CyEMLP+vH7ScPK/Olj+MP7L+DfrprDX9fs5uVfe4jvPvgCza1tuS5NkiRJykq/7TE97Ju4x1TqV5v31fO5O57lnue2M7uqnC+8+iTOnD4m12VJkiRJHXK1x1TSUTKpspSb3jyf7795PrWNrbz2u4/w8d88zZ7aplyXJkmSJHWr22AaQpgdQrgvhLA8efvkEMKnsvkmMcaF3e0tldR7Lz+xins/fBHvvHgmv31iM5d+dSG3LtnouU8lSZI0oGWyx/T7wCeAZoAY4zLgDf1ZlKSeKysq4BNXnsAfP3ABx44v5+O/Wcbrv7eY5z33qSRJkgaoTIJpWYzxsSPu8+SJ0gA3p3oUt77zXP77upN4fscBrrpxEf9910rqm1pzXZokSZJ0mEyC6a4QwrFABAgh/D2wtV+rktQn8vICrz9zKvd9+GJeddokvrPwBV7+9Qe5f+X2XJcmSZIkdcgkmL4X+B4wJ4SwGbgBeFd/FiWpb40tL+Yrrz2FX11/DqWF+bztx0t4181L2VpTn+vSJEmSpPTBNISQD7w7xngZMB6YE2O8IMa44ahUJ6lPnT1zLH/6wIV8/IrjWfj8Di776oP8YNFaWjz3qSRJknIobTCNMbYCZySv18YYXT1FGuSKCvJ4z4LjuPdDF3P2zLH8559WcM23Hmbphj25Lk2SJEnDVCaH8j4ZQrgjhPCmEMJr2i/9XpmkfjVlTBk/fMt8vvtPZ1BT38x133mEf71tGXs996kkSZKOsoIMnjMG2A28rNN9Efhtv1Qk6agJIXDFvGounDWOG+9bzQ8fXsfdz27jE1eewN+fMZm8vJDrEiVJkjQMhBhjrmvoMH/+/LhkyZJclyENWyu37edTv1vOkg17mT9tNP/56nnMqR6V67IkSZI0BIQQlsYY53f5WHfBNIRQArwdmAuUtN8fY3xbXxYJBlNpIGhri/zmiU188c8r2N/QwtsvmMEHL53FiOJMDrCQJEmSupYumGbyGdObgWrgcuBBYDLgIkjSEJWXF3jd/Cnc/5EFvG7+ZG56aC2Xfe1B7lq+lYF0hIUkSZKGjkyC6XExxk8DtTHGnwBXAyf1b1mScm30iCK++JqTue3d51FZVsS7fvYEb/vx47y4uy7XpUmSJGmIySSYNie/7gshzAMqgOn9VpGkAeWMaaP5w/vO51NXn8Bj6/bw8q8/yLfuW01jS2uuS5MkSdIQkUkwvSmEMBr4NHAH8BzwpX6tStKAUpCfx79cOJP7PrKAy06o4qv3Ps+VNy7ir2t25bo0SZIkDQGuyispawtX7eDf73iWDbvruPaUY/jU1ScwYVRJ9wMlSZI0bPV2Vd7PdHV/jPE/+qC2wxhMpcGjobmVby98ge8ufIGigjxuuGwWbzlvOoX5mRyIIUmSpOGmt6vy1na6tAJX4mdMpWGvpDCfD798Nnd/6CLmTx/Nf/5pBVfduIi/eXivJEmSspT1obwhhGLgjhjj5X1djHtMpcEpxsh9K3bwuT8+y8Y99Vx98kQ+edUJHFNZmuvSJEmSNED0do/pkcqAmb0rSdJQEkLgshOruPdDF/Phl8/mL89t59KvPsj/PrDG1XslSZLUrW6DaQjhmRDCsuTlWWAVcGP/lyZpsCkpzOcDl87iLx++mItnj+fLd6/i8q8/xAOrduS6NEmSJA1gmSx+NK3TzRZge4yxpT+K8VBeaWh56PmdfPYPz7J2Zy2XnVDFZ645kaljy3JdliRJknKgt6vyjkn3eIxxTy9qO4zBVBp6mlra+L+/ruPG+1bT0hZ518XH8u6Lj6W0KD/XpUmSJOko6m0wXQ9MAfYCAagEXkw+HGOMffZ5U4OpNHRtq2ngi3eu4PantjCpspRPX3Mil8+tIoSQ69IkSZJ0FPR28aO7gL+LMY6LMY4FrgF+G2Oc0ZehVNLQVl1Rwo1vOI1brj+HkSUFvOtnS3nzjx7jhZ0Hc12aJEmSciyTYHpmjPHP7TdijHcCF/dfSZKGsnNmjuWP77+Az/7diTy1cR9XfOMhvnjnCg40NOe6NEmSJOVIJsF0VwjhUyGE6SGEaSGETwK7+7swSUNXQX4e/3z+DB746AJefdokvvfgWi75ykJ++diLtLZld25lSZIkDX6ZBNM3AuOB3wG/ByYk75OkXhlXXsyX/v4Ubn/v+cwYN4JP/PYZrv7mIhat3pnr0iRJknQUdbv40WFPDmE0sC9mMygLLn4kDV8xRu5avo0v3rmSF/fUccnx4/nk1Sdw3ISRuS5NkiRJfaBHix+FED4TQpiTvF4cQrgfWANsDyFc1j+lShquQghcedJE7v3wRfzbVXNYsn4vl39jEZ+5fTl7aptyXZ4kSZL6UbpDeV8PrEpef0vyuRNILHz0//q5LknDVHFBPtdfdCwLP7aAfzx7Kj9/9EUu/vIDfP+htTS2tOa6PEmSJPWDdMG0qdMhu5cDv4wxtsYYVwAF/V+apOFsbHkx//HKedx9w4XMnzaaL/x5BS//2kPc+cxW+unTBJIkScqRdMG0MYQwL4QwHrgEuKfTY2X9W5YkJRw3YST/99az+OnbzqK0MJ93//wJXv+9xSzbtC/XpUmSJKmPpAumHwR+A6wEvh5jXAcQQrgKeLK7iUMIJSGEx0IIT4cQng0hfK5PKpY0LF00ezx/+sAFfPE1J7F210Gu/Z+/8uFfPcXWmvpclyZJkqReympV3qwmDiEAI2KMB0MIhcDDwAdjjItTjXFVXkmZONDQzHcWvsAPHl5HXoDrL5zJOy8+lhHFfspAkiRpoOrRqry9FRMOJm8WJi9+MExSr40sKeTjV8zh/o9czMtPrOab96/h4i8v5Md/XecCSZIkSYNQv+0xBQgh5ANLgeOA/40x/n9dPOd64HqAqqqqM2655ZZ+q0fS0LRmXyu/eb6JlXvaGFsSeNVxhZx3TAH5eSHXpUmSJCnpkksuSbnHtF+Dacc3CaES+B3w/hjj8lTP81BeST0VY+ThNbv4yt2reHpTDTPHj+AjLz+eK+dVk2dAlSRJyrl0h/Jm9IGsEMJ5wPTOz48x/jTTAmKM+0IIC4ErgJTBVJJ6KoTAhbPGc8Fx47jnue189Z5VvPcXT3DixFF87PLjWXD8eBIffZckSdJA020wDSHcDBwLPAW0f3grAmmDafI0M83JUFoKXAb8d6+qlaRuhBC4fG41l51QxR1Pb+br967mrT9+nPnTRvOxy4/n7Jljc12iJEmSjtDtobwhhBXAiTHLY35DCCcDPwHySSyydGuM8T/SjfFQXkl9ramljVuXbORb969m+/5GLpw1jo9dfjwnT67MdWmSJEnDSrpDeTMJpr8GPhBj3NofxXVmMJXUXxqaW7n5kQ18e+Ea9tY1c8Xcaj7yitnMqhqZ69IkSZKGhd5+xnQc8FwI4TGgsf3OGOO1fVSfJPW7ksJ83nHRTN5w1hR++PA6frBoHXc/t41XnzqJGy6bzdSxZbkuUZIkadjKZI/pxV3dH2N8sK+LcY+ppKNlT20T333wBX7yt/W0tkXecNYU3rPgOI6pLM11aZIkSUNSrw7lPZoMppKOtu37G/jW/au55bGNhACvOnUS71pwLMeOL891aZIkSUNKbz9jeg7wLeAEoIjEYka1McZRfV2owVRSrmzaW8f3H1rLLY9vpKm1jSvnVfOeBccxb1JFrkuTJEkaEnobTJcAbwB+DcwH3gzMijH+W18XajCVlGs7DzTyf39dx82PbOBAYwsXzR7Pexccy1kzxngeVEmSpF7odTCNMc4PISyLMZ6cvO9vMcbz+rpQg6mkgWJ/QzM3P7KBHz28jt21TZwxbTTvWXAsL5szwYAqSZLUA71dlbcuhFAEPBVC+BKwFRjRlwVK0kAzqqSQ915yHG87fwa3LtnITQ+t5e0/WcKc6pG8e8GxXH3SRAry83JdpiRJ0pCQyR7TacB2Ep8v/RBQAXw7xrimr4txj6mkgaq5tY3bn9rCdxau4YWdtUwbW8Y7LzqW686YRHFBfq7LkyRJGvB6vSpvCKEUmBpjXNXXxXVmMJU00LW1Re55bjvfXriGZZtqqBpVzL9cMJN/OHsqI4ozOQhFkiRpeEoXTLs9Di2E8HfAU8BdydunhhDu6NMKJWmQyMsLXDGvmtvfez4/e/vZzBxXzhf+vILz/ut+vnbPKrbvb8h1iZIkSYNOJofyLgVeBiyMMZ6WvK9jIaS+5B5TSYPREy/u5TsLX+De57ZTkAyubzlvOvOnjXahJEmSpKTeLn7UEmOs8ZcrSera6VNH8/03z2f9rlpuXryBW5ds5I/LtnLixFG85bxpXHvKJEqL/ByqJElSKpnsMf0hcB/wr8B1wAeAwhjju/q6GPeYShoK6ppa+P2TW/jpI+tZue0AFaWFvP7MKfzT2dOYOrYs1+VJkiTlRG/PY1oGfBJ4BRCAu4HPxxj7/INUBlNJQ0mMkcfW7eGnj2zgrme30RYjLzt+Am8+bzoXHjeOvDyPRJEkScNHr1flPVoMppKGqm01Dfzi0Q384rEX2XWwiZnjRvCmc6dx3RmTGVVSmOvyJEmS+l2Pgml3K+/GGK/tg9oOYzCVNNQ1trRy1/Jt/Phv63nyxX2UFeXzmtMn8eZzpzO7amSuy5MkSeo3PQ2mO4GNwC+BR0kcxtshxvhgH9dpMJU0rCzbtI+fPrKBO57eQlNLG+fOHMsbz57KK06soqTQxZIkSdLQ0tNgmg+8HHgjcDLwJ+CXMcZn+6tQg6mk4WhPbRO/enwjP1u8gc376hlZXMDVJ0/kujMme8oZSZI0ZPT6M6YhhGISAfXLwH/EGL/VtyUmGEwlDWdtbZHFa3dz2xObuXP5VuqaWpk6pozXnD6J606fzJQxrugrSZIGrx4H02QgvZpEKJ0O3AH8KMa4uR/qNJhKUlJtYwt3Ld/GbU9s4pG1u4kRzpoxhutOn8RVJ01kpAsmSZKkQaanh/L+BJgH3AncEmNc3n8lJhhMJemlNu+r5/dPbua2pZtYu6uWksI8Lp9bzXWnT+b848aR72lnJEnSINDTYNoG1CZvdn5SAGKMcVSfVonBVJLSiTHy5MZ9/PaJTfzh6a3U1DdTNaqYV502ib8/fTKzXNVXkiQNYJ7HVJKGmMaWVu5bsYPblm5i4fM7aW2LnDSpgtecPokr5lUzsaI01yVKkiQdxmAqSUPYroON3P7UFm5buonntu4H4NQplVwxr5rL51YzY9yIHFcoSZJkMJWkYWPNjoPc/ew27n52G8s21QAwp3okr5hbzRVzqzlh4khPPyNJknLCYCpJw9CmvXXc8+x27np2G4+v30OMMHVMWcee1NOmVJLnwkmSJOkoMZhK0jC380Ajf1mxnbuWb+NvL+yiuTUyYWQxl8+t5op51Zw1YwyF+Xm5LlOSJA1hBlNJUof9Dc08sHIHdy3fxsJVO6lvbqWyrJDLTqji8rnVXDhrHCWF+bkuU5IkDTEGU0lSl+qbWnlo9U7uXr6Nv6zYzv6GFooK8jh7xhgunDWOC2eNZ061n0uVJEm9ZzCVJHWrqaWNxWt38+DzO1m0eifPbz8IwPiRxVw4axwXzRrPBbPGMa68OMeVSpKkwShdMC042sVIkgamooI8Lpo9notmjwdgW00DD63eyaLVu3hg5Q5++8RmAOYeM4oLZ43nolnjOGP6aIoLPOxXkiT1Tr/tMQ0hTAF+ClQDbcBNMcYb041xj6kkDUxtbZFnt+znodU7eej5nSzdsJeWtkhpYT7nzByTCKqzx3Ps+BEe9itJkrqUk0N5QwgTgYkxxidCCCOBpcCrYozPpRpjMJWkweFgYwuLX9jNouQe1bW7agE4pqKEC2aN46wZYzlz+mimjikzqEqSJCBHh/LGGLcCW5PXD4QQVgCTgJTBVJI0OJQXF3DZiVVcdmIVABv31LFo9S4Wrd7JXcu3ceuSTQBMGFnMmdPHMH/6aM6cPoY51SMp8LQ0kiTpCEdl8aMQwnTgIWBejHF/que5x1SSBr+2tsjqHQd5fP0elqzfw+Pr97J5Xz0AI4ryOX3aaOZPG8OZ00dz6tRKyopc7kCSpOEgp4sfhRDKgduAG7oKpSGE64HrAaqqqli4cGF/lyRJOgomA5Or4VXVeeyuL2X13jae39fK6m17eHj1LiKQF2DaqDxmj85j9uh8ZlXmM6rYQ38lSRpu+nWPaQihEPgjcHeM8WvdPd89ppI0PNTUN/PEi3sTe1TX7eWpTftoamkDYOa4EZw2dTQnTRrFSZMrOXHiKEqLXPlXkqTBLleLHwXgJ8CeGOMNmYwxmErS8NTY0sryzTU8vn4vj6/bw9Ob9rHrYBMA+XmBWRPKmTepgpMnVzBvUgUnThxFSaFhVZKkwSRXwfQCYBHwDInTxQD8W4zxz6nGGEwlSQAxRrbtb2DZphqWb67p+Lq79vCwevLkCk6aVMFJkyuZUz3SsCpJ0gCWk2DaEwZTSVIqMUa21DTwzKYantm8j2c27+eZTfvYW9cMQEFeYHbVSE6aVMG8yRWcOHEks6pGMqqkMMeVS5IkMJhKkoaoGCOb99Unw+qhy75kWAWYVFnK7KpyZlePZE71SGZXjeTY8eXuXZUk6SjL6aq8kiT1lxACk0eXMXl0GVeeNBFIhNVNe+t5fvsBVm47wPPbD7Bq2wEeXrOL5tbEH2Pz8wLTx5ZxfDKotgfWaWNHkJ/nqsCSJB1tBlNJ0pASQmDKmDKmjCnj0hOqOu5vbm1j/a5aViWD6qptB3huy37uXL6N9oOHigvymFVVzuyqRFCdMW4Ex44fwZQxZRQXuIdVkqT+4qG8kqRhra6phTU7DnaE1VXbE3tZt+9v7HhOXoApY8qYMW4EM8eVM2P8CI4dN4IZ40dQPaqExEL0kiQpHQ/llSQphbKiAk6eXMnJkysPu7+mvpn1u2pZu+sg63bW8sKuWtbtrOXRtXuob27tND6f6WNHMHP8CGaOG8HM8eXMSIZWF16SJCkzBlNJkrpQUVrIKVMqOWVK5WH3t7UlTmWzblcta3ceZO2uWtburGXZphr+/MxW2jodiDRmRBFTxpQxdUwZU8eUMjV5iPHUMWVMrCj186ySJCUZTCVJykJeXuCYylKOqSzl/OPGHfZYY0srL+6u44WdtazbVcuLe+rYuKeOpzfu48/PbKW1U2otyAtMGn14WG2/TBlTRkWpe1slScOHwVSSpD5SXJDPrKrE+VOP1NLaxtaaBl7cU9cRWNu/3vnM1o7zsbarKC1kyphSjqlIhOBJyTB8TGUJkypLGVdeTJ57XCVJQ4TBVJKko6AgP69jteDzu3h8f0MzG/fUsXFP/aHQureO9btr+euaXdQ2tR72/ML8wMSKRFA9PLiWMil5X1mRb/OSpMHBdyxJkgaAUSWFzD2mgrnHVLzksRgj+xta2LKvvuOyeV9Dx/XFL+xm2/6Gwz7fClBZVkj1qBKqK0qoHlVCVfv15O3qUSVUlhW6qrAkKecMppIkDXAhBCpKC6koLeSEiaO6fE5LaxvbDzR2Cq71bN5bz7aaBrbtb2D55hp2HWx6ybjigrxEYB1VQlVFCdWjig8F2GSYHT+ymJJCz+MqSeo/BlNJkoaAgvw8JiUP6U2lqaWNHQca2L6/gW01jWzb3349EV6XbdrHPTUNNLa0vWTsqJICJowqYcLIYsaPLGbCyGImjCw5dH1UMePLSxhVWuAeWElS1gymkiQNE0UFeUweXcbk0WUpnxNjpKa+mW3JwLp9fwM7DzSy40Bjx9cnXtzLjv2NXQbYooI8xpcngmp7iB1fXsK4kUWMHVHM+JFFjCsvZmx5MSOK8g2xkiTAYCpJkjoJIVBZVkRlWRFzqrs+bBgSAfZAYws79rcH1kSA7Rxi1+2q5dF1e9h3xIrD7UoK8zpC6vjy9sB6KLiOKy9ifPJ6ZWmhqxBL0hBmMJUkSVkLITCqpJBRJYUcN6E87XObWtrYU9vEroON7DrYyO6Dh1/febCRzfsaWLapht21TYed77VdXoDRZUWMGZG4jC1Pfh1R3HG9/faYEUWMLiukID+vv16+JKmPGUwlSVK/KirI61gNuDttbZF99c3sPtjIzk4hdk9tE7trm9hzsIk9tU2s2naAPbVNLzn/a2eVZYXJsFqUDKtFjB5RxJjk19FlhYfdHlXi52MlKVcMppIkacDIywsdez9nVY3s9vktrW3srWtOBtdEgN1T28TuZIBtv3/drlqeqNvH3tomWrrYIwuQnxcSYbVTcO0ItGVFVCYfqywrpLIs8XhFqXtmJakvGEwlSdKgVZCfl1hgaWQx0H2QjTFysLGFvbXN7KlrYm9tE3vrmpJ7X5vYU9vMvuTtdbtqWbphH/vqUodZgJElBZ1CaxGVpYWJ0JoMr6PLiqho/1paSGVpIaNKC8n3M7OS1MFgKkmSho0QAiNLChlZUsjUsalXJ+6sfaGnfbXN7KtPHD68r66JfXXN7E1+3VeXvL++mQ27a9lX10xNferDjOFQoK0sTQTWirJEaK1M7omtLE0E2orkfe3PKynM85BjSUOOwVSSJCmNzgs9TSWzMAvQ2pY49U57aK2pb0rebu4Iru2P76tvZktNPTXJcNvVAlDtivLzGFVawKjSRGjt6tL+2KiSwo7QW1Fa6Cl6JA1YBlNJkqR+kN/p87LZiDFS29TasVd2f30irHYOszX1iftr6pvZfbCJtTtrE/c1NBNTZ1oK8gKjSgsZVVKQ/NoeZAsS4bvzYx2PH3puSWF+L38qktQ1g6kkSdIAEkKgvLiA8uICJo/ObmxbW+RgUws1dYeH186XAw0tHSF2f30z2/Y3dDy3saUt7fxFBXnJAFvAyJLOATcRbEd2CrxdXXePraRUDKaSJElDRF7eocOOp/RgfENz60uC6/6Glo6A2377QMOh+7fsq++43l2wzQskP+Nb0PF1VDLgdr5vZKfwm3jOoeulhYZbaSgymEqSJAmAksJ8Sgrzk6scZ6+xJRFs9yf3zO5vaO643fl64rHE45v31bMy+diBhmbSfLwWSByOXN4eZIsLKT8iuJYXdw64hwJv4v5D110VWRpYDKaSJEnqE8UF+RSX5zOuvGfBNsZIXVNrR4ht3zPbfr1zsD3YeOjxLfsaONB4IPm8lrSLR7UbUZRP+RFBtj28lncKvOXFBSmfN6KogDwDrtQnDKaSJEkaEEIIjCguYERxARMrejZHjJGG5rZOobY5GWIPhdvOwfbQYy1srWngYPKxg40tGX2/9s8Dl5ck6h5ZfGSY7ep24WHjyosLKCrI69kLloYIg6kkSZKGjBACpUX5lBblM2FUz+dpbYvUNrVwsCPIdhVqW5NB9lDAPdjYwvb9DR3h9mBjS9qVktsV5ed1hNSOgJsMu5330Caek9+xV7e8OL/jOe1jC/MNuRp8DKaSJEnSEfI7LSTVG21tkfrm1sOC68FOe3Jrk+H1QPv1hpZE4G1sZseBBmp3JT63W9vYQn1za0bfs7ggryOkHgqs+ZSXFCaCbFHBYSE48bzE/SM6Bdzy4gJKCvNcbEpHhcFUkiRJ6id5eYcOT67qxR5cgJbWNmqbWjvCbWJPbWtHuD2YDLAHmzpdT4bcXQeb2LC7riMA1zVlFnLz8wJlRfmH7ZFtD7qdQ+yIovxOIffw+w4F4HyK8g266prBVJIkSRoECvLzqCjNo6K0d3tx4dChyrWNhwJse8BNeV9TS8fe250HGhP3N7VQ19hKU2v6UwV1vIZkUO8cbhN7ag/tsT086HZ+zpHPy/f0QUOIwVSSJEkaZvrqUOV2TS1th0JsUwu1yVB7eLBtfUnobQ/Huw42Hva8pm7OidsuBA4LtmWdgmv7nt6yos4BNz95O3Ff2RGhuKwon+IC9+rmgsFUkiRJUq8UFeRRVFDE6BFFfTJfc2sbdY2tHGxqoS4ZbuuShzF3Drl1yZBb13R48N15oLEj+NY1tlLb1NLtOXLbtR++3DnodoTcw8LtEV+L8l/yeGlyntLCfE8t1I1+C6YhhB8B1wA7Yozz+uv7SJIkSRpaCvPzqCjLo6Ksb/botp9GqD2oJoLuoTB7sLGF+qbWw4JsewCuS37dfqCBul2HnnOwKbMVl9uVHRlok0G2rDD/UADuFIQ7wm1hfsfe3BHFBYfdHkp7d/tzj+mPgf8BftqP30OSJEmS0up8GiHK+2bOzmG3fUGpuuRhzHVNidudg23H107PqalvZltNfeLQ5+SYTA9jhuTe3U7B9g1nTeH6i47tmxd4lPVbMI0xPhRCmN5f80uSJElSrnQOu+PKi/ts3pbWNuqaWzv23Na3H7bc1JoMu4lDmF/ynKbWPq3jaMv5Z0xDCNcD1wNUVVWxcOHC3BYkSZIkSQNMAEYkLx2Kk5d2+2tYuHDN0Syrz+Q8mMYYbwJuApg/f35csGBBbguSJEmSJB1VebkuQJIkSZI0vBlMJUmSJEk51W/BNITwS+AR4PgQwqYQwtv763tJkiRJkgav/lyV9439NbckSZIkaejwUF5JkiRJUk4ZTCVJkiRJOWUwlSRJkiTlVIgx5rqGDiGEncCGXNeRxjhgV47nsIa+m8MarKGv57CGgVNDX8xhDQOnhr6Ywxqsoa/nsIaBU0NfzGENR8e0GOP4Lh+JMXrJ8AIsyfUc1jC0Xoc1DJwahsrrsIah9TqsYWi9DmsYODUMlddhDUPrdQyEGnJ58VBeSZIkSVJOGUwlSZIkSTllMM3OTQNgDmvouzmswRr6eg5rGDg19MUc1jBwauiLOazBGvp6DmsYODX0xRzWkGMDavEjSZIkSdLw4x5TSZIkSVJOGUwlSZIkSTllMM1QCOGKEMKqEMKaEMK/9mD8j0IIO0IIy3v4/aeEEB4IIawIITwbQvhgluNLQgiPhRCeTo7/XE/qSM6VH0J4MoTwxx6OXx9CeCaE8FQIYUkPxleGEH4TQliZ/Hmcm+X445Pfu/2yP4RwQ5ZzfCj5c1weQvhlCKEkqxeRmOODyfHPZvr9u+qjEMKYEMK9IYTVya+jsxz/2mQNbSGE+T2s4cvJf49lIYTfhRAqsxz/+eTYp0II94QQjsm2hk6PfTSEEEMI47Ks4bMhhM2d+uKqntQQQnh/clvxbAjhS1nW8KtO3399COGpbGsIIZwaQljc/v8rhHBWD+Y4JYTwSPL/6R9CCKPSjO9y25RpX6YZn3Ffppkjm75MNUdGvZlqfKfHM+nLVDVk1JvpasiiL1PVkFFvphmfcV+mmSOjvgwp3u8y7clu5simL1PNkVFfphmf8fYy1RydHk/bl2lqyHh7ma6GTPoyTQ0Zby/TzJFNX6aaI+PtZfL5h/0OlU1fphif1Xt4ijky3lamGJ/Ve3hXc3S6v9ttZZo6sn0ff0kNmfRkNzVk+z5+5Pis3sNTzJFVTw4ouT5fzWC4APnAC8BMoAh4GjgxyzkuAk4HlvewhonA6cnrI4Hns6kBCEB58noh8ChwTg9r+TDwC+CPPRy/HhjXi3+PnwD/krxeBFT28t92G4mT/WY6ZhKwDihN3r4V+Ocsv+88YDlQBhQAfwFm9aSPgC8B/5q8/q/Af2c5/gTgeGAhML+HNbwCKEhe/+8e1DCq0/UPAN/Ntobk/VOAu4EN6XosRQ2fBT6axb9hV3Nckvy3LE7enpDta+j0+FeBz/SghnuAK5PXrwIW9mCOx4GLk9ffBnw+zfgut02Z9mWa8Rn3ZZo5sunLVHNk1JupxmfZl6lqyKg304zPpi+7fa9J15tpasi4L9PMkVFfkuL9LtOe7GaObPoy1RwZ9WWa8RlvL1PNkWlfpqkho57sZo6M+jLda8ikJ7upIZu+TDVHxtvL5HMO+x0qm75MMT6r9/AUc2S8rUwxPqv38K7myLQnu6kj475MMT7jbWW615FpX6aoIav38BRzZNWTA+niHtPMnAWsiTGujTE2AbcAr8xmghjjQ8CenhYQY9waY3wief0AsIJEQMp0fIwxHkzeLExeYrZ1hBAmA1cDP8h2bF9I/tXnIuCHADHGphjjvl5MeSnwQoxxQ5bjCoDSEEIBiXC5JcvxJwCLY4x1McYW4EHg1d0NStFHryQR1kl+fVU242OMK2KMqzItPMUc9yRfB8BiYHKW4/d3ujmCbnozzf+nrwMf78X4jKWY493Af8UYG5PP2dGTGkIIAXgd8Mse1BCB9r+OVtBNb6aY43jgoeT1e4Hr0oxPtW3KqC9Tjc+mL9PMkU1fppojo97sZhudaV/2djufanw2fZm2hu56M834jPsyzRwZ9WWa97tstpVdzpFlX6aaI6O+TDM+4+1lN+/93fZlX/zukGaOjPqyuxoy2V6mmSObvkw1R8bbyxS/Q2Xcl12Nz/Y9PMUcGW8rU4zP6j08ze+SGW0ru5kjIynGZ7yt7K6GTPoyxfis3sNTzJFxTw40BtPMTAI2drq9iSx+WehrIYTpwGkk/lqXzbj85CEFO4B7Y4xZjU/6BomNRlsPxraLwD0hhKUhhOuzHDsT2An8X/KwhR+EEEb0opY30M0v/0eKMW4GvgK8CGwFamKM92T5fZcDF4UQxoYQykj8VWxKlnO0q4oxbk3WthWY0MN5+srbgDuzHRRC+EIIYSPwj8BnejD+WmBzjPHpbMd28r7k4Ug/6u5wqhRmAxeGEB4NITwYQjizh3VcCGyPMa7uwdgbgC8nf5ZfAT7RgzmWA9cmr7+WDHvziG1T1n3Z021bhnNk3JdHzpFtb3Ye39O+7OJ1ZNWbR4zvUV+m+Flm3JtHjL+BHvTlEXNk3Jcp3u+y6sm+eM/MYI60fZlqfDY92dUc2fRlmteQcU+mmCPjvuzm55hRT6aY4way6MsUc2SzvfwGL/0dKpu+7Gp8trqbo7ttZZfjs9xOvmSOHmwru6yDzPuyq/HZbitT1QCZ9WVX428gu21lV3P06D18IDCYZiZ0cV/Wexv7QgihHLgNuOGIv1B1K8bYGmM8lcRfws4KIczL8ntfA+yIMS7NZlwXzo8xng5cCbw3hHBRFmMLSBx2+J0Y42lALYlDX7IWQigi8R/311mOG03iL5wzgGOAESGEf8pmjhjjChKHy9wL3EXi8PCWtIMGgRDCJ0m8jp9nOzbG+MkY45Tk2Pdl+X3LgE/Sg0DbyXeAY4FTSfzB4as9mKMAGE3i8K6PAbcm/2qarTeS5R9MOnk38KHkz/JDJI8uyNLbSPzfXEriUMqm7gb0ZtvUF+PTzZFNX3Y1Rza92Xl88ntm3Zdd1JBVb3YxPuu+TPPvkVFvdjE+677sYo6M+7K373dHY45M+jLV+Gx6sos5TiaLvkxRQ1Y9mWKOjPuym3+LjHoyxRxZ9WWKOTLqy97+DtUXv4N1N0d3PZlufKY92dUc2b6Hp6kjo75MMz7jnszg3yNtX6YZn3FPppkj6/fwASMOgOOJB/oFOBe4u9PtTwCf6ME80+nhZ0yT4wtJHHv/4T54Tf9OFsfhJ8d8kcTe4vUkPpdZB/ysl3V8Nps6gGpgfafbFwJ/6uH3fiVwTw/GvRb4Yafbbwa+3cufw/8D3tOTPgJWAROT1ycCq7IZ3+n+hWT++ZSXzAG8BXgEKOvJ+E6PTcvk/0nnOYCTSPwFe33y0kJij3Z1D2vI6P9qF/8WdwELOt1+ARif5c+xANgOTO5hP9RAxzmqA7C/l/8es4HHuhn/km1TNn3Z1fhs+zLVHFn2ZdptbHe9eeT4HvZldzWk7c0U/xbZ9mWqn2VGvZmihqz6MoOfQ7d92em5/w58NJueTDVHtn2Zao5s+jJVDZn0ZIo5Pp1tX3ZTQ9qeTPPvkVVfpvg5ZrW97KKGrLeX3fwsUvYlKX6HyrQvU43PpifTzZFJT3ZXQyY9mWKO27LpyQzrSNmXaf4tMu7Jbn6W3fZlmhoy7skMfw4ZbysHwiXnBQyGS7LB1pLYQ9a++NHcHsyT8j9JBmMD8FPgGz0cP57kIkFAKbAIuKYXP5MF9GDxIxKfPRjZ6frfgCuynGMRcHzy+meBL/fwNdwCvLUH484GniXx2dJA4jMh7+/BPBOSX6cCK4HRPekj4MscvnDCl3rSh/QimAJXAM+l2oBnMH5Wp+vvB36T7RxHPLaebhZO6KKGiZ2ufwi4pQev413AfySvzybxEYCQzWtI/iwfzKKPjqxhBck3VhKfoV7agznaezOPxHbnbWnGdrltyrQvU43Ppi/T1JBxX6aZI6Pe7O51ZNKXaWrIqDfTjM+4L9O9jkx6M00NGfdlmjky6ktSvN9l2pPp5siyL1PVkVFfphmf8fayu9fRXV+mqSHj7WWaOTLqy3SvIZOe7KaGbPoy1RwZby87zbWAQwvVZPUefuT4bHoyTQ1ZvYd3MT7r9/BUr6O7nuymjp68j3cen9V7eKrXkWlfpqgh6/fwLubIuicHyiXnBQyWC4nPAD5P4q8nn+zB+F+SOKygmcRfN96e5fgLSBw+vAx4Knm5KovxJwNPJscvp5tVwjKYr8uNSQbjZpII9k+TCHc9+VmeCixJvpbfk2GgO2KOMmA3UNHD1/85EmFyOXAzyRXcspxjUfKN4Gng0p72ETAWuA9Ynfw6Jsvxr05ebyTxF767e1DDmuQGvL03060S2dX425I/y2XAH0gsOtPj/090HwC6quFm4JlkDXfQ6Q0uizmKSPzFcznwBPCybF8D8GPgXb3ohwuApcm+ehQ4owdzfJDE9u554L9IH6673DZl2pdpxmfcl2nmyKYvU82RUW+mGp9lX6aqIaPeTDM+m75M+TrIoDfT1JBxX6aZI6O+JMX7HdltK1PNkU1fppojo75MMz7j7WWqOTLtyzQ1ZLy9TDNHRn2Z7jVk0pPd1JBNX6aaI+PtZae5FnAoRGTclynGZ/UenmKOjLeVKcZn9R7e1RyZ9mQ3dWT1Pt7F+Iy3leleR6Z9maKGrN7DU8yRdU8OlEv7rmJJkiRJknLCxY8kSZIkSTllMJUkSZIk5ZTBVJIkSZKUUwZTSZIkSVJOGUwlSZIkSTllMJUkKY0QwsHk1+khhH/o47n/7Yjbf+vL+SVJGiwMppIkZWY6kFUwDSHkd/OUw4JpjPG8LGuSJGlIMJhKkpSZ/wIuDCE8FUL4UAghP4Tw5RDC4yGEZSGEdwKEEBaEEB4IIfyCxMneCSH8PoSwNITwbAjh+uR9/wWUJuf7efK+9r2zITn38hDCMyGE13eae2EI4TchhJUhhJ+HEEL7fCGE55K1fOWo/3QkSeqFglwXIEnSIPGvwEdjjNcAJANmTYzxzBBCMfDXEMI9yeeeBcyLMa5L3n5bjHFPCKEUeDyEcFuM8V9DCO+LMZ7axfd6DXAqcAowLjnmoeRjpwFzgS3AX4HzQwjPAa8G5sQYYwihsm9fuiRJ/cs9ppIk9cwrgDeHEJ4CHgXGArOSjz3WKZQCfCCE8DSwGJjS6XmpXAD8MsbYGmPcDjwInNlp7k0xxjbgKRKHGO8HGoAfhBBeA9T18rVJknRUGUwlSeqZALw/xnhq8jIjxti+x7S240khLAAuA86NMZ4CPAmUZDB3Ko2drrcCBTHGFhJ7aW8DXgXclcXrkCQp5wymkiRl5gAwstPtu4F3hxAKAUIIs0MII7oYVwHsjTHWhRDmAOd0eqy5ffwRHgJen/wc63jgIuCxVIWFEMqBihjjn4EbSBwGLEnSoOFnTCVJyswyoCV5SO6PgRtJHEb7RHIBop0k9lYe6S7gXSGEZcAqEofztrsJWBZCeCLG+I+d7v8dcC7wNBCBj8cYtyWDbVdGAreHEEpI7G39UI9eoSRJORJijLmuQZIkSZI0jHkoryRJkiQppwymkiRJkqScMphKkiRJknLKYCpJkiRJyimDqSRJkiQppwymkiRJkqScMphKkiRJknLq/wexj+WAEAdSTAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = [x for x, y in training_process]\n",
    "y = [y for x, y in training_process]\n",
    "plt.figure(figsize=((16,4)))\n",
    "plt.plot(x, y)\n",
    "plt.xticks(x, x)\n",
    "plt.xlabel(\"Iterations\")\n",
    "plt.ylabel(\"Mean Square Error\")\n",
    "plt.grid(axis=\"y\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-5:}}$ Plot the MSE versus iterations for $\\alpha=1, \\gamma=0.01$ and 100 iterations of SGD\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Put your figure here\n",
    "\n",
    "## **Find the graph in pdf and separately in the directory**\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-6:}}$ Show the plot for 50 iterations, $\\alpha=1, \\gamma=0.01$ of SGD without bias, i.e., $b=0$\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Paste your output and figures here or below this cell\n",
    "## Find the graph in pdf and separately in the directory\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-7:}}$ Repeat above with bias\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Paste your output and figures here or below this cell\n",
    "\n",
    "## Find the graph in pdf and separately in the directory\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-8:}}$ Show plots with values of regularization parameters to be 1, 0.1, 0.01, 0.001 and for 50 iterations, and $\\alpha=1, \\gamma=0.01$\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Paste all your plots here or below this cell.\n",
    "## Find the graph in pdf and separately in the directory\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-9:}}$ Prove that the loss function is not convex. Is the loss function differentiable?\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Write your answer here.\n",
    "## Find the answer in pdf and separately in the directory\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus Question"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-10:}}$ In the reference paper [1] above, additional bias terms are recommended, implement it\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Put your modified function here or below this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# These functions can be replaced with the functions present in the above class\n",
    "\n",
    "    def predict(self, i, j):\n",
    "            \"\"\"\n",
    "            Prediction: Predicted the rating of user i and item j\n",
    "            \"\"\"\n",
    "            b_u=np.mean(self.R[i][np.where(self.R[i] != 0)])\n",
    "            b_i=np.mean(self.R.T[j][np.where(self.R.T[j]!=0)])\n",
    "            prediction = self.b + self.P[i, :].dot(self.Q[j, :].T)+(b_u-self.b)+(b_i-self.b)\n",
    "\n",
    "            return prediction\n",
    "    \n",
    "    def full_matrix(self):\n",
    "        \"\"\"\n",
    "        The rating matrix using the biases P and Q\n",
    "        \"\"\"\n",
    "        new_values=np.zeros((self.R.shape[0],self.R.shape[1]))\n",
    "        for i in range(R.shape[0]):\n",
    "            for j in range(R.shape[1]):\n",
    "                b_u=np.mean(self.R[i][np.where(self.R[i] != 0)])\n",
    "                b_i=np.mean(self.R.T[j][np.where(self.R.T[j]!=0)])\n",
    "                new_values[i][j]=self.b+(b_u-self.b)+(b_i-self.b)\n",
    "        return new_values + self.P.dot(self.Q.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\color{red}{\\text{Question-11:}}$ Download the netflix ratings matrix, and run your algorithm.\n",
    "\n",
    "$\\color{red}{\\text{Answer:}}$ Put your MSE versus iterations here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000\n",
      "SGD Iteration: 10 ; error = 61.3745\n",
      "SGD Iteration: 20 ; error = 56.0818\n",
      "SGD Iteration: 30 ; error = 51.2782\n",
      "SGD Iteration: 40 ; error = 47.7073\n",
      "SGD Iteration: 50 ; error = 45.1576\n",
      "SGD Iteration: 60 ; error = 43.1596\n",
      "SGD Iteration: 70 ; error = 41.6426\n",
      "SGD Iteration: 80 ; error = 40.5469\n",
      "SGD Iteration: 90 ; error = 39.7352\n",
      "SGD Iteration: 100 ; error = 39.1547\n",
      "\n",
      "P x Q:\n",
      "[[2.86552247 3.10958073 4.43513636 ... 3.17506707 3.63346127 2.76967231]\n",
      " [3.50213643 2.90193726 4.09270524 ... 3.69447386 4.06737624 3.51821315]\n",
      " [3.24794478 3.08814539 4.18130004 ... 3.46544326 3.82633837 3.21392298]\n",
      " ...\n",
      " [3.5164263  3.41672043 3.84297678 ... 3.59732366 3.73627404 3.50782307]\n",
      " [3.30109185 4.28297914 3.58788608 ... 3.25488964 3.06717803 3.21295843]\n",
      " [3.54728283 3.08855718 3.97460169 ... 3.68913208 3.96603179 3.5607568 ]]\n",
      "\n",
      "Global bias:\n",
      "3.6463483146067417\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6wAAAEGCAYAAABhIxjlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABRjklEQVR4nO3deZxcVZ338c/pfd/TnU6nk84eIJBAwk5CQMQNUVQcxW3EEcVxHX1cxnkc1PHRGZ1xdJwRFUHGBQVBQEUFgQQIS8hOQhaydDrpLel938/zx7nVffv2rU5I0l3Vyff9etWrq+rce+tXvz53+dXdjLUWERERERERkXiTEOsARERERERERMKoYBUREREREZG4pIJVRERERERE4pIKVhEREREREYlLKlhFREREREQkLiXFOoDjUVRUZCsqKmIdhoiIiIiIiEyAjRs3NlhrpwXfnxIFa0VFBRs2bIh1GCIiIiIiIjIBjDEHw97XIcEiIiIiIiISl1SwioiIiIiISFxSwSoiIiIiIiJxSQWriIiIiIiIxCUVrCIiIiIiIhKXVLCKiIiIiIhIXFLBKiIiIiIiInFJBetJ6ukf5LaHd3C0vTfWoYiIiIiIiJxWVLCepM1VLfxqfRWv/e5aHtpSjbU21iGJiIiIiIicFia0YDXG5BljfmuM2WWM2WmMudQYc5sxptoYs8V7vHEiY5hol84r5JFPrqSiMJNP/XoLt/5ik/a2ioiIiIiInAITvYf1e8CfrbWLgaXATu/971prl3mPRyY4hgk3vziL+2+9jC++YTFP7D7Ctd9dy++31mhvq4iIiIiIyEmYsILVGJMDrAJ+CmCt7bPWtkzU58VaYoLho1fO45FPXsGswkw+cc9mbv3FJho6tLdVRERERETkRCRN4LTnAkeBu4wxS4GNwKe8to8bY94PbAA+a61tDo5sjLkFuAWgpKSENWvWTGCop9Ynz7L8JT2ZB3bW8cyeOt5/dioXlU5kqkVERERERE4/ZqIOWzXGrACeBy631r5gjPke0Ab8AGgALPB1oNRae/N401qxYoXdsGHDhMQ5kV6pb+dzv93G1kMtrJidz4eumMNrzy4hKVHXuhIREREREYkwxmy01q4Ivj+RldNh4LC19gXv9W+BC6y19dbaQWvtEPAT4KIJjCGmFpRkc/9HL+Wr159DfXsPt/5yE1d+ew0/eWo/rd39sQ5PREREREQkrk1YwWqtrQMOGWMWeW+9BnjZGFPqG+wGYPtExRAPkhIT+MBlFaz53FX86H3LmZmfzjce2cml33ycf35oOwcaOmMdooiIiIiISFyasEOCAYwxy4A7gBRgP/BB4PvAMtwhwZXAR6y1teNNZ6oeEhzN9upW7lpXye+31tA/NMTVi4p536WzWblgGokJJtbhiYiIiIiITKpohwRPaMF6qpxuBWvEkfYefvl8Fb94/iCNnX2U5KRyw/kzecfyMuYXZ8c6PBERERERkUmhgjWO9Q4M8uSuI/x242Ge3H2UwSHL0vI83rF8JtefN4PcjORYhygiIiIiIjJhVLBOEUfbe3loSzW/3XiYXXXtpCQm8NqzS3j78jJWLphGsq4wLCIiIiIipxkVrFPQjppWfrvxMA9tqaGps4+CzBTefF4pbzm/jPPL8zBG57uKiIiIiMjUp4J1CusbGOLpV47yu83VPPZyPb0DQ8wuzOCty8p46/llzCnKjHWIIiIiIiIiJ0wF62mivaefP2+v46EtNazb14C1sLQ8j7cum8F1581gWnZqrEMUERERERF5VVSwnobqWnv4/dYaHtxSzY6aNhIMXD6/iOuXzuB1S6aTk6aLNYmIiIiISPxTwXqae6W+nYe31vDw1hoONnaRkpTAaxYXc/3SGVy1uJi05MRYhygiIiIiIhJKBesZwlrL1sOtPLSlmt9vraWho5fs1CRet2Q6bzu/jEvmFpKQoIs1iYiIiIhI/FDBegYaGBzi+f1NPLSlmj9vr6O9d4AZuWnccEEZb7tgJvOmZcU6RBERERERERWsZ7qe/kEefbmeBzYd5qk9RxmysKw8j7cvn8mbzyslLyMl1iGKiIiIiMgZSgWrDDvS1sODW6q5f2M1u+vbSUlM4Jqzi7lxRTmrFkwjUYcMi4iIiIjIJFLBKmNYa9lR08b9mw7z8JYaGjv7KMtL528uLOedK8qZnpsW6xBFREREROQMoIJVxtU3MMRfd9Zzz/oqnn6lgQQDVy8u5t0XzeLKhdNISkyIdYgiIiIiInKailawJsUiGIk/KUkJvPHcUt54bilVjV38+sUq7tt4mL/u3EBpbho3rijnXReWMyMvPdahioiIiIjIGUJ7WCWq/sEhHt95hHvWV/HUK0cxwNWLS3jvJbNYtWCabo8jIiIiIiKnhPawyquWnJjA65dM5/VLpnOoye11/c2Lh/jrznpmFWRw08WzuHH5TAqzUmMdqoiIiIiInIa0h1Velb6BIf6yo45fPH+QFw40kZKYwBvOnc57L5nNitn5GKO9riIiIiIi8urooktyyr1S384vX6ji/o2Hae8dYPH0bG5ZNZc3L51Bsi7SJCIiIiIix0kFq0yYrr4BHt5Sw53rDrCnvoPS3DRuvnwO77qonOy05FiHJyIiIiIicU4Fq0w4ay1rdh/lR0/t4/n9TWSnJXHTxbO4+fI5lOTonq4iIiIiIhJOBatMqm2HW/jRU/v500u1JCYY3rKsjFtWzWVhSXasQxMRERERkTijglVioqqxi58+s5/fbDhET/8Qr1lczEeunMeFFbpAk4iIiIiIOCpYJaaaO/v43+cOcvdzlTR19nHBrDw+euU8rjmrRPdzFRERERE5w6lglbjQ3TfIfRsP8eOn9nO4uZt50zL5yKp5vOX8GaQmJcY6PBERERERiQEVrBJXBgaHeGR7Hbev2cfLtW2U5KRy8+VzeM8ls8lKTYp1eCIiIiIiMolUsEpcstby9CsN3L52H8/uayQ3PZkPXl7BBy+bQ26GbokjIiIiInImUMEqcW/roRZ+8OReHnu5nqzUJN536Ww+dMUcirJSYx2aiIiIiIhMIBWsMmXsrG3jv5/cyx9fqiU1KYF3XzSLW1bNpTQ3PdahiYiIiIjIBFDBKlPOvqMd/HDNPn63uZpEY3j78pl8bPU8ygsyYh2aiIiIiIicQjEpWI0xecAdwBLAAjcDu4HfABVAJfBOa23zeNNRwXpmO9TUxe1r93HfhsMMWcs7ls/k76+ar8JVREREROQ0EauC9W7gaWvtHcaYFCAD+EegyVr7LWPMF4F8a+0XxpuOClYBqGvt4fa1+/jV+iqGhixvv8AVrrMKVbiKiIiIiExlk16wGmNygK3AXOv7EGPMbmC1tbbWGFMKrLHWLhpvWipYxc9fuA4OWd5+QRkfv2qBClcRERERkSkqWsE6kTe8nAscBe4yxiwFNgKfAkqstbUAXtFaHCXgW4BbAEpKSlizZs0EhipTzeocWHpFKo8c6OeBTYf57cbDXD4jiTfPS6Y4IyHW4YmIiIiIyCkwkXtYVwDPA5dba18wxnwPaAM+Ya3N8w3XbK3NH29a2sMq46lvc3tcf/mC9riKiIiIiExF0fawTuSuqMPAYWvtC97r3wIXAPXeocB4f49MYAxyBijJSeOf33wOT3/+Kt53yWwe3FLD1f++hi/8dhuHmrpiHZ6IiIiIiJygCStYrbV1wCFjTOT81NcALwMPAx/w3vsA8NBExSBnlpKcNG673hWu771kNr/bUs1V31nDlx5Q4SoiIiIiMhVN9FWCl+Fua5MC7Ac+iCuS7wVmAVXAjdbapvGmo0OC5UTUtfbwwzV7uWf9ISyWdywv5++vmsfMfB0qLCIiIiIST2JyW5tTRQWrnIza1m7+58l9/ObFQwxZy9suKOPW1fOZU5QZ69BERERERAQVrCLUtHTz46f2c8/6KvoHh3jz0hn8/VXzWViSHevQRERERETOaCpYRTxH2nv46dMH+PnzB+nqG+R155Tw8asWcO7M3FiHJiIiIiJyRlLBKhLQ3NnHXc9W8rN1B2jrGeDKhdP4xNXzWVFREOvQRERERETOKCpYRaJo7+nn588f5I6nD9DU2cfKBUV89tpFLCvPi3VoIiIiIiJnBBWsIsfQ1TfAL54/yO1r99PU2cdrFhfzmdcuZEmZDhUWEREREZlIKlhFjlNH7wB3P1vJj5/aT2t3P687p4TPvHYhi6fnxDo0EREREZHT0gkVrMaYROAv1tprJjK4Y1HBKrHQ1tPPnc8c4KdPH6Cjb4A3nVvKp69ZyPzirFiHJiIiIiJyWolWsCaMN5K1dhDoMsbomEg54+SkJfPpaxby9Beu4mOr5/HEriNc+921/P2vNrG9ujXW4YmIiIiInPaOeUiwMeZe4BLgMaAz8r619pMTG9oI7WGVeNDY0csdzxzgF88dpL3XXVX4Y6vncdGcAowxsQ5PRERERGTKOuFzWI0xHwh731p79ymK7ZhUsEo8ae3u5xfPH+TOZw7Q2NnH8tn53HrlPF5zVrEKVxERERGRE3BSF10yxqQAC72Xu621/ac4vnGpYJV41NM/yL0bDvGjtfupbulmUUk2t66ex3XnlZKUOO7R9iIiIiIi4nMye1hXA3cDlYAByoEPWGufOuVRRqGCVeJZ/+AQv99aww/X7OOVIx3MKsjgY6vn8bYLZpKSpMJVRERERORYTqZg3QjcZK3d7b1eCNxjrV0+IZGGUMEqU8HQkOWxnfX84Im9vFTdyozcND66eh7vXFFOWnJirMMTEREREYlbJ3SVYE9ypFgFsNbuAZJPZXAip4OEBMPrzpnOwx+/nJ998EJK89L5ykM7WPlvT3LH0/vp6huIdYgiIiIiIlPK8exhvQsYAn7uvfUeIMla+8EJjm2Y9rDKVGSt5bn9jfzgib08u6+RgswUPnTFHN536Wxy0vSbj4iIiIhIxMkcEpwK/D1wBe4c1qeA/7HW9k5EoGFUsMpUt/FgEz94Yi9P7j5KdmoS77lkNjdfXkFxTlqsQxMRERERibkTKliNMQnANmvtkokM7lhUsMrpYnt1Kz9cu48/vVRLUkICb19exi2r5jGnKDPWoYmIiIiIxEy0gjVpvJGstUPGmK3GmFnW2qqJC0/kzLCkLJf/vukCKhs6+cnT+7lv42F+/eIhXn/OdD565TyWlufFOkQRERERkbhxPIcEPwFcCKwHOiPvW2uvn9jQRmgPq5yujrb38rNnD/Dz5w7S1jPApXML+ejqeaxaUIQxJtbhiYiIiIhMipM5h/XKsPettWtPUWzHpIJVTncdvQPc80IVP33mAHVtPZxdmsOtq+fxhiXTSUrUvVxFRERE5PSmc1hFpoC+gSEe3FLN7Wv3sf9oJ7MKMrhl1VzesXym7uUqIiIiIqetE7oPq7V2CNhqjJk1YZGJyLCUpATeuaKcv37mSm5/73LyM1P4pwe3c8W/Psn/rNlLW09/rEMUEREREZk0OodVJI5Za3l+fxM/XLuPp/YcJSs1ifdcPIsPXj6H6bm6JY6IiIiInB50DqvIFLe9upXb1+7jkZdqSTCG65fN4MMr53JWaU6sQxMREREROSmvumA1xiy21u7ynqdaa3t9bZdYa5+fsGgDVLCKjDjU1MVPnznAvRsO0dU3yMoFRXx45VxW6srCIiIiIjJFnUjBuslae0HwedjriaaCVWSs1q5+frn+ID9bV8mR9l4WT8/m71bO5fqlM0hJ0pWFRURERGTqOJGLLpkoz8Nei8gky81I5mOr5/PMF67mOzcuxVr43H1bueJfn+BHa/fR0TsQ6xBFRERERE7KeAWrjfI87LWIxEhKUgLvWD6TP396Jf9780UsLMnmm3/axWXffJzv/GU3jR29x56IiIiIiEgcShqnbaYx5vu4vamR53ivy45n4saYSqAdGAQGrLUrjDG3AR8GjnqD/aO19pETiF1EfIwxrFo4jVULp7H1UAu3r93Hf6/Zyx3P7OdvVpTz4VVzmZmfEeswRURERESO23jnsH5gvBGttXcfc+KuYF1hrW3wvXcb0GGt/c7xBqlzWEVOzN4jHfz4qX38bnM11sL1y2bw0SvnsbAkO9ahiYiIiIgMi3YOa9Q9rMdTkIpIfJtfnMW/vWMpn75mIXc8fYB71lfxwKZqrl5czM2Xz+Hy+YW6srCIiIiIxK3xDgk+FSzwqDHGAj+y1v7Ye//jxpj3AxuAz1prm4MjGmNuAW4BKCkpYc2aNRMcqsjpbVU2XLAylb9W9fPE/iM8sesIM7MM11Ykc0lpEimJKlxFREREJL5EPST4lEzcmBnW2hpjTDHwGPAJYDfQgCtmvw6UWmtvHm86OiRY5NTq6R/k91tr+OkzB9hV105hZgrvuWQ277tkNtOyU2MdnoiIiIicYV71fVgnIIDbCJy7aoypAP5grV0y3rgqWEUmhrWW5/Y1cue6Azy+6wjJCQm8eekM/m7lHM4qzYl1eCIiIiJyhnjV57D6RlwI/BAosdYuMcacB1xvrf2XY4yXCSRYa9u959cCXzPGlFpra73BbgC2v9ovIyKnhjGGy+YXcdn8Ig40dHLXugPct+Ew9286zKqF0/joqrlcOk/nuYqIiIhIbBxzD6sxZi3wf3DnoJ7vvbf9WHtFjTFzgd95L5OAX1lrv2GM+TmwDHdIcCXwEV8BG0p7WEUmT2tXP7944SB3raukoaOXc8tyuWXVXN6wZDpJiePdullERERE5MSc8CHBxpgXrbUXGmM2+wrWLdbaZRMT6lgqWEUmX0//IL/bXM1PntrP/oZOygvS+fDKudy4vJz0lMRYhyciIiIip5FoBevx7C5pMMbMw+0RxRjzDmDcPaIiMvWlJSfy7otm8dd/uJIfvW8507JS+cpDO7jsW4/zH4/t4Wh7b6xDFBEREZHT3PHsYZ0L/Bi4DGgGDgDvsdYenPjwHO1hFYkPGyqbuH3tPv668wgpSQm8ddkMPnTFXBZNz451aCIiIiIyhZ3QRZeMMYnArdbaa/wXUZqoIEUkvq2oKOCOigL2He3grnUH+O3Gw9y74TArFxTxdyvnsmpBkS7QJCIiIiKnzPHsYX3CWnv1JMUTSntYReJTc2cfv1pfxd3PVnKkvZcFxVl86Io5vPX8MtKSdZ6riIiIiByfk7no0r8DC4D7gM7I+9baB051kNGoYBWJb30DQ/xhWw0/efoAO2vbKMxM4T0Xz+K9l8ymOCct1uGJiIiISJw7mYL1rpC3rbX25lMV3LGoYBWZGqy1PLevkTvXHeDxXUdISjC8+bwZ3HzFHJaU5cY6PBERERGJUydcsMYDFawiU8+Bhk7ufraSezccoqtvkIsqCrj5igpee/Z0EhN0nquIiIiIjDiZPaxpwIeAc4DhY/u0h1VEjkdrdz/3bTjEXesqqW7pZmZ+On97WQXvvLCcnLTkWIcnIiIiInHgZO7D+nNgOvA6YC0wE9CVgkXkuOSmJ/N3K+ey9v+s5vb3XsCM3HT+5Y87ueT/Pc5XHtrO/qMdsQ5RREREROLU8exh3WytPd8Ys81ae54xJhn4y2ReOVh7WEVOLy8dbuWuZw/wh6219A0OsXrRND54+RzdFkdERETkDHUye1j7vb8txpglQC5QcQpjE5EzzLkzc/mPdy5j3Rev5tPXLGB7dRsfuHM91/zHWn7+/EG6+gZiHaKIiIiIxIHj2cP6d8D9wHnAXUAW8BVr7e0TH56jPawip7fegUH+uK2Wu9ZV8lJ1KzlpSbzrolm8/9LZzMzPiHV4IiIiIjLBdJVgEYl71lo2VTVz57pK/ry9Dmst1549nQ9eXsFFcwp0uLCIiIjIaSpawZp0HCN+Jex9a+3XTkVgIiIRxhiWzy5g+ewCalq6+fnzB7lnfRV/3lHH2aU5/O3lFVy/dAZpyYmxDlVEREREJsHxHBL8Wd/LNOA6YKduayMik6G7b5AHt1Rz17oD7KnvoDAzhZsunsV7L5lNSU7asScgIiIiInHvlB0SbIxJBR621r7uVAV3LCpYRcRay7P7Grlr3QEe33WEBGO49uwS3nfJbC6dV6jDhUVERESmsBM+JDhEBjD35EMSETl+xhgun1/E5fOLONjYya9eqOI3Gw7xp+11zJ2WyXsvns3bl88kNz051qGKiIiIyClyPIcEvwREBkoEpgFfs9b+YIJjG6Y9rCISpqffXV34Fy8cZHNVC2nJCbxlaRnvu3Q2S8pyYx2eiIiIiBynEz4k2Bgz2/dyAKi31k7qTRJVsIrIsWyvbuWXLxzkwc01dPcPsrQ8j/dcNIvrlpaSkXIiB5OIiIiIyGQ5mYK1YLx2a23TScZ2TCpYReR4tXb387tNh/nlC1W8cqSD7NQkbrigjJsunsXi6TmxDk9EREREQpxMwVoJlAPNgAHygCqv2VprJ/x8VhWsIvJqWWvZcLCZXz5/kEe219E3MMTy2fncdNEs3nReqW6NIyIiIhJHTqZgvR13VeBHvNdvAK6x1n523BFPIRWsInIymjv7uH/TYX71QhX7GzrJTU/mhvPLeNsFZZxblqsrDIuIiIjE2MkUrButtcsD720Im9hEUcEqIqeCtZbn9jfyqxeqeHRHPX2DQ8ydlskNy8p46/lllBdkxDpEERERkTPSyRSsfwGeBn6Bu1rwe4FVug+riExlrV39PLK9lt9trmb9AXcq/orZ+bzl/DKuO7eU/MyUGEcoIiIicuY42Ysu/TOwynvrKeCrk3GxpQgVrCIykQ43d/Hw1hoe3FzNnvoOkhMNVy4s5m8uLOeqRdNISkyIdYgiIiIip7UTLlgDE8kHWuyrGekUUMEqIpPBWsvO2nYe3FLNg5urOdLeS0lOKu9cUc47V5TrkGERERGRCfKqC1ZjzFeAe621u4wxqcCfgKXAIHCTtfavExmwnwpWEZlsA4NDPLHrCPesr2LNnqMArFwwjZsuKuc1Z5WQrL2uIiIiIqfMiRSsO4Al1lprjLkFuAl4DbAQuNtae9FEBuynglVEYqm6pZt7XzzEvRsOUdvaQ1FWKjeumMk7ls9k3rSsWIcnIiIiMuWdSMG62Vp7vvf8fuBRa+2PvNebrLUXTGTAfipYRSQeDAwOsXbPUe5Zf4gndtUzZGFpeR5vO7+MNy+dQYEu1CQiIiJyQk6kYH0e+DugHtgNLLfWHvDadllrFx/Hh1YC7bjDiAestSu8izj9BqgAKoF3Wmubx5uOClYRiTf1bT08vKWGBzZXs7O2jaQEw+pFxbztgjKuXlxMWnJirEMUERERmTJOpGC9GLgbmAb8p7X26977bwTeZ61993F8aCWwwlrb4Hvv34Ama+23jDFfBPKttV8YbzoqWEUknu2sbeN3m0cu1JSTlsSbzivlrcvKuLCigIQEE+sQRUREROLaKblK8Al8aCVjC9bdwGprba0xphRYY61dNN50VLCKyFQwOGR5dl8DD2yq5s/b6+juH6QkJ5U3nTuD65aWcn55HsaoeBUREREJilXBegBoBizwI2vtj40xLdbaPN8wzdba/JBxbwFuASgpKVn+61//esLiFBE51XoGLFuODvJC7QAvHR1kwEJhmuHi0iQump7I7JwEFa8iIiIinquuuiomBesMa22NMaYYeAz4BPDw8RSsftrDKiJTWVtPP4/uqOcP22p45pUGBoYsc4oyue68Uq47bwaLpmfHOkQRERGRmIrJHtZAALcBHcCH0SHBInKGau7s48876vj91hqe39/IkIUFxVm8ySte5xfrNjkiIiJy5jmpgtUYcxnuqr5Jkfestf97jHEygQRrbbv3/DHga7h7uTb6LrpUYK39/HjTUsEqIqejo+29/Hl7Lb/fVsuLlU1YC4unZ/PmpTO47rxSZhdmxjpEERERkUlxwgWrMebnwDxgC+72NADWWvvJY4w3F/id9zIJ+JW19hvGmELgXmAWUAXcaK1tGm9aKlhF5HRX19rDIy/V8odtNWyqagFgSVkO1549nWvPKWFRSbbOeRUREZHT1skUrDuBs+1kHTscQgWriJxJqlu6eWRbLY9sr2XLoRashfKCdK49ezqvPbuEFbPzSUpMiHWYIiIiIqfMyRSs9wGftNbWTlRwx6KCVUTOVEfae3h85xEe3VHHun2N9A0MkZ+RzNWLS7j2nBJWLZhGekpirMMUEREROSknU7A+CSwD1gO9kfettdef4hijUsEqIgIdvQM8tecoj71czxO7jtDa3U9acgIrF0zj2rNLuOasEvIzU2IdpoiIiMirFq1gTQobOOC2Ux+OiIi8WlmpSbzx3FLeeG4p/YNDvHigiUdfrufRHXU89nI9CQYumlMwfOhweUFGrEMWEREROSmTdlubk6E9rCIi0Vlr2V7dxqMv1/Hojnp217cDcHZpDlcvLmbVwmmcPyuPZJ33KiIiInHqZA4JvgT4L+AsIAVIBDqttTkTEWgYFawiIsevsqGTR192e103VbUwOGTJTk3isvmFXLmwmFULi5iZr72vIiIiEj9OpmDdALwLuA9YAbwfWGCt/ceJCDSMClYRkRPT2t3Ps3sbeOqVozy1p4Hqlm4A5k3LZNXCaaxeVMzFcwpIS9aFm0RERCR2TqpgtdauMMZss9ae5733rLX2sgmKdQwVrCIiJ89ay76jHazd08DaPUd5YX8jvQNDpCcncsWCIq5eXMxVi4qZnpsW61BFRETkDHMyF13qMsakAFuMMf8G1AKZpzpAERGZWMYY5hdnM784mw9dMYee/kGe29fIE7uO8MSuIzz2cj0A58xw575etbiYpTPzSEwwMY5cREREzlTHs4d1NlCPO3/1M0Au8D/W2r0TH56jPawiIhPLWsue+g6veK1n48FmhizkZyRz6bxCLp9fxOXziphdmIExKmBFRETk1DrhQ4K9kdOBWdba3RMR3LGoYBURmVwtXX2s3XOUp19pYN3eBmpbewAoy0vn8vmugL1sXhHTslNjHKmIiIicDk7mHNY3A98BUqy1c4wxy4CvWWuvn5BIQ6hgFRGJHWstBxo6Wbe3gXV7G3l2XwNtPQMALCrJ5vL5RaxcUMRFcwrITD2eM01ERERERjuZgnUjcDWwxlp7vvfe8AWYJoMKVhGR+DE4ZNlR08q6vY2s29vA+som+gaGSE40nD8rn5Xzi7hiQRHnluWSpHu/ioiIyHE4mYL1BWvtxcaYzSpYRUQkqKd/kA2VzTy99yjPvNLAjpo2ALLTkrhsXiGXzi3ksvlFLCjO0vmvIiIiEupkrhK83RhzE5BojFkAfBJ49lQHKCIiU1Oad1ucKxYUwRugsaOXZ/e5va9Pv9LAX3a4qw8XZaVwydxCLvWK2DlFmSpgRUREZFzHs4c1A/gycC1ggL8AX7fW9kx8eI72sIqITF2Hmrp4bl8jz+1357/Wt/UCMD0njUvnFXLZvEKuWFBEaW56jCMVERGRWDmpqwTHmgpWEZHTQ+QCTs/tb+S5fY08v7+Rho4+AOZOy+QK7+rDl84rJDc9OcbRioiIyGR51QWrMebh8SaoqwSLiMjJstayu76dZ7zb57xwoImuvkESDJw7M48r5hdyydxCzp+VT5auQCwiInLaOpGC9ShwCLgHeAF3OPAwa+3aCYgzlApWEZEzQ9/AEFsPtwwXsJsPtTA4ZEkwcPaMHFbMLuDCigJWVORTkpMW63BFRETkFDmRgjUReC3wbuA84I/APdbaHRMZaBgVrCIiZ6aO3gE2VzXzYmUzGyqb2FzVQnf/IACzCjJYUZHPhRUFXFiRz7xpugqxiIjIVHVS57AaY1Jxheu3ga9Za//r1IcYnQpWEREB6B8cYmdt23AB+2JlMw0d7iJO+RnJLJ9dwEVz8llRUcCSGbmkJOk+sCIiIlPBCRWsXqH6JlyxWgE8DNxpra2eoDhDqWAVEZEw1loONnaxvrKJDZVNbKhsZn9DJwCpSQksK89jRUU+y8rzWVqeS3G2DiMWERGJRydySPDdwBLgT8CvrbXbJzbE6FSwiojI8Tra3svGg02sP9DMhoNN7KhpY3DIretm5KaxtDyPZeV5LC3P49yyXDJ1MScREZGYO5GCdQjo9F76BzKAtdbmnPIoo1DBKiIiJ6q7b5AdNa1sOdTC1sOtbD3UQlVTFwAJBhYUZ7O8Ip+LvIs5zczPiHHEIiIiZx7dh1VERMTT1NnH1kMtbDnUwuZDLWw+2Ex77wDg9sKu8C7kdOGcAhYWZ5OQoIs5iYiITKRoBauOgxIRkTNOQWYKVy0u5qrFxQAMDll21bWxobKZ9ZVNPL+/kYe31gCQk5bE0vI8ls7M8/7mUqxb6oiIiEwK7WEVEREJsNZyqKmbFyub2HCwiS2HWtlT3z58Luz0nDSWludy3kx3Pux5M3PJTkuOcdQiIiJTl/awioiIHCdjDLMKM5hVmMHbl88ERs6FjZwHu+1wC3/ZUe8NDwuLs7lgdh7nl+dzwew85hZl6VBiERGRk6SCVURE5DikpySyoqKAFRUFw++1dPWx9XArW6pa2FTVzB+31XLP+kOAO5R42ax8Lpjl9sKeMyOXadmpsQpfRERkSprwgtUYkwhsAKqttdcZY24DPgwc9Qb5R2vtIxMdh4iIyKmWl5HClQunceXCaQAMDVn2N3SwqaqFzVXNbDrYwvcef4XI2TclOamcMyOXc2bkeI9cZuanY4z2xIqIiISZjD2snwJ2Av7b4HzXWvudSfhsERGRSZOQYJhfnM384mzeuaIcgPaefrZXt7GjppWXa9rYXtPKmt1H8E6HJTc9mXNm5HBuWS5LynI5tyyX2YUZKmJFRESY4ILVGDMTeBPwDeAfJvKzRERE4lF2WjKXzivk0nmFw+/19A+yq66d7dWt7Khxxexd6yrpGxwC3OHEkeL13Jm5nFeWR3mB9sSKiMiZZ6L3sP4n8HkgO/D+x40x78cdKvxZa21zcERjzC3ALQAlJSWsWbNmYiMVERGZZDOBmQXwugIYODuN6o4hDrQOUdk2xMGjzazf38iAtyc2Oxnm5iUyLy+BebmJzMlNICNZBayIiJzeJuy2NsaY64A3Wms/ZoxZDXzOO4e1BGgALPB1oNRae/N409JtbURE5EzUNzDEnvp2th1uZcuhZjZXtbD3aAfWuisTLyjOYll5HufPyuecGTksKM4mPSUx1mGLiIi8atFuazORBes3gfcBA0Aa7hzWB6y17/UNUwH8wVq7ZLxpqWAVERFx2nr62Xaolc1VzWw+5C7u1NzVD0CCgYqiTM6ansPi6dksLnV/dWEnERGJd5NesAY+fDUje1hLrbW13vufAS621r5rvPFVsIqIiISz1lLV1MXO2nZ21bWxy/t7sKlr+OrEWalJLJ6ezdne1YnPLs1lQUkWacnaGysiIvEhWsEai/uw/psxZhnukOBK4CMxiEFEROS0YIxhdmEmswszef2S6cPvd/YOsKe+nV117eysbWNnbRsPbKrmf587CEBigmH+tCzOnpHD2aU5nFWaw6Lp2bpXrIiIxJVJ2cN6srSHVURE5OQNDVkONXfxck0bL9e2saOmjZdr2qhr6xkepjAzhcWl2SwqcYcTL5qezcISnRsrIiITK572sIqIiEgMJCSM7I19w7mlw+83dvSyq87tjd1d18buunZ+tf4gPf3uNjvGQEVhJmeX5gzvkT17Rg7F2ak6N1ZERCaUClYREZEzXGFWKpfPT+Xy+UXD7w0OuXNjd9e1DR9W/FJ1K398qXZkvMyUUQXsounZzC3KIiUpIRZfQ0RETkMqWEVERGSMxATDnKJM5hRl8volI3tj23r62VXbzss1rbxc6w4tvmtdJX2Dbm9scqJh3rSs4asUL5qezVnTcyjJ0d5YERF59VSwioiIyHHLSUvmojkFXDSnYPi9/sEh9h3tYHddOztr3WHF6w808eCWmuFhctOTWViSxfzibBYUZ7GgJIsFxdkqZEVEZFy66JKIiIhMiNaufnbXe7fbqWtnb30He4600+LdNxYgOzWJecVZw0XswhJ3kafS3DQVsiIiZxBddElEREQmVW7G2L2x1loaO/vYe6SDV450sLe+nVeOdLBmz1Hu23h4eLjs1CTml2SxsDibBSVZw1cr1oWeRETOLCpYRUREZNIYYyjKSqUoK5VL5haOamvp6mNPfQe769t5pb6dPfXtPLaznt9sODQ8THZaEguK3Z7Y+cVZLChxhxhrj6yIyOlJhwSLiIhIXGvo6GVPfbvbK1vfMfy8sbNveJgs79Diud6FovyPzFT9Pi8iEu90SLCIiIhMSZE9spfNKxr1fmNH7/Chxa/Ut7P3aAfrDzTxu83Vo4YryUn1itcs5k3LZF5xFvOnZTEjL53EBO2VFRGJZypYRUREZEoqzEqlMCuViwOHFnf3DXKwqZMDRzvZ39DJAe/xp+21oy74lJqUwJwiV8DOm+aK2TlFmVQUZZKTljzZX0dEREKoYBUREZHTSnpKIoun57B4es6YtqbOPvYd7WDfkQ7392gn26tb+dNLtQz5zpIqzEyhoiiT2YUZzCl0RWykmM3SIcYiIpNGS1wRERE5YxRkplCQWcCFFQWj3u/pH+RgYxcHGjqpbOyk0vv77N5GHtg0+hDjoqxU5hRlUOEvZAszqSjKICNFm1YiIqeSlqoiIiJyxktLTmTR9GwWTc8e09bVN8DBxi4qGzo5EClmG7pYs+coR3234gEozk5lVkEGswoyKC/IYHZhxvDrabolj4jIq6aCVURERGQcGSlJnFWaw1mlYw8x7ugdGN4bW9nQycHGLqqaunhufyO/21KN/2YMackJzCpwe2bnTMscPtR4blGmilkRkShUsIqIiIicoKzUJJaU5bKkLHdMW0//INUt3VQ1dXGoqYuqxi4qG7vY39DJmt1H6RscGh42MyWRCu8c2YrCDGYXeocZF2rPrIic2VSwioiIiEyAtORE7+rDWWPaBocsNS3dw1cwjpw7u726lT9vr2PQdwWo9OREZhe6PbOzC92hxuXeYcZleemkJCVM5tcSEZlUKlhFREREJllighkuPFctnDaqrX9wiJqWbiobuzjY6M6XPdjYyStH2nli15FRe2aNgek5aW5a+RmUF6QzM98VsmV56UzPTVNBKyJTmgpWERERkTiSnJjA7MJMZhdmAqOL2aEhS317D1WNXRxq7uaQd7jxoeYu1u1toL69Z9R5s8a4C0HNyEtnRl46M/PSKct3xezM/AzK8tN1mx4RiWtaQomIiIhMEQkJhtLcdEpz07k4pL2nf5C61h5qWro53NJNTUs31c3d1LR283JNG4+9XE/fwNCocfIykpnpL2J9Re2MvHTyM5J1Dq2IxIwKVhEREZHTRFryyMWbwgwNWRo6e6lu7uZwczfVLd0cbu7icHM3+4928tSeBrr7B0eNk56cyIy8NMryMyjLS6M0N53i7FSKslKZlp1KUXYqRVkppCYlTsZXFJEzjApWERERkTNEQoKhODuN4uw0zp+VP6bdWktTZx81LT1UR/bQ+v6+XNNKQ0df6LRz05MpykphWnYqZXne/WcL04fvSTstS1c7FpFXTwWriIiIiABgjKEwK5XCrFTOnTn2Vj3gDjtu7OzjaHsvDe29HO3w/e3opb6tl3V7G7i/rWfUeGnJCZTnu0J2hndBqOk5aZTmplGS6/5mpGjTVERG01JBRERERI5bWnLi8FWIx9PTP8jh5m4ONY/ch/ZQcxdVTd1srGqmpat/zDg5aUlMz02jJCeNwswUirziuSjLPXev3XNd/VjkzKCCVUREREROubTkROYXZzG/eOx9aAG6+wapb+uhtrWHurZu6lp7qWvtpra1hyPtvVQ2dtLQ3jfmnFpwVz8uyU6jLD991AWjZuaPXDAqLVnn1IqcDlSwioiIiMikS08Z/wJREV19AzS099HQ6Q49bujoo76tZ/iCUZuqmvnjtloGhuyo8bJSkyjITKEgM4VC729BlntemOkuGFWck8q0rFTyM1JISND5tSLxSAWriIiIiMStjJQkZhUmMaswI+owg0OW+rYe78rHXdS09NDQ0UtTZx9NnX3Utvawo6aNps4++gaHxoyfnGgoykqlODuVadlpFOd4V0GOHIrsXRW5KCuFrNQkXTxKZBKpYBURERGRKS0xwTDDu28sFEQdzlpLR+8AjR19HO3o5UhbL0fa3SHIR9t7OdLey+HmLjZXNdPU1Ye1Y6eRmpRAUVYqBZkp5GUkk5uePPI3PYXcjGTy0pMpyEyhJMcVv7rlj8iJU8EqIiIiImcEYwzZaclkpyUf81DkgcEhmrr63OHI3hWQ3aOPhvZemrv6aOnup7q5m5buflq6+hgKKXABCjJTKM5OHb4ycnFOGiU5qRRkpJCXkUJ+ZjL5Ga4AVnErMtqEF6zGmERgA1Btrb3OGFMA/AaoACqBd1prmyc6DhERERGR45WUmDB8z9rjMTRk6egboLWrn9bufhq8Pbh1bT3UDz962VHTRkNHb+jeW4CMlMTh4jUvY/Re28jrHO95QWYK+Rkp5Gckk5SoqybL6Wky9rB+CtgJ5Hivvwg8bq39ljHmi97rL0xCHCIiIiIiEyIhwZCTlkxOWjLlxxi2f3CIxo4+mru8R2e/22Pb1UdzV+S522tb19pGa3c/LV39Yy4s5Zebnjx8cal870JT+Zkpw4VubvroIjgvI1lXUpYpYUILVmPMTOBNwDeAf/Defguw2nt+N7AGFawiIiIicoZITkxwhwfnHt/eW3Dn33b2DdLiFbOt3f00dbqCN1L8Nnb20dTRx6GmLrYcaqGlq4/+wehFbkpSgiuy05PISUsmOy2JnPRkr/B2z6dlRy5GlUpxdhoFmSkk6orKMokmeg/rfwKfB7J975VYa2sBrLW1xpjiCY5BRERERGRKM8aQlZpEVmoSM/OPbxxrLV19g8Pn2LZ29XvP+2npdq/begZo6+mnrbuf9p4Bqlu6aet27/UNjL2icmKCoTAzhWleEZuZmkRGciIZKYmkpySRPvzc/c1NTyY/M4WCDLfHNydNV1mWV2fCClZjzHXAEWvtRmPM6hMY/xbgFoCSkhLWrFlzSuMTERERETnTZHiPGf4XYyQBSfQOWtp6LS29ltbI3z5La+8grb2dVNZ20DNo6RuE3kFL7wAMRN+hC0CigcxkQ3YKZCUbslIMGUmGjGT3vntuyEhyrzO95xnJhpREFbpnImOjnfF9shM25pvA+4ABIA13DusDwIXAam/vaimwxlq7aLxprVixwm7YsGFC4hQRERERkVNjYHCIrv5BevoG6ewbpLW7n2bv0OWm4b/uvabOPlq73eHNbT39dPUNjjvt1KQEctOTyUl3txHKTXeHLmekJpGZkkhmahKZKUlkpCa6v957uekjF6hKT9F5u/HKGLPRWrsi+P6E7WG11n4J+JL34auBz1lr32uM+TbwAeBb3t+HJioGERERERGZPEmJCeQkunNjX62+gaHhw5NdETswUtB2+993f4+097D3yABdfQN09g7S3T9+wQuu6HX30E2hIDOZvIwUslKSSE9JJC05kbTkBNKTE32vE8lOTRq+UnOkUNZVmSdPLO7D+i3gXmPMh4Aq4MYYxCAiIiIiInEkJSmBoqxUirJST2j8wSFLd/8gXb0DdPYN0tk7QEfvwPAVl5u8C1Y1dborMjd19lHb0kZn3wDdfYP09A/RNzj2vN0ww0VsRvLwBauyUt1f9zyJ7LRkstKSyE5NItM7/zgrNYnMVLfnNzUpQefzHodJKVittWtwVwPGWtsIvGYyPldERERERM4MiQkjF6Y6UYNDlp5+t7fWFbGDtPd6e3q9wre1e2D4olUt3l7fxoYuOnrdxao6egei3mfXLynBkJXmDmNOTUogJfJIDHmemEBSoiEp0Xue4J4nJxqSEhLITE0cfUujDHeLo4yUxClfFMdiD6uIiIiIiEjcSUww7lzYkyh6I7cg6ugZoL2nn/beATq9R0fvyJ5f/3u9A4P0Dbg9vH0D7tHROzD8vG9wiIFBy8DQEP2DloHBIfqHLP2DQ+MWxylJCRRkpFCWn879t152wt8pllSwioiIiIiInCL+WxC9mnvtnqjBIUtH74C7kFWXuxdvU1ff8Ovmzj4MU3cvqwpWERERERGRKSoxwQxfDKqCzFiHc8rp8lYiIiIiIiISl1SwioiIiIiISFxSwSoiIiIiIiJxSQWriIiIiIiIxCUVrCIiIiIiIhKXVLCKiIiIiIhIXFLBKiIiIiIiInFJBauIiIiIiIjEJWOtjXUMx2SMOQocjHUcx1AENMRxezzEoBjjoz0eYlCMk9MeDzEoxslpj4cYFOPktMdDDIpxctrjIQbFODnt8RJDrM221k4b8661Vo9T8AA2xHN7PMSgGOOjPR5iUIyKMZ5iUIyKMZ5iUIyKMZ5iUIxnTozx/NAhwSIiIiIiIhKXVLCKiIiIiIhIXFLBeur8OM7b4yEGxRgf7fEQg2KcnPZ4iEExTk57PMSgGCenPR5iUIyT0x4PMSjGyWmPlxji0pS46JKIiIiIiIicebSHVUREREREROKSClYRERERERGJT7G+TPFUfwCvB3YDe4EvhrTfCRwBtoe0lQNPAjuBHcCnQoZJA9YDW71hvholjkRgM/CHkLZK4CVgCyGXtAbygN8Cu7xYLg20L/LGjTzagE8HhvmMF9924B4gLdD+Ka9tB/DpsLwABcBjwCtANXA00H6jN/4QsCLKNL7tfY9t3vcOTuPrXtsW7zMaovxvPgfYkPFv88bbAjQCLcHxgU94fWKH91nBGH/jy2U70B9oXwY877U3AE2B9qXAc97/9DHgKQJ9yJfLA974uwPt/lxuCBk/ksedXg6C4/vzuBZYF5xGIOcW2BOYRiSXO7w8VAXH93K5D+j08u0fP5LHHUA30BNo9+dxa5TvGcnldqDZy+nwfObL416glZE+/NWQPG4nMJ/68viS938ITt+fx7/i5uHQeR34opfHYAyRPG718rQ/OL6Xxz1AF64/+seP5HEr0Ovl0t/uz+NGry34Pf198ve4Zcrw8ojR8/ZjQGGgPThvJwba/fP177zp+dv9eXwUmEnI8pCR+boo5DMiedziPa4LToPR8/a3A+P75+tK76+/3Z/HDcAlgfZgDg8SWG4H8hj5P/nb/XmsCRnfn8fOkPGDeTwUnEZILl8OTMOfxz5fLjaE5LEPt3zxj+/P44D3Pf3twTyGfU9/Lv8EPIhv/RbI45PAQ4F2fx5XE1g/BvL4h5Dxg3lcHJxGSB4fDkzDn8eXgGeC4/vyuNP76x/fn8cq3LrK3x7M49Uh3zOSx9245V/kO7Xh1uORPFbiluEvBdr9edztiyfSHsnjbi++4Pj+PD6DW/aNmoYvj//q5TE4jUged+L60r7g+F4e93vt9YHxI3ncieuv3YF2fx5fwi1ng9/T3x+3e9Ma3lZidH/cHdLuz+N/ENjWYnR/3BYyvj+PkXyP2VZjpC/+U8hnRPK4xft7MDgNRvpjLd62lW98f39s8vLob/fnscr7f/jbg8vHz+PbpgxZPu7CLZv87f48fidk/GAeg+MH87jL3x6Sx38M+Qx/Hg/j5p1R0wjk8WhgfH8eGxlZD3w6ZPlYhevv/vZgHnOC27/x+oh5AFP5gdvg2QfMBVJwG3FnB4ZZBVxAeFFUClzgPc/GLeiC4xsgy3ueDLwAXBIyrX8AfkX0grVonO9xN/B33vMUIO8Y37kOd2PfyHtluMIo3Xt9L/C3vvYl3kybASThNsxvCuYF+De8oh+4HfhZoP0sXPG8BrdROya3wLVAkvf8lyHTyPE9/x5uBR0sOMuBv3jf82rGFqyfi/a/Ba7yvl+q9/r6aP9/r/3XwA8D03gUeIP3/PO4jQl/+4vAld7zzwA/CfahSC69PvZ93Mrc3x7J5bPAe0LGv9b7X5V6OQyO78/jPwG/DevHjPwoU4srEvzTuA23YA+dD3y5nO3lsDg4fd989HPgK4Hx/Xl8DyMbs/5hXgSuxM1nt+JWSMPzmS+Pxpv+vwba/X1yZXA+9eXR4DY0guP78/hJ4I6weZ2RPlnl5dE/jUgeQ5cVvjymAlleHscsS7zxv+99T//4/jy+EXg65DP8ffJmb5zh5RGj5+0vAo8H2oPz9j8E2v3z9b+GjB/M47MEloe+HB70chj8jNvw5u2wZSpj5+2vBD/DN+6/4wol//jBPO4NtAdz2EJguR3IYzPw/UC7P481IeP789gaMn4wj+3BaQRyOQAsDLQN55GQdQ+j+2MlsHic9U0r8K3Ae8E89oR8hj+X64Dfe89TcD+m+PO4EfhToN2fxz8QWD8G8vhSyPjBPO4OTiOQxw5GNigj0/Dnccw6OpDHu4HPBKfvi2EH8GBg/GAe60I+I9gnv45vG4Cx8/W/BtpHzdfecP724HwdHD+Yx9uD04gyb/unMZxH37T87cH5ujg4/cB8/ZXA+ME8rgn5jMh6pgxXgHzTG+Ze4G8ZWc+U4Yq5fw+0+9fX1QS2tRhZz5ThlhvB8XO812W4Iucn/vZADg97eQx+xm249Uzo9p4vj3O89lnBz/DF0Ap8LTD+o8AbvPY64KlAu78v/l/cD6/+bcoFvjwu8abx74H2SB5fxC1/g+NH8rjE+z8Fx4/kcQlu+XqHvz2QxxrcDwfBz4jkMWy7eIEvj+d77bODn+GLoQE3T/rHj+RxCW75+lSgfcw8HW35G28PHRJ8ci4C9lpr91tr+3DFx1v8A1hrn8ItgMaw1tZaazd5z9txnbssMIy11nZ4L5O9h/UPY4yZCbwJN/O8KsaYHFzh9VPv8/qstS3jjPIaYJ+19mDg/SQg3RiThJsBa3xtZwHPW2u7rLUDuD1yMxmbl7fgVrwAXwVW+huttTuttbt9r8fk1lr7qPcZ4IrRvEB7m+9lHW5jJ+i7uEKxB7dRGCrK//ZW3AZWrzfMwyHDAGCMMcDluF/MRk0ayPGeH8KtoPwW4RZC4L7j5d5n+fvQW4C7rbW1wDeBt/rbfbnsw21MEWh/1Fo74I3/EDAz0O7P4yBu4R7Wj7+L+4W5N0r7ePNBJJcHrbWbrLVHoswndbi9IPcE2v15HMIVqcHPWIRbMVrchunbGT2fRfJogZ8Ab/W3B/pkt/fX3x7JowWexvV7f7s/j5m4ve2jpuHL4+e97xHWPt6yYrhPWms7rLVHwsb33ID7Rdvf7s9jLq5PBj/D3ydfwvVJ//LIP28/iityh9sDeSwmsDwLzNevAOcE2v15LAVmMXZ5GMmh9YaJusyMskwdzqPXfmXY+N58/W5crvzt/jzOwf144G/35/Ax3HI0yJ/HDtwG8sgHBJaRQYE89gIzAu3B/hjsHxGRXJ6IUctI3EbXGF4eM4EHAk3B/jjAWIuAp7z1WzkwD0at394C3O21T8P96Dzc7stjIm7D8aeB9kettQPe+MV4y3hfuz+PBbj+FraO/S5uXZcK/CKkfbx19K3At7xxVwH/Oc74i4D/E2j353E6rj8GPyPYJ9/O6G0Af3+8G7d8HG6P0h/97f7++Dxu+ehvj9Yfg9sh/nk7rD3I3x5cZx8JG9/rj+/ELR/97cH+WBPyGf489gM3BLaV/HnsBq73twfymEhgWyuQxwFgVqDdn8dEIDFkW82fw/G254jSHumPfV57xzjjZwIPB9r9eUwGjgTa/Tk8CiQHtilv8OXxLNyPmtf52315zMDtBOgKtEfyeBZuD+X0QHskj2fhtsv6A5/vz2MS7kfyYIz4phHcLr7Bl8f5XvvBccZPA/430B7J41m4vdSHA+1h8/SUoIL15JQxsvEG7pepsijDjssYU4FbMb4Q0pZojNmC+0XpMWttcJj/ZPTGbJAFHjXGbDTG3BJom4ub+e8yxmw2xtxhjMkcJ9R34RbYIxO3thp3eEUVbk9aq7X2Ud8g24FVxphCY0wGbiOrPGTaJV6BhPe3aJw4jsfNuF92RzHGfMMYcwi31+0/Am3XA9XW2q3jTPfjxphtxpg7GVnARiwEVhpjXjDGrDXGXDjOdFbiDj+qDLz/aeDbXozfwf1q6Lcdt+cW3CEu5V7sFYz0oWAui8frYyHj+90M/CnYHsjjV4LTCMtlyGcM59IYkx9oH5PLKDGuBOqtta8E2oN5/FJIDP5cvhO3MPfPZ8N59N6fR/T5MOEY8+nNwBXB9kAebwtOIySPT4Z8RiSPdxljXgq0B/O4O0qMkf54X6A9mMcvh3xPfx4jRZh/eeTP4z/ifuSItrz6LOMvz74G/CDY7svjx4AP+NtDcviNKJ/xcWPMNtyPC18PtA/nEXco78+ixLgStyHxiUD7pxmdxw8F2oPzdRJjl9v+PA4AFVGW6zD+ch9cgbI82B7oj83BaQRyaYH7Qj4jksdC4PFAuz+PJcAzUWJciesntwfag3lsCvmekVzO9V4vCKzfInmci/vBqyLK+i/dy0G09WPk6KryYLsvj+/F7c0ZNY1IHnGH6Q8B/xXyGR/HHWGTBfwy0L7Qy9E6L88PR4nxHbgfJ74caPfn8VthMRK+rvFvA4xZzxCyjRAQrf1m3FEJo9rD1jP+YaKss4OfMWo9E2gPW2eHxTi8ngm0f5qQ9UxgmO3A9d620gu49Yx/W6nEuh9uq3Hr+/mEb0v14X7YCN3W8savA94cbDfGfAP3o0A3bl033B7I4SDw31E+4+O4/1F1SHukPz6A62+Ho3yHebgCdE2g/dO4Q3KfxxXVbwi0+/tiBZAbsk0Z6Y/bgQuBkijbnJ3A+SHjR2zHHRL/dLDdy+P3gfOA7/rbA3nsBy6N8hkfB/4FuNEYMzfQHsnjPwE3GWOuiRJjGm4d0RRoj+Tx+950/jXQHrr9OCXYONjNO1UfuH/2Hb7X7wP+K2S4CqIcEuq1Z+EOS3rbMT4vD7exusT33nXA/3jPVxN+eNoM728x7rDlVb62FbgNn4u9198jyiECuBVzA26h4H8/H3gC90t1Mu58ofcGhvkQsAn3y87tuF+hRuUFaAmM0xqWN0YfXhSaW+DLuHPdouYet2L5z0g77le3F4Bc73Ul3mEZvnFKcAvTBNxG772B9u24BYXB7YE/ME6MP8RtnAfz8H3g7d7zd+Kdu+NrX4zbS7UR+GfcIT6j+lBILpvD+hgjh2CG9kFfHqP2US+PX/UPEyWXswMxBnP5v4H2YC4ro8QYyWMwB8E8/jVkmLBc5uHNZ1HyONwepU+GtUfyaMLa/XkMTOO8kDwWBWIM5vHOQHtYnwyL8YfAZ4PfISyPIcNE8rgXd859I77lUSSPeMsrL4/D7b4YXgLui7Y8w/WR/d53CWu/DrfB89VIO2P7Yj1wZ/AzfHl8M64/3Bloj+TxOtyRDQeixPAIsC5k+t/H/Zp9He6Qsb8G2oN9sSm43MbXH3F7R5sJX66vAV4/znL/y7gNTxPW7uuP/x4Sgz+Xh3D90d/u74/f8/Lob/f3xzd6eQyL8YeMzA/+8YP98amQYSK53InbAG+1vvUbI/0xsv5r87f7YtjIOOtHXF8eitbuvfeDkGG+HcmjF4MFrg3EGMnjhd74vw+0R/IY+Q41Xk6DMf7Wy8HFgfH9efxi2PcgfPk4vA1A+PIxbBthjRdntG2IyPIxtD2wnhkehvD1TGkgxuDy8WeB9rDlY1iMkfVMSmD8sPVMcJhIHrfgll9N+LaVGOmPkW2pFkK2pXDbAi8SZVvL+58dHac9H3ca27d97e8P5LAK94PdqGn48ljg5flXgfZIHvO9GA9GieGnuKNkgtOPLB/zvWmtDbQH+2IHY7cp/cvHD+Hmi+H2QH/8WnD8QH/cFK3dG+Z+XEHt//xgX/xUSIz+/vgHXD/xt/v747/giv+wGH+IOzovOH1/f/wh7jxqf/uYeTo4r8XrI+YBTOUH7heYv/hefwn4UshwFUQvmpJxGy//cJyf+c+MPs/qm4ycuF2H+7X2F+OMf1tg/OlApe/1SuCPUcZ9C/BoyPs3Aj/1vX4/XhEdZTr/D7cXZFRecIemlnrPS3EL1lddsOL2rjyHW5mNl/vZ3mdGCtZzcXuNKr3HAO6XxF1Rxq/wj++992dgte/1PkLOYcX9MlaPOwQqmIdWRu6RbHDnkUX7DgtxK4dRfSiQy3LcL4pj+piXy4vD+qAvjznj9VEvj9v9w0TJZTfwz1GmMd/7nv7vMJxL3HzSBfzfKHmsCMlBMI9tx/geC4H1/vmMsX1yd5T5cA1enwy2+/KYEW0+9ucxMEzkXB1/Hqtw823YNIb7ku87hPXJaYEYh/tjMMawPEb7HrjlUR1uJTu8PIrk0Wuvwf36PGZ5hdvIiRx1MKrdy2O19xjT7vv8WtyeiEj7/YEcDnl5PPQqpvGLSB4ZWeb2e7H6Y0zCzWu1wRgjefSNPxT2+cG+6L2+jfH7422M3x+H2wnvj6PGj9Ifb2P8/hg2jQpG+mPkO0Trj/4Yw/pjZPzx+mMwD9O9XEfm65XAHxnpj9Nx/WC3v903/rO4QyrHrB+9PG4ADoa1+95bAfQFhnncl8dDuII1ksdgDJHvsD3wHSL9cbo3nUge/TEm4QqYQ8EYA3mcDgwd43ssxJ1W8ajvvWB/rCZ8G2GNl4cx2xCMXl+HbmP4+6N/GMLXM0fxziMNmUYFbhnj/w7B/lgLPBkYz7++fktg/LD1zHjbSvcx0h/fj/vRI9Ifb8QVgrv97b7xdwAP+V4Pt3t53A38LKzd9/m/9vWl9zNyxE4kh4O4YnD6ONP4TWAa/8NIf7wRV5RG+qM/xiRcMX5PMEZGlo+R8dvCPj/K8jGyTRlt+fj/gI+Ns3wcbid8+Thq/ODy0Wv/FFGWjeNMoyIwjY8RffnojzFs+RgZP3T5GOXzR+Ux3h86JPjkvIg71GiOMSYFdwjIw8c7sjHG4GbMndba/4gyzDRjTJ73PB24BndlMgCstV+y1s601lZ4n/+Etfa9vvEzjTHZkee4k8q3+8avAw4ZYxZ5b70Gd2W0MO8m/FCeKuASY0yG951eg/tl2/89ir2/s3B74MKm8zBuYYH397EocURljHk98AXc4TddIe0LfC+vxy0MALDWvmStLbbWVnj5PIzbGzLgG7/UN/4NeOdF+jyIu1ATxpiFuF9aw85hvQZXCB8OaavBnR+HN63KwHeI5DIBd9hIH2P70MPAB7z/x0Mh7X5fCbb784jbQxBsD+Yx2z+MP5e48/V6cOeCftU3jVLvr8H1h5pAjA8CV3vt9+JWpP8SiD0yP3wt5DsG89gf8j2KfX+/ijv80D+fRfI4DfgI8FDYfIgrqLO8aQ23+/L4QVxfCLb783gTXn/0DbPZl8cLve90AW6lFJlGJI/TcPPo9kCMkTxOM8Ys9+LoCHyHa7zP7gjGGMjjDbg9BMHvUey1fxn36+2tjF4ePQx8wFr7JdwvwN8lZHmF27vxpuDyzJfHZdbaspD2BeCWh7gV88O+9rcH5uvIRkR5YBqlvmn8G+7QNn+MDwJXe+1X44rNvwl8h2tw5y2VhiyTa3AXu/gSbvm2OfD5/vn6NtyeoOByO9IfM4EP4/rjmOU67tf7jOD4vjy+C/crf7Dd3x/fgdtj7h/mRV9/PAdXoFyA+7EpMo1S3zjv8t7zx/ggrj9mGmPOx/XHrsB3uAa3bG0NyYG/P76RkXnG/z0i/TFyvvaD3uvI+i3SH+sY2Qvjb4/oA+qD60dfHt8AVIW0+/N4KdAaGGaTr0+W437gudGLJzKNSH+M/AB0MBDjg7j+WIfbS5Ph/fV/h2u855Uh63h/Hs8BOkO+R3Bd08zodXdwnd3G+IcDj9qGCFlfB9uD65ld/mGirLOfZuR80LB1NoEYH2T0OjsXdySHn399HdwOCq5nXgn5HpH+eAh4LXBnYFspkscq770/RtmW6gGWBLe1fHn8JHBhSHskj1VerK/42h8I5LDey2NbYBqlvmms8t7zxxjJY6Q9lZH+GPkO1+DWH+eFbC9G8hjJwd7A5wf74i+91/5tSn9//Bhu+RhtmzM/OH5guycrpD24vt7la//fkL54rbW2LjANf398H2555Y8xkkeMMZfglo/pge8QWTf3heTA3x/fhvtf+z8/mMfbmSpiXTFP9QduhbkHt9L8ckj7Pbhf7PpxHfhDvrYrcL+sbmPkMtVvDIx/Hm7DZhtuZf2VcWJZzdjD0+biDpPaivt1LizGZbhfirfhZpb8kGEycIcD5Ub57K/iZqDtuCu2pgban8atJLfiFkBj8oJ3vhNugV/nPfztN3jPe3EL1dqQaezFrRS24FaunYH2+70Yt+E2turD/jdezB0hMfwcd9jiNu+9+kB7Cm5vynbcoRqPh/3/cRujH42Shytwh2tsxS3wjwbaP4Xrc3u8zxrTh3y5jPx6vyPQHslln9feHmiP5PEVr70x0O7P47qwGEL6eTCGSC73Mvr2GJH2SC73e+37gtP38vjtKDnw53FHlGEiuazE/S9HzWe+PB70crQj0O7PY783jL89ksdduI3yxkC7P49rfM/HzOu4ZUFvSAyRPEY28F8OtEfy+IoXw77g9L08fp2QZU0gj9sYuey/fxh/n/wWgUN2GT1vP447rMzfHpy3/xJo98/XW3ArWX+7P4+/x11LYLg9kMdKvKvKBqbhn7cfxv1C728PzttXBz/Dy+NHw5bJgTy+ACwPtPtz+ENCltu+PB5gpK/52/157McVEP72SB5fxh3x0BBo9+fxcUbfwujLgTzOxfX7YAyRPO7C9ccdgfZIHncz0h9HTd/L4z9FyYE/j1u8fAWH8efyTgLrN0b3x8g5yf52fx4bGbmlS6Td3x8j54T724P98bXBGAK5rPGm5Z+Gvz+uYWT+i7T7++NOL45R02dkPbMsJAfB/vjukGH8efwOgW2AQB6fxP04628Pztd9gXZ/HrfiCjJ/ezCP84IxBPJ4MCQGfx7/GNLuz+Nm73+dG5huJI9jtoNC8nh5yDD+PD5DYFspkMf93nD+dn8eO7yHv92fxzrcto+/3Z/H3d7w0bbVKnE/2AVj9Odxtxerv92fx1pc4TnqM3x5HLO9GMjjYdwyzt8eXMeM2qYM6Y8t3mf42/157PPy6G/357ED11f87f48NjKy7HlNlPXMcyEx+vM4ZhqBPHZ40xn1Gb48huXAn8c2XH/yt49ZV4fNS/H4iOw2FhEREREREYkrOiRYRERERERE4pIKVhEREREREYlLKlhFREREREQkLqlgFRERERERkbikglVERERERETikgpWERGRE2SMidy7tsIYc9MpnvY/Bl4/eyqnLyIiMhWoYBURETl5FbibyR83Y0ziMQYZVbBaay97lTGJiIhMeSpYRURETt63gJXGmC3GmM8YYxKNMd82xrxojNlmjPkIgDFmtTHmSWPMr3A3kMcY86AxZqMxZocx5hbvvW8B6d70fum9F9mba7xpbzfGvGSM+RvftNcYY35rjNlljPmlMcZEpmeMedmL5TuTnh0REZETlBTrAERERE4DXwQ+Z629DsArPFuttRcaY1KBdcaYR71hLwKWWGsPeK9vttY2GWPSgReNMfdba79ojPm4tXZZyGe9DVgGLAWKvHGe8trOB84BaoB1wOXGmJeBG4DF1lprjMk7tV9dRERk4mgPq4iIyKl3LfB+Y8wW4AWgEFjgta33FasAnzTGbAWeB8p9w0VzBXCPtXbQWlsPrAUu9E37sLV2CNiCO1S5DegB7jDGvA3oOsnvJiIiMmlUsIqIiJx6BviEtXaZ95hjrY3sYe0cHsiY1cA1wKXW2qXAZiDtOKYdTa/v+SCQZK0dwO3VvR94K/DnV/E9REREYkoFq4iIyMlrB7J9r/8C3GqMSQYwxiw0xmSGjJcLNFtru4wxi4FLfG39kfEDngL+xjtPdhqwClgfLTBjTBaQa619BPg07nBiERGRKUHnsIqIiJy8bcCAd2jvz4Dv4Q7H3eRd+Ogobu9m0J+BjxpjtgG7cYcFR/wY2GaM2WStfY/v/d8BlwJbAQt83lpb5xW8YbKBh4wxabi9s585oW8oIiISA8ZaG+sYRERERERERMbQIcEiIiIiIiISl1SwioiIiIiISFxSwSoiIiIiIiJxSQWriIiIiIiIxCUVrCIiIiIiIhKXVLCKiIiIiIhIXFLBKiIiIiIiInHp/wOO75DcMpnlmwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "with open('./combined_data_1.txt') as f:\n",
    "    i=0\n",
    "    arr=np.zeros((10000,1000))\n",
    "    x=0\n",
    "    while i<1000000:\n",
    "        i+=1\n",
    "        lines=f.readline()\n",
    "        line=lines.split(',')\n",
    "        if len(line)>1:\n",
    "#             print(line[1])\n",
    "            if (int)(line[0])<10000:\n",
    "                arr[(int)(line[0])][x]=(int)(line[1])\n",
    "        else:\n",
    "            x+=1\n",
    "    lf = LF(arr, K=2, alpha=0.01, gamma=0.001, iterations=100)\n",
    "    training_process = lf.train()\n",
    "    print()\n",
    "    print(\"P x Q:\")\n",
    "    print(lf.full_matrix())\n",
    "    print()\n",
    "    print(\"Global bias:\")\n",
    "    print(lf.b)\n",
    "    print()\n",
    "    x = [x for x, y in training_process]\n",
    "    y = [y for x, y in training_process]\n",
    "    plt.figure(figsize=((16,4)))\n",
    "    plt.plot(x, y)\n",
    "    plt.xticks(x, x)\n",
    "    plt.xlabel(\"Iterations\")\n",
    "    plt.ylabel(\"Mean Square Error\")\n",
    "    plt.grid(axis=\"y\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thats All Folks! Further Readings or Infomation \n",
    "https://www.coursera.org/learn/networks-illustrated/lecture/8GPZT/netflix-timeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keep Recommending!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}