notebooks/02_model/rbm_deep_dive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "source": [
    "# RBM Deep Dive with Tensorflow \n",
    "\n",
    "In this notebook we provide a complete walkthrough of the Restricted Boltzmann Machine (RBM) algorithm with applications to recommender systems. In particular, we use as a case study the [movielens dataset](https://movielens.org), comprising user's ranking of movies on a scale of 1 to 5. A quickstart version of this notebook can be found [here](../00_quick_start/rbm_movielens.ipynb).  \n",
    "\n",
    "### Overview \n",
    "\n",
    "A Restricted Boltzmann Machine (RBM) is a generative neural network model typically used to perform unsupervised learning. The main task of an RBM is to learn the joint probability distribution $P(v,h)$, where $v$ are the visible units and $h$ the hidden ones. The hidden units represent latent variables while the visible units are clamped on the input data. Once the joint distribution is learnt, new examples are generated by sampling from it.    \n",
    "\n",
    "The implementation presented here is based on the article by Ruslan Salakhutdinov, Andriy Mnih and Geoffrey Hinton [Restricted Boltzmann Machines for Collaborative Filtering](https://www.cs.toronto.edu/~rsalakhu/papers/rbmcf.pdf) with the exception that here we use multinomial units instead of the one-hot encoded used in the paper.  \n",
    "\n",
    "### Advantages of RBM: \n",
    "\n",
    "The model generates ratings for a user/movie pair using a collaborative filtering based approach. While matrix factorization methods learn how to reproduce an instance of the user/item affinity matrix, the RBM learns its underlying probability distribution. This has several advantages: \n",
    "\n",
    "- Generalizability : the model generalize well to new examples as long as they do not differ much in probability\n",
    "- Stability in time: if the recommendation task is time-stationary, the model does not need to be trained often to accomodate new ratings/users. \n",
    "- The tensorflow implementation presented here allows fast, scalable  training on GPU \n",
    "\n",
    "### Outline \n",
    "\n",
    "This notebook is organized as follows:\n",
    "\n",
    "1. RBM Theory \n",
    "2. Tensorflow implementation and model parameters  \n",
    "3. Data preparation and inspection\n",
    "4. Model application, performance and analysis  \n",
    "\n",
    "Sections 1 and 2 require basic knowledge of linear algebra, probability theory and tensorflow while  \n",
    "sections 3 and 4 only require some basic data science understanding. **Feel free to jump to the section you are most interested in!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 Global Settings and Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44) \n",
      "[GCC 7.3.0]\n",
      "Pandas version: 0.23.4\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "\n",
    "# set the environment path to find Recommenders\n",
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import papermill \n",
    "\n",
    "#RBM \n",
    "from reco_utils.recommender.rbm.rbm import RBM\n",
    "from reco_utils.dataset.python_splitters import numpy_stratified_split\n",
    "from reco_utils.dataset.sparse import AffinityMatrix\n",
    "\n",
    "#Evaluation libraries\n",
    "from reco_utils.dataset import movielens \n",
    "\n",
    "from reco_utils.evaluation.python_evaluation import (\n",
    "    map_at_k,\n",
    "    ndcg_at_k,\n",
    "    precision_at_k,\n",
    "    recall_at_k,\n",
    ")\n",
    "\n",
    "from reco_utils.evaluation.parameter_sweep import generate_param_grid\n",
    "#For interactive mode only\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Pandas version: {}\".format(pd.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. RBM Theory \n",
    "\n",
    "## 1.1 Overview and main differences with other recommender algorithms\n",
    "\n",
    "A Restricted Boltzmann Machine (RBM) is an undirected graphical model originally devised to study the statistical mechanics (or physics) of magnetic systems. Statistical mechanics (SM) provides a probabilistic description of complex systems made of a huge number of constituents (typically $\\sim 10^{23}$); instead of looking at a particular instance of the system, the aim of SM is to describe their **typical** behaviour. This approach has been succesfull for the description of gases, liquids, complex materials (e.g. semiconductors) and even the famous [Higgs boson](https://en.wikipedia.org/wiki/Higgs_boson)!\n",
    "\n",
    "Being designed to handle and organize a large amount of data, SM finds ideal applications in modern learning algorithms. In the context of **recommender systems**, the idea is to learn typical user behaviour instead of particular instances. To better understand this consider the most general setup of a recommendation problem: there are $m$ users rating $n$ items according to some scale (e.g. 1 to 5). In a typical scenario of online shopping, streaming services or decision processes, the user only rates a subset $l \\ll m$ of the products. If we now create a matrix representation of this problem, we obtain the user/item affinity matrix $X$. In a more readable table form, $X$ will look like this:\n",
    "\n",
    "\n",
    "|  $X$   |$i_1$  |$i_2$  |$i_3$  |  ... |$i_m$  | \n",
    "|-----|-------|-------|-------|------|-------|\n",
    "|$u_1$|5      |0      |2      |0 ... |1      |\n",
    "|$u_2$|0      |0      |3      |4 ... |0      |\n",
    "|...  |...    |...    |...    |...   |...    |\n",
    "|$u_m$|3      |3      |0      |5...  |2      |\n",
    "\n",
    "\n",
    "where the zeroes denote unrated items. In a nutshell, the recommender task is to \"fill in\" the missing ratings (later we will see that in practice this is not the only criteria used to recommend a product). The classical approach to this problem is called matrix factorization: the basic idea is to decompose $X$ into a user ($P$) and item ($Q$) matrix, such that $X = Q^T P$. The dimensions of the two matrices are $dim(Q) = (f, n)$ and $dim(P)= (f,m)$ where $f \\le m,n$ is the number of latent factors, e.g. the genre of a movie, the type of food etc... and it is an hyperparameter of the model, for more details see the [ALS notebook](../02_model/als_deep_dive.ipynb). By learning $Q$ and $P$ we try to reproduce a particular instance of $X$ (provided by the available data) and use this information to fill up the missing matrix elements. \n",
    "\n",
    "The RBM approach is to look at $X$ as a particular realization (sample) of a more general process; instead of learning a specific $X$, we try to learn the matrix distribution from which $X$ has been sampled from. Effectively, we learn the typical distribution of *tastes* (i.e. latent factors) and use this information to *generate* new ratings. For this reason, this class of neural network models is also called **generative**. Consider the following example: imagine you are given the  income distribution per age window of a particular country (this is easy to find from goverments data), then we could fix the age window and *generate* virtual citizens with various incomes by sampling from this distibution.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Model \n",
    "\n",
    "The central quantity of every SM model is the [Boltzmann distribution ](https://en.wikipedia.org/wiki/Boltzmann_distribution); this can be seen as the least biased probability distribution on a given probability space $\\Sigma$ and can be obtained using a maximum entropy principle on the space of distributions over $\\Sigma$. Its typical form is: \n",
    "\n",
    "$$P = \\frac{1}{Z} \\, e^{- \\beta \\, H},$$ \n",
    "\n",
    "where $Z$ is a normalization constant known as the partition function, $\\beta$ is a noise parameter with units of inverse energy and $H$ is the Hamiltonian, or energy function of the system. For this reason, this class of models is also known as *energy based* in computer science. In physics, $\\beta$ is the inverse temperature of the system in units of Boltzmann's constant, but here we will effectively rescale it inside $H$, so that this is now a pure number. $H$ describes the behaviour of two sets of stochastic vectors, typically called $v_i$ (visibles) and $h_j$ (hidden). The former constitute both the input *and* the ouput of the algo (this will be clear later), while the hidden units are the latent factors we want to learn. This structure results in the following Neural Network topology:\n",
    "\n",
    "![rbm1](https://recodatasets.blob.core.windows.net/images/RBM1.png)\n",
    "\n",
    "The input of the movielens database consists of ratings from 1 to 5; we shall thus consider a discrete configuration space of $m$ visible variables, each taking values in a finite set $\\chi_v = \\{ 1, 2, 3,4,5 \\}$. A global configuration of the system is determined by $\\mathbf{v} = (v_1, v_2, ..., v_m) \\in \\chi_v^m$ and we reserve $0$ for an unrated movie. We also need to specify the hidden units, that we take as random binary variables $\\chi_h = \\{0,1 \\}$ denoting if the particular unit is active or not and $\\mathbf{h} = (h_1, h_2, ...,h_n) \\in \\chi_h^n$. The hidden units may describe attributes such as the genre of a movie; for example, given a sci-fi/horror movie, only the hidden units describing such attributes should be active. The minimal model for such a system is defined by the following Hamiltonian: \n",
    "\n",
    "$$H = - \\sum_{i,j \\in G} v_i \\, w_{ij} \\, h_j - \\sum_{i=1}^m v_i \\, a_i - \\sum_{j=1}^n h_i \\, b_i$$\n",
    "\n",
    "The first term is an \"interaction term\", capturing the correlations between the visible and hidden units, while the other two terms are \"potential terms\", taking into account the bias of the units. The correlation matrix $w_{ij}$ and the two biases $a_i$ and $b_i$ are learning parameters to be fixed by the minimization of a properly defined cost function. Remember that this is an unsupervised problem, i.e. there is no real output and therefore we cannot directly minimize the error function between the prediction and the labeled data. As in every SM problem, the right quantity to minimize is the Free energy (remember that $\\beta =1$)\n",
    "\n",
    "$$ F =- \\log Z =- \\log \\sum_{ v_i, h_i } P(v, h) $$.\n",
    "\n",
    "In the language of probability theory, the above quantity is the cumulant generating function. One way of evaluating the free energy is to use a [Markov-chain Montecarlo sampling](https://en.wikipedia.org/wiki/Monte_Carlo_method#Computer_graphics) algorithm such as the Metropolis-Hasting; here we will use instead an approximate method called Contrastive divergence, based on [Gibbs sampling](https://en.wikipedia.org/wiki/Gibbs_sampling) (see below). The latter has the advantage of being faster than Montecarlo. Once the candidate $F$ has been found, we fix the learning parameters by minimizing $F$. Let us see how this works in practice in the next section. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Learning Algorithm \n",
    "\n",
    "Instead of sampling directly from the joint probability distribution, one can evaluate the conditional distributions   \n",
    "\n",
    "$$ P(v, h) = P(v|h) P(h) = P(h|v) P(v) $$ \n",
    "\n",
    "where the second equality follows from the fact that the model is undirected or, in physical terms, it is in equilibrium. Gibbs sampling essentially consists of two steps called **positive** and **negative** phases:\n",
    "\n",
    "### Positive \n",
    "\n",
    "**Fix the visible units on the data and evaluate $P(h_j =1| \\mathbf{v})$**, i.e. the probability that the jth hidden unit is active given the entire input vector. In practice, it is convenient to evaluate the generating function: \n",
    "\n",
    "$$ Z[v,b] = \\prod_j \\sum_{h_j = 0,1}  e^{(\\sum_i w_{ij} v_i + b_j) h_j} = \\prod_j \\left( 1+  e^{\\sum_i w_{ij} v_i + b_j} \\right).$$\n",
    "\n",
    "Taking the gradients with respect to the bias we obtain \n",
    "\n",
    "$$\\frac{\\partial}{\\partial b_j}\\log Z[v,b] =  \\frac{1}{1+ e^{-(\\sum_i w_{ij} v_i + b_j)}} = \\sigma( \\phi_j(v, b) ),$$\n",
    "\n",
    "where $\\phi_j(v,b) = \\sum_i w_{ij} v_i + b_j $ and we have identified the logistic function $\\sigma(.) \\equiv P(h_j=1|v,b)$. \n",
    "\n",
    "**Use $\\sigma$ to sample the value of $h_j$** \n",
    "\n",
    "### Negative \n",
    "\n",
    "**Use the sampled value of the hidden units to evaluate $P(v_i = q |h)$**, where $q=1,...,5$. This is given by the multinomial expression\n",
    "\n",
    "$$ P(v_i = q |h,a) =  \\prod_{v_i=1}^q e^{v_i (\\sum_j w_{ij} \\, h_j + a_i ) }/Z_q $$,\n",
    "\n",
    "where $Z_q$ is the partition function evaluated over the $q$ outcomes (note that $0$ should not be included in the sum). Finally, sample the values of $v_i$ from the above distribution. Clearly, these new $v_i$ are not necessarily those we have used as an input, at least not at the beginning of training. The above steps are repeated $k$ times, where $k$ is usually increased during training according to a given protocol. \n",
    "\n",
    "At the end of each k-step Gibbs sampling, we evaluate the difference between the initial free energy at $k=0$ (given v) and the one after k-steps \n",
    "\n",
    "$$ \\Delta F = F_0 - F_k, $$\n",
    "\n",
    "and update the learning parameters $w_{ij}$, $b_i$ and $a_i$: \n",
    "\n",
    "$$ \\frac{\\partial}{\\partial b_j} \\Delta F = \\frac{\\partial}{\\partial b_j} (\\log Z_0[v,b] - \\log Z_k[v,b]) = P_0(h_j=1|v,b) - P_k(h_j=1|v,b) $$\n",
    "\n",
    "$$ \\frac{\\partial}{\\partial w_{ij} } \\Delta F = v_i \\, P_0(v_i = q|h, a) - v_i P_k(v_i| h,a) \\equiv \\langle v_i\\rangle_0 - \\langle v_i \\rangle_k. $$\n",
    "\n",
    "This process is repeated for each training epoch, eventually until $\\Delta F =0$, i.e. the learned distribution faithfully reproduces the empirical one. In this sense, $v_i$ serves both as an input and output of the model. As $w_{ij}$ contains informations on how users' votes are correlated, we can use this information to generate ratings for the unseen movies by sampling from the learned, marginal distribution:\n",
    "\n",
    "$$ \\langle v_i \\rangle = \\sum_{v_i} v_i \\, P(v) $$ \n",
    "\n",
    "The entire workflow is summarised below \n",
    "\n",
    "![gibbs](https://recodatasets.blob.core.windows.net/images/Gsampling.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. TensorFlow implemetation and model parameters \n",
    "\n",
    "In this section we briefly describe how the algorithm is implemented in Tensorflow and which parameters can be customized by the user during training. We also discuss some best practices to be used when training the RBM model on a recommendation task. Further technical details are explained directly in the code. \n",
    "\n",
    "Tensorflow (TF) is an open source framework to develop deep learning (DL) models in a fast and efficient way. One of the shared characteristics of DL frameworks is autodifferentiation, i.e. the symbolic evaluation of gradients, that will be particulary useful here. The other advantage of TF is the generation and optimization of the symbolic operations defined on a computational graph, for fast and scalable deployment on both CPU and GPU. For more informations on TF see [here](www.tensorflow.com). Unfortunately, TF is tailor made for supervised learning tasks, so its application to unsupervised model needs some more work. Note: although TF has recently started developing a [set of libraries to perform probabilistic inference](www.tensorflow_probability.com), we found their performance still not optimal and therefore we will not use them here. \n",
    "\n",
    "\n",
    "The RBM model is instantiated as a class with several methods to build the graph, perform sampling, training and inference. The skeleton of the graph is built at the moment the class is instantiated; mandatory fields are: \n",
    "\n",
    "- `hidden_units` integer (Default =500) : number of hidden units\n",
    "\n",
    "\n",
    "- `training_epoch`integer (Default = 20): number of training epochs \n",
    "\n",
    "\n",
    "- `minibatch_size`integer (Default = 100): size of the batch to be chosen at random at each training epoch \n",
    "\n",
    "\n",
    "The optional parameters are: \n",
    "\n",
    "- `keep_prob` : float (Default = 0.7) we use dropout regularization on the hidden units, so this parameter specifies the probability of keeping the connection to a hidden unit active. Dropout will affect specific matrix elements of $w_{ij}$, decreasing in this way the model's complexity and improving generalization. \n",
    "\n",
    "\n",
    "- `sampling_protocol` : Array (Default = $[50, 70, 80,90,100]$) percentage of the entire training epochs when the the k-sampling step is increased in an annealing fashion. In the default case, the first 50% of the training epochs are sampled with a single k-step. As training converges, the number of k-steps is increased by $1$ at each percentage.\n",
    "\n",
    "\n",
    "- `debug`: Boolean (Default = False) if True, prints the output of some of the intermediate steps for inspection. \n",
    "\n",
    "\n",
    "- `with_metrics`: Boolean (Default= False) if True it evaluates, print and finally plot the mean squared root error per training epoch on the training set. At the end, it also evaluates and print the total model accuracy both on the training and test set. We suggest to switch it off only for benchmarking execution time.  \n",
    "\n",
    "\n",
    "- `init_stdv`: float (Default = 0.1) standard deviation used to inititialize the correlation matrix. \n",
    "\n",
    "\n",
    "- `learning_rate`: float (Default = 0.004) init learning rate used in the optimization algorithm. Note that the optimizer uses a different, effective learning rate scaled to the batch size $\\alpha$ = `learning_rate/minibatch_size`. \n",
    "\n",
    "\n",
    "- `display_epoch `: integer (Default = 10) the number of epochs after which the rmse error is printed out during the learning phase. \n",
    "\n",
    "\n",
    "Although optional, it is likely that `sampling_protocol` needs to be modified for different recommenders; we recommend to keep this in mind when training on a new dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 Data preparation and inspection \n",
    "\n",
    "The Movielens dataset comes in different sizes, denoting the number of available ratings. The number of users and rated movies also changes across the different dataset. The data are imported in a pandas dataframe including the **user ID**, the **item ID**, the **ratings** and a **timestamp** denoting when a particular user rated a particular item. Although this last feature could be explicitely included, it will not be considered here. The underlying assumption of this choice is that user's tastes are weakly time dependent, i.e. a user's taste typically chage on time scales (usually years) much longer than the typical recommendation time scale (e.g. hours/days). As a consequence, the joint probability distribution we want to learn can be safely considered as time dependent. Nevertheless, timestamps could be used as *contextual variables*, e.g. recommend a certain movie during the weekend and another during weekdays.  \n",
    "\n",
    "Below, we first load the different movielens data in pandas dataframes, explain how the user/affinity matrix is built and how the train/test set is generated. As this procedure is common to all the datasets considered here, we explain it in details only for the 1m dataset.  \n",
    "\n",
    "We start with downloading the different datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>movieID</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID  movieID  rating  timestamp\n",
       "0     196      242       3  881250949\n",
       "1     186      302       3  891717742\n",
       "2      22      377       1  878887116\n",
       "3     244       51       2  880606923\n",
       "4     166      346       1  886397596"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MOVIELENS_DATA_SIZE = '100k'\n",
    "\n",
    "mldf_100k = movielens.load_pandas_df(\n",
    "    size=MOVIELENS_DATA_SIZE,\n",
    "    header=['userID','movieID','rating','timestamp']\n",
    ")\n",
    "\n",
    "# Convert the float precision to 32-bit in order to reduce memory consumption \n",
    "mldf_100k.loc[:, 'rating'] = mldf_100k['rating'].astype(np.int32) \n",
    "\n",
    "mldf_100k.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>movieID</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID  movieID  rating  timestamp\n",
       "0       1     1193       5  978300760\n",
       "1       1      661       3  978302109\n",
       "2       1      914       3  978301968\n",
       "3       1     3408       4  978300275\n",
       "4       1     2355       5  978824291"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MOVIELENS_DATA_SIZE = '1m'\n",
    "\n",
    "mldf_1m = movielens.load_pandas_df(\n",
    "    size=MOVIELENS_DATA_SIZE,\n",
    "    header=['userID','movieID','rating','timestamp']\n",
    ")\n",
    "\n",
    "# Convert the float precision to 32-bit in order to reduce memory consumption \n",
    "mldf_1m.loc[:, 'rating'] = mldf_1m['rating'].astype(np.int32) \n",
    "\n",
    "mldf_1m.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Split the data using the stratified splitter  \n",
    "\n",
    "As a second step, we split the data into train and test set by mantaining the same matrix size. Clearly, the two matrices will contain different ratings in different proportions.  \n",
    "\n",
    "- First, we use the `AffinityMatrix` class to generate the $(m,n)$ user/affinity matrix $X$ defined in section **1.1**; this also returns the sparseness percentage. For example, for the 1m dataset, $95$ % of the matrix entries are zeros. This represents a challenge for the learning task: fixing $95$ % of entries with only $5$ % of data points. \n",
    "\n",
    "- Second, use the `numpy_stratified_split()` to split $X$ into train and test set. By default, we choose a $75$% to $25$% ratio. The split function selects, for every user, $25$ % of rated movies and it moves them in the new test matrix. This way of splitting the data makes sure the rating distribution remains the same across the train/test set, both locally (user-wise) and globally. If you consider the user/item matrix $X$ defined above, we would have\n",
    "\n",
    "### Train\n",
    "\n",
    "|  $X_{tr}$   |$i_1$  |$i_2$  |$i_3$  |  $...$ |$i_n$  |    \n",
    "|-----|-------|-------|-------|--------|-------|\n",
    "|$u_1$|$0$    |$0$    |$2$    |$0...$  |$0$    |\n",
    "|$u_2$|$0$    |$0$    |$3$    |$0...$  |$0$    |\n",
    "|$...$|$...$  |$...$  |$...$  |$...$   |$...$  |\n",
    "|$u_m$|$3$    |$0$    |$0$    |$0...$  |$2$    |\n",
    "\n",
    "\n",
    "### Test \n",
    "\n",
    "| $X_{tst}$    |$i_1$  |$i_2$  |$i_3$  |  ... |$i_n$  | \n",
    "|-----|-------|-------|-------|------|-------|\n",
    "|$u_1$|5      |0      |0      |0 ... |1      |\n",
    "|$u_2$|0      |0      |0      |4 ... |0      |\n",
    "|...  |...    |...    |...    |...   |...    |\n",
    "|$u_m$|0      |3      |0      |5...  |0      |\n",
    "\n",
    "The Train and Test matrices have exactly the same dimensions (i.e. same numbers of users and movies) but contain different ratings. Once the model is trained, at inference time, we use the test set user vectors to obtain the inferred values for the ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#to use standard names across the analysis \n",
    "header = {\n",
    "        \"col_user\": \"userID\",\n",
    "        \"col_item\": \"movieID\",\n",
    "        \"col_rating\": \"rating\",\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating the user/item affinity matrix...\n",
      "Matrix generated, sparseness percentage: 95\n"
     ]
    }
   ],
   "source": [
    "#instantiate the splitter \n",
    "am1m = AffinityMatrix(DF = mldf_1m, **header)\n",
    "\n",
    "#obtain the sparse matrix \n",
    "X1m = am1m.gen_affinity_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we split the matrix above into train and test set sparse matrices "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xtr_1m, Xtst_1m = numpy_stratified_split(X1m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is useful to inspect the distribution of ratings in the test/train matrix to make sure that the splitter keeps it constant. We can inspect this by plotting the normalized histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(0,0.5,'density'), Text(0.5,0,'ratings')]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAFNCAYAAACuWnPfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFjJJREFUeJzt3X+wpXV9H/D3R1ZqBYxJ2FoL6KKhtZhatSu2Y6qNUQeqgokasSE1GTJEKy0ZzTRrG5lq2lRx1DZT2orRpiYStCY227LG/JDUMVVkQSQiYSQEBdQIVfkhKq58+sc91NN1Wc4u59nvufe+XjN3eH6e+3mYnc+87/f5nuep7g4AAOM8aHQBAACbnUAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhAxrpVVYdV1Z1V9ajRtQDAAyGQccjMwtO9P/dU1dfn1n/iQD+vu7/d3Ud29+emqBdgX5bdy+Y+92NVdcYya2X92DK6ADaP7j7y3uWquiHJz3T3H9zX8VW1pbv3HIraABZ1oL0MFmGEjJVRVf+6qt5TVb9ZVXckOaOq/t7sr8avVtUXqupXqurBs+O3VFVX1bbZ+m/M9n+gqu6oqo9W1fEDLwnYhGbTKV5bVddX1a1V9e6qevhs3xFVdVFVfXnW1y6tqu+tqjcneUqSX52NtL157FVwqAlkrJofTXJhku9J8p4ke5Kck+ToJE9LcnKSn93P+f8oyWuTfF+SzyX5pSmLBdiHn0/ynCQ/lOTYJN9K8tbZvp/J2t2pY7LW185Ocnd3vzrJZVkbbTtyts4mIpCxaj7S3f+ju+/p7q9392XdfWl37+nu65NckOQZ+zn/fd29u7u/leTdSZ54SKoG+I6XJ9nR3Z/v7m8keV2Sl1RVZS2cbU3y2Flfu6y7vzayWFaDOWSsmhvnV6rqcUnenOTvJHlo1v7NXrqf8784t3xXkiPv60CAZZuFruOS7Kqqntv1oCTfn+QdSf5qkvdV1ZFJ3pXktd397UNeLCvFCBmrpvdaf1uSTyX5ge5+WJJzk9QhrwpgAd3dSW5O8szufvjcz0O6+9bu/mZ3n9vdj0vy9CQvTnL6vaePqpvxBDJW3VFJbkvytar6m9n//DGAVfCfk7yhqo5Lkqr6K1X1/Nnys6rqxKp6UJLbszZP9p7ZeX+R5DEjCmY8gYxV9+okL0tyR9ZGy94zthyA+3Vekj9I8qHZN8b/d5Inz/Ydk+R3stbTPpVkV77T196a5B9X1Veq6rxDWzKj1droKgAAoxghAwAYTCADABhMIAMAGEwgAwAYTCADABhs3T2p/+ijj+5t27aNLgM4hC6//PJbu3vr6DoeKP0LNp9F+9e6C2Tbtm3L7t27R5cBHEJV9dnRNSyD/gWbz6L9yy1LAIDBBDIAgMEEMgCAwQQyAIDBBDIAgMEEMgCAwQQyAIDBBDIAgMEEMgCAwQQyAIDBBDIAgMHW3bssAYDv2Lbj4tElHJQb3vDc0SWsFCNkAACDGSGD+AsTgLGMkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADDZpIKuqk6vq2qq6rqp27Oe4F1ZVV9X2KesBAFhFkwWyqjosyflJTklyYpKXVtWJ+zjuqCTnJLl0qloAAFbZlCNkJyW5rruv7+67k1yU5LR9HPdLSd6Y5BsT1gIAsLKmDGTHJLlxbv2m2bb/p6qenOS47r54wjoAAFbasEn9VfWgJG9J8uoFjj2rqnZX1e5bbrll+uIAlkT/AhYxZSC7Oclxc+vHzrbd66gkP5jkj6rqhiR/N8nOfU3s7+4Lunt7d2/funXrhCUDLJf+BSxiykB2WZITqur4qjo8yelJdt67s7tv6+6ju3tbd29L8rEkp3b37glrAgBYOZMFsu7ek+TsJB9Mck2S93b31VX1+qo6darfCwCw3myZ8sO7e1eSXXttO/c+jv0HU9YCALCqPKkfAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYLAtowtgY9m24+LRJQAcFP2LkYyQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAw2aSCrqpOr6tqquq6qduxj/8ur6k+q6sqq+khVnThlPQAAq2iyQFZVhyU5P8kpSU5M8tJ9BK4Lu/tvdfcTk5yX5C1T1QMAsKqmHCE7Kcl13X19d9+d5KIkp80f0N23z60ekaQnrAcAYCVtmfCzj0ly49z6TUmeuvdBVfXKJK9KcniSZ05YDwDASho+qb+7z+/uxyb5hSS/uK9jquqsqtpdVbtvueWWQ1sgwAOgfwGLmDKQ3ZzkuLn1Y2fb7stFSV6wrx3dfUF3b+/u7Vu3bl1iiQDT0r+ARUwZyC5LckJVHV9Vhyc5PcnO+QOq6oS51ecm+cyE9QAArKTJ5pB1956qOjvJB5McluSd3X11Vb0+ye7u3pnk7Kp6VpJvJflKkpdNVQ8AwKqaclJ/untXkl17bTt3bvmcKX8/AMB6MHxSPwDAZieQAQAMJpABAAwmkAEADCaQAQAMNum3LIFpbdtx8egSDsoNb3ju6BKAwdZr/0qm6WFGyAAABhPIAAAGE8gAAAYTyAAABhPIAAAGE8gAAAYTyAAABhPIAAAGE8gAAAYTyAAABhPIAAAGE8gAAAYTyAAABlsokFXV86tKeAOYozcCy7JoI3lJks9U1XlV9bgpCwJYR/RGYCkWCmTdfUaSJyX5syS/VlUfraqzquqoSasDWGF6I7AsCw+1d/ftSd6X5KIkj0zyo0muqKp/OlFtACtPbwSWYdE5ZKdV1fuT/FGSByc5qbtPSfK3k7x6uvIAVpfeCCzLlgWP+7Ekb+3uD89v7O67qurM5ZcFsC7ojcBSLHrL8ot7N5yqemOSdPcfLr0qgPVBbwSWYtFA9ux9bDtlmYUArEN6I7AU+71lWVWvSPJPkjy2qq6a23VUkj+esjCAVaU3Ast2f3PILkzygST/NsmOue13dPeXJ6sKYLXpjcBS3V8g6+6+oapeufeOqvo+jQfYpPRGYKkWGSF7XpLLk3SSmtvXSR4zUV0Aq0xvBJZqv4Gsu583++/xh6YcgNWnNwLLtuiDYZ9WVUfMls+oqrdU1aOmLQ1gtemNwLIs+tiL/5Tkrqq69+nTf5bk1yerCmB90BuBpVg0kO3p7k5yWpL/0N3nZ+3r3QCbmd4ILMWir066o6pek+SMJE+vqgdl7b1tAJuZ3ggsxaIjZC9J8s0kZ3b3F5Mcm+RNk1UFsD7ojcBSLDRCNms0b5lb/1ySd01VFMB6oDcCy7Lotyx/rKo+U1W3VdXtVXVHVd0+dXEAq0xvBJZl0Tlk5yV5fndfM2UxAOuM3ggsxaJzyP5CwwH4LnojsBSLjpDtrqr3JPnvWZvAmiTp7t+epCqA9UFvBJZi0UD2sCR3JXnO3LZOoukAm5neCCzFot+y/OmpCwFYb/RGYFkW/ZblX6+qP6yqT83Wn1BVvzhtaQCrTW8ElmXRSf1vT/KaJN9Kku6+KsnpUxUFsE7ojcBSLBrIHtrdH99r255lFwOwzuiNwFIsGshurarHZm2yaqrqRUm+MFlVAOuD3ggsxaLfsnxlkguSPK6qbk7y50l+YrKqANYHvRFYiv0Gsqp61dzqriSXZG1U7WtJXpi5d7gBbBZ6I7Bs93fL8qjZz/Ykr0jyvUkenuTlSZ58fx9eVSdX1bVVdV1V7djH/ldV1aer6qrZN5UefeCXAHDIPaDeCLC3/Y6QdffrkqSqPpzkyd19x2z9XyW5eH/nVtVhSc5P8uwkNyW5rKp2dven5w77RJLt3X1XVb0ia++Fe8lBXgvAIfFAeiPAviw6qf8RSe6eW797tm1/TkpyXXdf3913J7koyWnzB3T3Jd1912z1Y0mOXbAegFVwML0R4LssOqn/XUk+XlXvn62/IMmv3c85xyS5cW79piRP3c/xZyb5wIL1AKyCg+mNAN9l0Vcn/Zuq+kCSvz/b9NPd/YllFVFVZ2RtLsYz7mP/WUnOSpJHPepRy/q1AA/IIr1R/wIWsegIWbr7iiRXHMBn35zkuLn1Y2fb/j9V9awk/zLJM7r7m/fxuy/I2lfLs3379j6AGgAmdX+9Uf8CFrHoHLKDcVmSE6rq+Ko6PGuvE9k5f0BVPSnJ25Kc2t1fmrAWAICVNVkg6+49Sc5O8sEk1yR5b3dfXVWvr6pTZ4e9KcmRSf5bVV1ZVTvv4+MAADashW9ZHozu3pW1hybObzt3bvlZU/5+AID1YMpblgAALEAgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhsy+gC2LdtOy4eXQIAcIgIZAAslT8o4cC5ZQkAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADDYpIGsqk6uqmur6rqq2rGP/U+vqiuqak9VvWjKWgAAVtVkgayqDktyfpJTkpyY5KVVdeJeh30uyU8luXCqOgAAVt2WCT/7pCTXdff1SVJVFyU5Lcmn7z2gu2+Y7btnwjoAAFbalLcsj0ly49z6TbNtAADMWReT+qvqrKraXVW7b7nlltHlACxM/wIWMWUguznJcXPrx862HbDuvqC7t3f39q1bty6lOIBDQf8CFjFlILssyQlVdXxVHZ7k9CQ7J/x9AADr0mSBrLv3JDk7yQeTXJPkvd19dVW9vqpOTZKqekpV3ZTkxUneVlVXT1UPAMCqmvJblunuXUl27bXt3Lnly7J2KxMAYNNaF5P6AQA2MoEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYLBJXy6+CrbtuHh0CQAA+7XhAxnAeuUPStg83LIEABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYbNJAVlUnV9W1VXVdVe3Yx/6/VFXvme2/tKq2TVkPAMAqmiyQVdVhSc5PckqSE5O8tKpO3OuwM5N8pbt/IMlbk7xxqnoAAFbVlCNkJyW5rruv7+67k1yU5LS9jjktyX+dLb8vyY9UVU1YEwDAypkykB2T5Ma59Ztm2/Z5THfvSXJbku+fsCYAgJWzZXQBi6iqs5KcNVu9s6quPYDTj05y6/KrWimb4RoT17lh1BsP+BofPVUtU9O/FrIZrnMzXGOySa7zAHvYQv1rykB2c5Lj5taPnW3b1zE3VdWWJN+T5P/s/UHdfUGSCw6miKra3d3bD+bc9WIzXGPiOjeSzXCN99K/7t9muM7NcI2J63wgprxleVmSE6rq+Ko6PMnpSXbudczOJC+bLb8oyYe6uyesCQBg5Uw2Qtbde6rq7CQfTHJYknd299VV9foku7t7Z5J3JPn1qrouyZezFtoAADaVSeeQdfeuJLv22nbu3PI3krx4yhpykLcK1pnNcI2J69xINsM1LsNm+f+0Ga5zM1xj4joPWrlDCAAwllcnAQAMtmEDWVW9s6q+VFWfGl3LVKrquKq6pKo+XVVXV9U5o2uaQlU9pKo+XlWfnF3n60bXNJWqOqyqPlFV/3N0LVOpqhuq6k+q6sqq2j26nlWkf20c+tfGMmX/2rC3LKvq6UnuTPKu7v7B0fVMoaoemeSR3X1FVR2V5PIkL+juTw8ubalmb284orvvrKoHJ/lIknO6+2ODS1u6qnpVku1JHtbdzxtdzxSq6oYk27t7wz+r6GDpXxuH/rWxTNm/NuwIWXd/OGvf3NywuvsL3X3FbPmOJNfku9+GsO71mjtnqw+e/Wy4vySq6tgkz03yq6NrYSz9a+PQv1jUhg1km01VbUvypCSXjq1kGrOh8CuTfCnJ73f3RrzOf5fknye5Z3QhE+skv1dVl8+eYs8mp39tCPrXAySQbQBVdWSS30ryc919++h6ptDd3+7uJ2btjQ8nVdWGuo1TVc9L8qXuvnx0LYfAD3X3k5OckuSVs9tzbFL61/qnfy2HQLbOzeYk/FaSd3f3b4+uZ2rd/dUklyQ5eXQtS/a0JKfO5idclOSZVfUbY0uaRnffPPvvl5K8P8lJYytiFP1rw9C/lkAgW8dmk0XfkeSa7n7L6HqmUlVbq+rhs+W/nOTZSf50bFXL1d2v6e5ju3tb1t5Y8aHuPmNwWUtXVUfMJnCnqo5I8pwkG/abhNw3/Wvj0L+WY8MGsqr6zSQfTfI3quqmqjpzdE0TeFqSn8zaXyNXzn7+4eiiJvDIJJdU1VVZe0fq73f3hv1a9Qb3iCQfqapPJvl4kou7+3cH17Ry9K8NRf/aOCbtXxv2sRcAAOvFhh0hAwBYLwQyAIDBBDIAgMEEMgCAwQQyAIDBBDJWSlX9XFU9dG59173P8AFYdXoYB8tjLzjkZg+ErO7+rneezZ70vL27bz3khQEsQA9jCkbIOCSqaltVXVtV78rak43fUVW7q+rqqnrd7Jh/luSvZe0hipfMtt1QVUfPzr+mqt4+O+f3Zk+9TlU9paqumj1Y8k1V9anZ9sdX1cdn26+qqhPGXD2w3ulhTE0g41A6Icl/7O7HJ3l1d29P8oQkz6iqJ3T3ryT5fJIf7u4fvo/zz5+d/9UkL5xt/y9Jfnb28t5vzx3/8iT/frZ9e5KbJrkqYLPQw5iMQMah9Nnu/ths+cer6ookn0jy+CQnLnD+n3f3lbPly5Nsm83NOKq7PzrbfuHc8R9N8i+q6heSPLq7v/7ALwHYxPQwJiOQcSh9LUmq6vgkP5/kR7r7CUkuTvKQBc7/5tzyt5Ns2d/B3X1hklOTfD3Jrqp65sEUDTCjhzEZgYwRHpa1xnZbVT0iySlz++5IctSiH9TdX01yR1U9dbbp9Hv3VdVjklw/u43wO1m7tQDwQOlhLN1+0zlMobs/WVWfSPKnSW5M8sdzuy9I8rtV9fn7mIOxL2cmeXtV3ZPkfyW5bbb9x5P8ZFV9K8kXk/zyUi4A2NT0MKbgsRese1V1ZHffOVvekeSR3X3O4LIAFqKHkRghY2N4blW9Jmv/nj+b5KfGlgNwQPQwjJABAIxmUj8AwGACGQDAYAIZAMBgAhkAwGACGQDAYAIZAMBg/xcBVhPqgD4QsgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, (ax1m, ax2m) = plt.subplots(1, 2, sharey=True, figsize=(10,5))\n",
    "ax1m.hist(Xtr_1m[Xtr_1m !=0], 5, density= True)\n",
    "ax1m.set_title('Train')\n",
    "ax1m.set(xlabel=\"ratings\", ylabel=\"density\")\n",
    "ax2m.hist(Xtst_1m[Xtst_1m !=0], 5, density= True)\n",
    "ax2m.set_title('Test')\n",
    "ax2m.set(xlabel=\"ratings\", ylabel=\"density\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now repeat the same operations for the other datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating the user/item affinity matrix...\n",
      "Matrix generated, sparseness percentage: 93\n"
     ]
    }
   ],
   "source": [
    "#100k\n",
    "am100k = AffinityMatrix(DF = mldf_100k, **header)\n",
    "X100k= am100k.gen_affinity_matrix()\n",
    "Xtr_100k, Xtst_100k = numpy_stratified_split(X100k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(0,0.5,'density'), Text(0.5,0,'ratings')]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmsAAAFNCAYAAABfUShSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHklJREFUeJzt3X+wX3dd5/Hni5TC2hYFe1exaUiKwdoqW+BSdwepv1pIF2xQQKLWLU6dWLdd64CzpKvWJVgX6lhcZ+tClK6i1oDUH3e3wYpScFDa5vYHhRQzpKG2iSCBlrZAaZv2vX98T/Dby036TXJPvp97v8/HzJ2c8znn873vk8m855XzPT9SVUiSJKlNTxl3AZIkSdo/w5okSVLDDGuSJEkNM6xJkiQ1zLAmSZLUMMOaJElSwwxrWpKSLEvypSQrxl2LJEmHw7CmJnTBat/P40keGlr/yYP9vKp6rKqOraq7+6hXkuaz0L1s6HNvSHLuQtaqxeOocRcgAVTVsfuWk9wF/ExV/c3+9k9yVFXtPRK1SdKoDraXSaPwzJoWhSS/luQ9Sf4kyYPAuUn+Q/e/zS8m+UyS307y1G7/o5JUkpXd+h9129+f5MEkH02yaoyHJGkCdZdo/EqSnUk+n+SPk3xTt+2YJJuT3Nv1tRuTPDPJbwIvBn6vO0P3m+M9Ch1phjUtJj8CXA18I/AeYC9wMXA88BJgDfCzB5j/E8CvAM8C7gbe0mexkjSPXwReBnwvsBx4FHh7t+1nGHzjdQKDvnYR8EhVvRHYyuAs3bHduiaIYU2LyUeq6v9W1eNV9VBVba2qG6tqb1XtBDYB33eA+e+rqtmqehT4Y+C0I1K1JP2rC4ANVfXPVfVV4M3A65KEQXCbAp7b9bWtVfXlcRarNnjNmhaTe4ZXkpwM/CbwIuAbGPx7vvEA8z87tPwV4Nj97ShJC60LZCcCW5LU0KanAN8MvAv4VuB9SY4F3g38SlU9dsSLVVM8s6bFpOasvxP4BPDtVfUM4FIgR7wqSRpBVRWwG/jBqvqmoZ+nV9Xnq+rhqrq0qk4GzgBeC6zbN31cdWv8DGtazI4D7ge+nOQ7OfD1apLUgncAb01yIkCSf5vkh7vlM5OckuQpwAMMrst9vJv3L8BJ4yhY42dY02L2RuA84EEGZ9neM95yJOlJXQ78DfDB7s72fwBe2G07AfhLBj3tE8AW/rWvvR34T0nuS3L5kS1Z45bBWVlJkiS1yDNrkiRJDes1rCVZk2R7kh1JNhxgv1d3DzCd7tZXdq/ouK37eUefdUqSJLWqt0d3JFkGXAmcBewCtiaZqao75ux3HIMHm8595MKdVeVzsCRJ0kTr88za6cCOqtpZVY8Am4G18+z3FuBtwFd7rEWSJGlR6jOsncATH2K6qxv7miQvBE6sqmvnmb8qya1JPpzkpT3WKUmS1KyxvcGge47MFcDr59n8GWBFVX0hyYuAv0hyalU9MOcz1gPrAY455pgXnXzyyT1XLaklN9988+eramrcdRwqe5g0uQ6mf/UZ1nYzeK3GPsu7sX2OA74L+NDgDRx8KzCT5JyqmgUeBqiqm5PcCTwPmB3+BVW1icH7IJmenq7Z2SdslrTEJfmncddwOOxh0uQ6mP7V59egW4HVSVYlOZrBKzNm9m2sqvur6viqWllVK4EbgHOqajbJVHeDAklOAlYDO3usVZIkqUm9nVmrqr1JLgKuA5YBV1XVtiQbgdmqmjnA9DOAjUkeZfCqjQuq6t6+apUkSWpVr9esVdUWBq/LGB67dD/7fv/Q8jXANX3WJkmStBj4BgNJkqSGGdYkSZIaZliTJElqmGFNkiSpYYY1SZKkhhnWJEmSGmZYkyRJaphhTZIkqWFje5G7tBis3HDtuEs4JHe99RXjLkGStEA8syZJktQww5okSVLD/BpUkqQlyks5lgbPrEmSJDXMsCZJktQww5okSVLDDGuSJEkNM6xJkiQ1zLAmSZLUMMOaJElSwwxrkiRJDTOsSZIkNazXsJZkTZLtSXYk2XCA/V6dpJJMD41d0s3bnuTlfdYpSZLUqt5eN5VkGXAlcBawC9iaZKaq7piz33HAxcCNQ2OnAOuAU4FvA/4myfOq6rG+6pUkSWpRn2fWTgd2VNXOqnoE2AysnWe/twBvA746NLYW2FxVD1fVp4Ed3edJkiRNlD7D2gnAPUPru7qxr0nyQuDEqpr7ptknnStJkjQJxnaDQZKnAFcAbzyMz1ifZDbJ7J49exauOEk6AuxhkkbRZ1jbDZw4tL68G9vnOOC7gA8luQv498BMd5PBk80FoKo2VdV0VU1PTU0tcPmS1C97mKRR9BnWtgKrk6xKcjSDGwZm9m2sqvur6viqWllVK4EbgHOqarbbb12SpyVZBawGbuqxVkmSpCb1djdoVe1NchFwHbAMuKqqtiXZCMxW1cwB5m5L8l7gDmAvcKF3gkqSpEnUW1gDqKotwJY5Y5fuZ9/vn7N+GXBZb8VJkiQtAr7BQJIkqWGGNUmSpIYZ1iRJkhpmWJMkSWqYYU2SJKlhhjVJkqSGGdYkSZIaZliTJElqmGFNkiSpYYY1SZKkhhnWJEmSGmZYkyRJaphhTZIkqWGGNUmSpIYZ1iRJkhpmWJMkSWqYYU2SJKlhhjVJkqSGGdYkSZIaZliTJElqmGFNkiSpYb2GtSRrkmxPsiPJhnm2X5Dk40luS/KRJKd04yuTPNSN35bkHX3WKUmS1Kqj+vrgJMuAK4GzgF3A1iQzVXXH0G5XV9U7uv3PAa4A1nTb7qyq0/qqT5IkaTHo88za6cCOqtpZVY8Am4G1wztU1QNDq8cA1WM9kiRJi06fYe0E4J6h9V3d2BMkuTDJncDlwM8PbVqV5NYkH07y0h7rlCRJatbYbzCoqiur6rnAm4Bf7oY/A6yoqhcAbwCuTvKMuXOTrE8ym2R2z549R65oSVoA9jBJo+gzrO0GThxaX96N7c9m4FUAVfVwVX2hW74ZuBN43twJVbWpqqaranpqamrBCpekI8EeJmkUfYa1rcDqJKuSHA2sA2aGd0iyemj1FcCnuvGp7gYFkpwErAZ29lirJElSk3q7G7Sq9ia5CLgOWAZcVVXbkmwEZqtqBrgoyZnAo8B9wHnd9DOAjUkeBR4HLqiqe/uqVZKkA1m54dpxl6AJ1ltYA6iqLcCWOWOXDi1fvJ951wDX9FmbJEnSYjD2GwwkSZK0f4Y1SZKkhhnWJEmSGtbrNWvSPl6cK0nSofHMmiRJUsMMa5IkSQ0zrEmSJDXMsCZJktQww5okSVLDDGuSJEkNM6xJkiQ1zLAmSZLUMMOaJElSwwxrkiRJDTOsSZIkNcywJkmS1DDDmiRJUsMMa5IkSQ0zrEmSJDXMsCZJktQww5okSVLDDGuSJEkN6zWsJVmTZHuSHUk2zLP9giQfT3Jbko8kOWVo2yXdvO1JXt5nnZIkSa3qLawlWQZcCZwNnAL8+HAY61xdVd9dVacBlwNXdHNPAdYBpwJrgN/pPk+SJGmi9Hlm7XRgR1XtrKpHgM3A2uEdquqBodVjgOqW1wKbq+rhqvo0sKP7PEmSpInSZ1g7AbhnaH1XN/YESS5McieDM2s/f5Bz1yeZTTK7Z8+eBStcko4Ee5ikUYz9BoOqurKqngu8Cfjlg5y7qaqmq2p6amqqnwIlqSf2MEmj6DOs7QZOHFpf3o3tz2bgVYc4V5IkaUnqM6xtBVYnWZXkaAY3DMwM75Bk9dDqK4BPdcszwLokT0uyClgN3NRjrZIkSU06qq8Prqq9SS4CrgOWAVdV1bYkG4HZqpoBLkpyJvAocB9wXjd3W5L3AncAe4ELq+qxvmqVJElqVW9hDaCqtgBb5oxdOrR88QHmXgZc1l91kiRJ7Rv7DQaSJEnaP8OaJElSwwxrkiRJDTOsSZIkNcywJkmS1DDDmiRJUsMMa5IkSQ0zrEmSJDXMsCZJktQww5okSVLDDGuSJEkNM6xJkiQ1zLAmSZLUMMOaJElSwwxrkiRJDTtq3AVIWngrN1w77hIO2V1vfcW4S5CkpnhmTZIkqWGGNUmSpIb5NagkSWrKYr2Uo6/LODyzJkmS1DDDmiRJUsN6DWtJ1iTZnmRHkg3zbH9DkjuS3J7kb5M8Z2jbY0lu635m+qxTkiSpVb1ds5ZkGXAlcBawC9iaZKaq7hja7VZguqq+kuTngMuB13XbHqqq0/qqT5IkaTEY6cxakh9OcrBn4U4HdlTVzqp6BNgMrB3eoaqur6qvdKs3AMsP8ndIUnMOsWdK0rxGbSavAz6V5PIkJ4845wTgnqH1Xd3Y/pwPvH9o/elJZpPckORVI/5OSWrBofRMSZrXSGGtqs4FXgDcCfx+ko8mWZ/kuIUoIsm5wDTwG0PDz6mqaeAngN9K8tx55q3vAt3snj17FqIUSTpso/ZMe5ikUYx8mr6qHgDex+DrzGcDPwLckuS/7GfKbuDEofXl3dgTJDkT+CXgnKp6eOj37e7+3Al8iEHjm1vTpqqarqrpqampUQ9Fkno3Ss+0h0kaxajXrK1N8ucMQtNTgdOr6mzg3wFv3M+0rcDqJKuSHA2sA55wV2eSFwDvZBDUPjc0/swkT+uWjwdeAgzfmCBJzTrEnilJ8xr1btAfBd5eVX83PNjdxXn+fBOqam+Si4DrgGXAVVW1LclGYLaqZhh87Xks8KdJAO6uqnOA7wTemeRxBoHyrXPuIpWklh10z5Sk/Rk1rH12btNJ8raqelNV/e3+JlXVFmDLnLFLh5bP3M+8fwC+e8TaJKk1h9QzJWk+o16zdtY8Y2cvZCGStITYMyUtmAOeWeseVPufgecmuX1o03HA3/dZmCQtNvZMSX14sq9Br2bw7LP/AQy/LurBqrq3t6okaXGyZ0pacE8W1qqq7kpy4dwNSZ5l85GkJ7BnSlpwo5xZeyVwM1BAhrYVcFJPdUnSYmTPlLTgDhjWquqV3Z+rjkw5krR42TMl9WHUh+K+JMkx3fK5Sa5IsqLf0iRpcbJnSlpIoz66438DX0my7+nbdwJ/2FtVkrS42TMlLZhRw9reqipgLfC/qupKBreiS5K+nj1T0oIZ9Q0GDya5BDgXOCPJUxi8706S9PXsmZIWzKhn1l4HPAycX1WfBZYzeK+nJOnr2TMlLZiRzqx1zeaKofW7gXf3VZQkLWb2TEkLadS7QX80yaeS3J/kgSQPJnmg7+IkaTGyZ0paSKNes3Y58MNV9ck+i5GkJcKeKWnBjHrN2r/YdCRpZPZMSQtm1DNrs0neA/wFg4tmAaiqP+ulKkla3OyZkhbMqGHtGcBXgJcNjRVg45Gkr2fPlLRgRr0b9Kf7LkSSlgp7pqSFNOrdoM9L8rdJPtGtPz/JL/dbmiQtTvZMSQtp1BsMfhe4BHgUoKpuB9b1VZQkLXL2TEkLZtSw9g1VddOcsb0LXYwkLRH2TEkLZtSw9vkkz2VwgSxJXgN85skmJVmTZHuSHUk2zLP9DUnuSHJ795XBc4a2ndc9VPJTSc4bsU5JasEh9UxJms+od4NeCGwCTk6yG/g08JMHmpBkGXAlcBawC9iaZKaq7hja7VZguqq+kuTnGDxI8nVJngX8KjDNoNnd3M297yCOTZLG5aB7piTtzwHDWpI3DK1uAa5ncDbuy8CrGXr33TxOB3ZU1c7uszYDa4GvhbWqun5o/xuAc7vllwMfqKp7u7kfANYAf/LkhyRJ43GYPVOS5vVkZ9aO6/78DuDFwF8CAX4KmHs9xlwnAPcMre8CvucA+58PvP8Ac094kt8nSeN2OD1TkuZ1wLBWVW8GSPJ3wAur6sFu/b8D1y5UEUnOZfCV5/cd5Lz1wHqAFStWLFQ5knRIDrZn2sMkjWLUGwy+BXhkaP2RbuxAdgMnDq0v78aeIMmZwC8B51TVwwczt6o2VdV0VU1PTU096UFI0hEyUs+0h0kaxag3GLwbuCnJn3frrwJ+/0nmbAVWJ1nFIGitA35ieIckLwDeCaypqs8NbboO+PUkz+zWX8bgmUWStBgcSs+UpHmN+rqpy5K8H3hpN/TTVXXrk8zZm+QiBsFrGXBVVW1LshGYraoZ4DeAY4E/TQJwd1WdU1X3JnkLg8AHsHHfzQaS1LpD6ZmStD+jnlmjqm4BbjmYD6+qLQzuiBoeu3Ro+cwDzL0KuOpgfp8kteJQeqYkzWfUa9YkSZI0BoY1SZKkhhnWJEmSGmZYkyRJaphhTZIkqWGGNUmSpIYZ1iRJkhpmWJMkSWqYYU2SJKlhhjVJkqSGGdYkSZIaZliTJElqmGFNkiSpYYY1SZKkhhnWJEmSGmZYkyRJaphhTZIkqWGGNUmSpIYZ1iRJkhpmWJMkSWqYYU2SJKlhvYa1JGuSbE+yI8mGebafkeSWJHuTvGbOtseS3Nb9zPRZpyRJUquO6uuDkywDrgTOAnYBW5PMVNUdQ7vdDbwe+MV5PuKhqjqtr/okSZIWg97CGnA6sKOqdgIk2QysBb4W1qrqrm7b4z3WIUmStGj1+TXoCcA9Q+u7urFRPT3JbJIbkrxqYUuTJElaHPo8s3a4nlNVu5OcBHwwycer6s7hHZKsB9YDrFixYhw1StIhs4dJGkWfZ9Z2AycOrS/vxkZSVbu7P3cCHwJeMM8+m6pquqqmp6amDq9aSTrC7GGSRtFnWNsKrE6yKsnRwDpgpLs6kzwzydO65eOBlzB0rZskSdKk6C2sVdVe4CLgOuCTwHuraluSjUnOAUjy4iS7gNcC70yyrZv+ncBsko8B1wNvnXMXqSRJ0kTo9Zq1qtoCbJkzdunQ8lYGX4/OnfcPwHf3WZsk6chaueHacZcgLUot32CgedjsJEmaLL5uSpIkqWGGNUmSpIYZ1iRJkhpmWJMkSWqYYU2SJKlhhjVJkqSGGdYkSZIaZliTJElqmGFNkiSpYYY1SZKkhhnWJEmSGmZYkyRJaphhTZIkqWGGNUmSpIYZ1iRJkhpmWJMkSWqYYU2SJKlhhjVJkqSGGdYkSZIaZliTJElqmGFNkiSpYb2GtSRrkmxPsiPJhnm2n5HkliR7k7xmzrbzknyq+zmvzzolSZJa1VtYS7IMuBI4GzgF+PEkp8zZ7W7g9cDVc+Y+C/hV4HuA04FfTfLMvmqVJElqVZ9n1k4HdlTVzqp6BNgMrB3eoaruqqrbgcfnzH058IGqureq7gM+AKzpsVZJkqQm9RnWTgDuGVrf1Y0t2Nwk65PMJpnds2fPIRcqSeNgD5M0ikV9g0FVbaqq6aqanpqaGnc5knRQ7GGSRtFnWNsNnDi0vrwb63uuJEnSktFnWNsKrE6yKsnRwDpgZsS51wEvS/LM7saCl3VjkiRJE6W3sFZVe4GLGISsTwLvraptSTYmOQcgyYuT7AJeC7wzybZu7r3AWxgEvq3Axm5MkiRpohzV54dX1RZgy5yxS4eWtzL4inO+uVcBV/VZnyRJUusW9Q0GkiRJS51hTZIkqWGGNUmSpIYZ1iRJkhpmWJMkSWqYYU2SJKlhhjVJkqSGGdYkSZIaZliTJElqWK9vMGjZyg3XjrsESTok9i9psnhmTZIkqWGGNUmSpIYZ1iRJkhpmWJMkSWqYYU2SJKlhhjVJkqSGGdYkSZIaZliTJElqmGFNkiSpYYY1SZKkhhnWJEmSGtZrWEuyJsn2JDuSbJhn+9OSvKfbfmOSld34yiQPJbmt+3lHn3VKkiS1qrcXuSdZBlwJnAXsArYmmamqO4Z2Ox+4r6q+Pck64G3A67ptd1bVaX3VJ0mStBj0eWbtdGBHVe2sqkeAzcDaOfusBf6gW34f8ENJ0mNNkiRJi0qfYe0E4J6h9V3d2Lz7VNVe4H7gm7ttq5LcmuTDSV7aY52SJEnN6u1r0MP0GWBFVX0hyYuAv0hyalU9MLxTkvXAeoAVK1aMoUxJOnT2MEmj6PPM2m7gxKH15d3YvPskOQr4RuALVfVwVX0BoKpuBu4Enjf3F1TVpqqarqrpqampHg5BkvpjD5M0ij7D2lZgdZJVSY4G1gEzc/aZAc7rll8DfLCqKslUd4MCSU4CVgM7e6xVkiSpSb19DVpVe5NcBFwHLAOuqqptSTYCs1U1A7wL+MMkO4B7GQQ6gDOAjUkeBR4HLqiqe/uqVZIkqVW9XrNWVVuALXPGLh1a/irw2nnmXQNc02dtkiRJi4FvMJAkSWqYYU2SJKlhhjVJkqSGGdYkSZIaZliTJElqmGFNkiSpYYY1SZKkhhnWJEmSGmZYkyRJaphhTZIkqWGGNUmSpIYZ1iRJkhpmWJMkSWqYYU2SJKlhhjVJkqSGGdYkSZIaZliTJElqmGFNkiSpYYY1SZKkhhnWJEmSGmZYkyRJalivYS3JmiTbk+xIsmGe7U9L8p5u+41JVg5tu6Qb357k5X3WKUmS1KrewlqSZcCVwNnAKcCPJzllzm7nA/dV1bcDbwfe1s09BVgHnAqsAX6n+zxJkqSJ0ueZtdOBHVW1s6oeATYDa+fssxb4g275fcAPJUk3vrmqHq6qTwM7us+TJEmaKH2GtROAe4bWd3Vj8+5TVXuB+4FvHnGuJEnSknfUuAs4HEnWA+u71S8l2X4Q048HPr/wVTVnEo5zEo4RJuQ487aDOs7n9FlL3w6jh03EvwU8zqVkEo6xt/7VZ1jbDZw4tL68G5tvn11JjgK+EfjCiHOpqk3ApkMpLslsVU0fytzFZBKOcxKOETzOpehQe9ik/B15nEvHJBwj9HecfX4NuhVYnWRVkqMZ3DAwM2efGeC8bvk1wAerqrrxdd3doquA1cBNPdYqSZLUpN7OrFXV3iQXAdcBy4Crqmpbko3AbFXNAO8C/jDJDuBeBoGObr/3AncAe4ELq+qxvmqVJElqVa/XrFXVFmDLnLFLh5a/Crx2P3MvAy7rsbxD+vp0EZqE45yEYwSPU/9qUv6OPM6lYxKOEXo6zgy+dZQkSVKLfN2UJElSwyYurCW5Ksnnknxi3LX0JcmJSa5PckeSbUkuHndNfUjy9CQ3JflYd5xvHndNfUmyLMmtSf7fuGvpS5K7knw8yW1JZsddT4smoX/BZPSwSepfYA877M+etK9Bk5wBfAl4d1V917jr6UOSZwPPrqpbkhwH3Ay8qqruGHNpC6p728UxVfWlJE8FPgJcXFU3jLm0BZfkDcA08IyqeuW46+lDkruA6apa8s9iOlST0L9gMnrYJPUvsIcdrok7s1ZVf8fgztMlq6o+U1W3dMsPAp9kCb4Boga+1K0+tftZcv/7SLIceAXwe+OuReM1Cf0LJqOHTUr/AnvYQpi4sDZpkqwEXgDcON5K+tGdWr8N+Bzwgapaisf5W8B/BR4fdyE9K+Cvk9zcPdlfWtI9bEL6F9jDDpthbQlLcixwDfALVfXAuOvpQ1U9VlWnMXjLxelJltRXQ0leCXyuqm4edy1HwPdW1QuBs4ELu6/8NMGWeg9b6v0L7GEL9cGGtSWquwbiGuCPq+rPxl1P36rqi8D1wJpx17LAXgKc010LsRn4wSR/NN6S+lFVu7s/Pwf8OXD6eCvSOE1SD1vC/QvsYQvCsLYEdReuvgv4ZFVdMe56+pJkKsk3dcv/BjgL+MfxVrWwquqSqlpeVSsZvOHjg1V17pjLWnBJjukuJCfJMcDLgCV9x6P2bxJ62CT0L7CHLdTnT1xYS/InwEeB70iyK8n5466pBy8BforB/2Bu637+47iL6sGzgeuT3M7gXbQfqKole1v4EvctwEeSfIzBe4Cvraq/GnNNzZmQ/gWT0cPsX0tLrz1s4h7dIUmStJhM3Jk1SZKkxcSwJkmS1DDDmiRJUsMMa5IkSQ0zrEmSJDXMsKZFI8kvJPmGofUt+55TJEkts3/pcPjoDjWlexhmqurr3iHXPQF7uqo+f8QLk6QnYf9SXzyzprFLsjLJ9iTvZvDE53clmU2yLcmbu31+Hvg2Bg+RvL4buyvJ8d38Tyb53W7OX3dPBCfJi5Pc3j1U8zeSfKIbPzXJTd347UlWj+foJS1m9i8dCYY1tWI18DtVdSrwxqqaBp4PfF+S51fVbwP/DPxAVf3AfuZf2c3/IvDqbvz/AD/bvSz5saH9LwD+Zzc+Dezq5agkTQL7l3plWFMr/qmqbuiWfyzJLcCtwKnAKSPM/3RV3dYt3wys7K4HOa6qPtqNXz20/0eB/5bkTcBzquqhwz8ESRPK/qVeGdbUii8DJFkF/CLwQ1X1fOBa4OkjzH94aPkx4KgD7VxVVwPnAA8BW5L84KEULUnYv9Qzw5pa8wwGje/+JN8CnD207UHguFE/qKq+CDyY5Hu6oXX7tiU5CdjZfT3xlwy+spCkw2H/Ui8OmN6lI62qPpbkVuAfgXuAvx/avAn4qyT/vJ/rPuZzPvC7SR4HPgzc343/GPBTSR4FPgv8+oIcgKSJZf9SX3x0h5a0JMdW1Ze65Q3As6vq4jGXJUlPyv6lfTyzpqXuFUkuYfBv/Z+A14+3HEkamf1LgGfWJEmSmuYNBpIkSQ0zrEmSJDXMsCZJktQww5okSVLDDGuSJEkNM6xJkiQ17P8DpiOoUe21e64AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, (ax1k, ax2k) = plt.subplots(1, 2, sharey=True, figsize=(10,5))\n",
    "ax1k.hist(Xtr_100k[Xtr_100k !=0], 5, density= True)\n",
    "ax1k.set_title('Train')\n",
    "ax1k.set(xlabel=\"ratings\", ylabel=\"density\")\n",
    "ax2k.hist(Xtst_100k[Xtst_100k !=0], 5, density= True)\n",
    "ax2k.set_title('Test')\n",
    "ax2k.set(xlabel=\"ratings\", ylabel=\"density\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plots above we can see that the two datasets have very similar rating distributions. The main difference is in the degree of sparsness of the user/item affinity matrix; this is an important factor as it states the ratio between datapoints and unrated movies to infere. Note that the split function returns the total (or per dataset) sparsness, not the user-wise one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#collection of evaluation metrics for later use\n",
    "def ranking_metrics(\n",
    "    data_size,\n",
    "    data_true,\n",
    "    data_pred,\n",
    "    time_train,\n",
    "    time_test,\n",
    "    K\n",
    "):\n",
    "\n",
    "    eval_map = map_at_k(data_true, data_pred, col_user=\"userID\", col_item=\"movieID\", \n",
    "                    col_rating=\"rating\", col_prediction=\"prediction\", \n",
    "                    relevancy_method=\"top_k\", k= K)\n",
    "\n",
    "    eval_ndcg = ndcg_at_k(data_true, data_pred, col_user=\"userID\", col_item=\"movieID\", \n",
    "                      col_rating=\"rating\", col_prediction=\"prediction\", \n",
    "                      relevancy_method=\"top_k\", k= K)\n",
    "\n",
    "    eval_precision = precision_at_k(data_true, data_pred, col_user=\"userID\", col_item=\"movieID\", \n",
    "                               col_rating=\"rating\", col_prediction=\"prediction\", \n",
    "                               relevancy_method=\"top_k\", k= K)\n",
    "\n",
    "    eval_recall = recall_at_k(data_true, data_pred, col_user=\"userID\", col_item=\"movieID\", \n",
    "                          col_rating=\"rating\", col_prediction=\"prediction\", \n",
    "                          relevancy_method=\"top_k\", k= K)\n",
    "\n",
    "    \n",
    "    df_result = pd.DataFrame(\n",
    "        {   \"Dataset\": data_size,\n",
    "            \"K\": K,\n",
    "            \"MAP\": eval_map,\n",
    "            \"nDCG@k\": eval_ndcg,\n",
    "            \"Precision@k\": eval_precision,\n",
    "            \"Recall@k\": eval_recall,\n",
    "            \"Train time (s)\": time_train,\n",
    "            \"Test time (s)\": time_test\n",
    "        }, \n",
    "        index=[0]\n",
    "    )\n",
    "    \n",
    "    return df_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Model application, performance and analysis of the results  \n",
    "\n",
    "The model has been implemented as a Tensorflow (TF) class with the TF session hidden inside the `fit()` method, so that no explicit call is needed. The algorithm operates in three different steps: \n",
    "\n",
    "- Model initialization: This is where we tell TF how to build the computational graph. The main parameters to specify are the number of hidden units, the number of training epochs and the minibatch size. \n",
    "\n",
    "- Model fit: This is where we train the model on the data. The method takes two arguments: the training and test set matrices. Note that the model is trained **only** on the training set, the test set is used to display the test set accuracy of the trained model, that in turn is an estimation of the generazation capabilities of the algorithm. It is generally useful to look at these quantities to have a first idea of the optimization behaviour.  \n",
    "\n",
    "- Model prediction: This is where we generate ratings for the unseen items. Once the model has been trained and we are satisfied with its overall accuracy, we sample new ratings from the learned distribution. In particular, we extract the top_k (e.g. 10) most relevant recommendations according to some predefined scorea. The prediction is then returned in a dataframe format ready to be analysed and deployed.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 1m Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 1.12.0\n"
     ]
    }
   ],
   "source": [
    "#First we initialize the model class\n",
    "model_1m = RBM(hidden_units= 1200, training_epoch = 30, minibatch_size= 350, with_metrics=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the first time the fit method is called it may take longer to return the result. This is due to the fact that TF needs to initialized the GPU session. You will notice that this is not the case when training the algorithm the second or more times. As for the `minibatch_size`, you would like to choose a value that gives you a good generalization error while mantaining a reasonable running time. The lower the size, the closer you get to stochastic gradient descent, but training takes longer. A big size value (say 1/2 of batch size) will speed up training but will increase the generalization error.     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Creating the computational graph\n",
      "Initialize Gibbs protocol\n",
      "training epoch 0 rmse 0.961369\n",
      "training epoch 10 rmse 0.902065\n",
      "training epoch 20 rmse 0.864718\n",
      "training epoch 30 rmse 0.848415\n",
      "done training, Training time 10.9566452\n",
      "Train set accuracy 0.3561482\n",
      "Test set accuracy 0.3516242\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEOCAYAAABxdpuaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VOX5xvHvk40kEAiEEIEQdoSIyBJWQdCqxRVFRXGpWBW1au1iFeuvtaV136q1LqiIaAVxa93qhlIUUQiL7LsIAQohCLIHkuf3xwwaI0gGkpnM5P5cVy7OnGXmOY7k5pz3Pe9r7o6IiEgo4iJdgIiIRB+Fh4iIhEzhISIiIVN4iIhIyBQeIiISMoWHiIiETOEhIiIhU3iIiEjIFB4iIhKyhEgXUFUaNmzoLVq0iHQZIiJRZcaMGRvdPfNg+8VseLRo0YL8/PxIlyEiElXM7KuK7KfbViIiEjKFh4iIhEzhISIiIYvZNg8RkUOxZ88eCgoK2LVrV6RLqVLJyclkZ2eTmJh4SMcrPEREyigoKCAtLY0WLVpgZpEup0q4O0VFRRQUFNCyZctDeg/dthIRKWPXrl1kZGTEbHAAmBkZGRmHdXWl8BARKSeWg2Ofwz3HsIaHmQ00s8VmtszMRuxne3Mzm2hmc8xskplll9mWY2bvmdlCM1tgZi2qosZde0q48z8LWb1pR1W8vYhITAhbeJhZPPAP4BQgFxhqZrnldrsPGOvunYCRwJ1lto0F7nX3DkAPYENV1Llx227++dkqbnp5DqWlmt9dRMJr8+bNPProoyEfd+qpp7J58+YqqGj/wnnl0QNY5u4r3L0YGA8MKrdPLvBhcPmjfduDIZPg7u8DuPs2d6+SS4Ps+qn832kdmLqiiLFTV1bFR4iIHNCBwmPv3r0/etzbb79Nenp6VZX1A+EMj6bA6jKvC4LryvoCGBxcPhtIM7MMoB2w2cxeNbNZZnZv8EqmSpzfvRkDjszkrncW8eXG7VX1MSIiPzBixAiWL19O586d6d69O/369ePMM88kNzdwo+ass86iW7duHHXUUYwaNerb41q0aMHGjRtZuXIlHTp04Morr+Soo47i5JNPZufOnZVeZ3Xrqnsj8IiZDQMmA2uAEgJ19gO6AKuAF4FhwNNlDzaz4cBwgJycnEMuwsy4+5xOnPTAf/nthNm8dHUf4uNivwFNRL7vz2/MZ8Habyr1PXOb1OW2M4464Pa77rqLefPmMXv2bCZNmsRpp53GvHnzvu1SO3r0aBo0aMDOnTvp3r0755xzDhkZGd97j6VLlzJu3DiefPJJhgwZwiuvvMLFF19cqecRziuPNUCzMq+zg+u+5e5r3X2wu3cBbg2u20zgKmV28JbXXuBfQNfyH+Duo9w9z93zMjMPOijkj8qqm8zIQR2ZuWozT3284rDeS0TkUPXo0eN7z2I8/PDDHHPMMfTq1YvVq1ezdOnSHxzTsmVLOnfuDEC3bt1YuXJlpdcVziuP6UBbM2tJIDQuAC4su4OZNQQ2uXspcAswusyx6WaW6e6FwAlAlQ+ZO6hzE96Z9z/uf28Jx7dvRLustKr+SBGpRn7sCiFcateu/e3ypEmT+OCDD5g6dSqpqakMGDBgv89q1KpV69vl+Pj4KrltFbYrj+AVw3XAu8BCYIK7zzezkWZ2ZnC3AcBiM1sCZAG3B48tIXBLa6KZzQUMeLKqazYz/np2R+okJ/CbCbPZU1Ja1R8pIjVcWloaW7du3e+2LVu2UL9+fVJTU1m0aBGfffZZmKv7TljbPNz9beDtcuv+WGb5ZeDlAxz7PtCpSgvcj4Z1anHH2R25+vmZPPrRcm44sW24SxCRGiQjI4Njjz2Wjh07kpKSQlZW1rfbBg4cyOOPP06HDh048sgj6dWrV8TqNPfYfJYhLy/PK3MyqBvGz+KtOev417XH0rFpvUp7XxGpXhYuXEiHDh0iXUZY7O9czWyGu+cd7FgNT1JBfz7zKBrUTuI3E2aze29JpMsREYkohUcFpacmcfc5nViyfht/++CHvRtERGoShUcIjm/fiPPzmvHEf5cz46uvI12OiFSRWL2dX9bhnqPCI0T/d3oHGtdL4caXvmBnsW5ficSa5ORkioqKYjpA9s3nkZycfMjvUd2eMK/20pITuffcTlz41Ofc8+6iatEPXEQqT3Z2NgUFBRQWFka6lCq1bybBQ6XwOAR92jTk0t7NeWbKSn7SPou+bRtGuiQRqSSJiYmHPLteTaLbVofo5lPa0zqzNlc/P4MZX22KdDkiImGl8DhEqUkJ/POKXjSsk8Slo6erAV1EahSFx2E4ol4y44f3DgbINAWIiNQYCo/DdES9ZMYN76UAEZEaReFRCRrXS2Hc8F5kBANk5ioFiIjENoVHJWlcL4Xx+wLkaQWIiMQ2hUclalwvhXFX9qJBMEBmKUBEJEYpPCpZk/RAgNSvncTPFCAiEqMUHlWgSXrgFpYCRERilcKjipQPkNmrN0e6JBGRSqPwqEJN0gO9sNJrJzLsmWksL9wW6ZJERCqFwqOKNU1P4fnLexJvxrBnplG4dXekSxIROWwKjzBonlGbp4d1p3Drbn4+Zjrbd++NdEkiIodF4REmnZul88jQrsxfu4XrXpjJ3pLSSJckInLIFB5hdGJuFn85qyMfLS7kD/+eF9OTzYhIbNN8HmF2Uc/mrPl6J49OWk7T9BSuO6FtpEsSEQmZwiMCfvfTI1m3ZRf3vbeEI+qlcG63Q5/NS0QkEhQeEWBm3H1OJzZs3cWIV+aQVbcW/dpmRrosEZEKU5tHhCQlxPHYxd1o06gO1zw/kwVrv4l0SSIiFRbW8DCzgWa22MyWmdmI/WxvbmYTzWyOmU0ys+xy2+uaWYGZPRK+qqtO3eREnrmsO2nJCVw2ZhprNu+MdEkiIhUStvAws3jgH8ApQC4w1Mxyy+12HzDW3TsBI4E7y23/CzC5qmsNp8b1UhhzWQ92FJcwbPQ0tuzYE+mSREQOKpxXHj2AZe6+wt2LgfHAoHL75AIfBpc/KrvdzLoBWcB7Yag1rI48Io0nLunGyqLt/PzZ6WzaXhzpkkREflQ4w6MpsLrM64LgurK+AAYHl88G0swsw8zigPuBG6u8ygjp07ohD1/QhblrtnD2o1NYtkHjYIlI9VXdGsxvBPqb2SygP7AGKAF+Abzt7gU/drCZDTezfDPLLywsrPpqK9kpRzdm3JW92L57L2c/OoXJS6LvHESkZghneKwBmpV5nR1c9y13X+vug929C3BrcN1moDdwnZmtJNAu8jMzu6v8B7j7KHfPc/e8zMzo7PrarXl9/nXtsTRNT+GyMdMZO3VlpEsSEfmBcIbHdKCtmbU0syTgAuD1sjuYWcPgLSqAW4DRAO5+kbvnuHsLAlcnY939B721YkV2/VRevqYPA9pl8sd/z+eP/56nsbBEpFoJW3i4+17gOuBdYCEwwd3nm9lIMzszuNsAYLGZLSHQOH57uOqrburUSmDUz/K46rhWjJ36FZeNmc6WneqJJSLVg8Xq4Hx5eXmen58f6TIqxYTpq7n1X3PJaZDK05d2p0XD2pEuSURilJnNcPe8g+1X3RrMZT+GdG/Gc5f3pGh7MWc9OoXPVhRFuiQRqeEUHlGiV6sM/n3tsWTUTuKSpz9nQv7qgx8kIlJFFB5RpHlGbV79xbH0apXBTS/PYcqyjZEuSURqKIVHlKmXksioS/JonVmb30yYzdd6Gl1EIkDhEYVSkuJ56IIubNpezIhX52hGQhEJO4VHlOrYtB43/bQ9785fz/jpav8QkfBSeESxy/u2pF/bhox8YwHLCzUWloiEj8IjisXFGfeddwzJiXHcMH4WxXv1FLqIhIfCI8pl1U3m7nM6MW/NN9z/3uJIlyMiNYTCIwacfNQRXNgzhycmr1D3XREJC4VHjPjDabnqvisiYaPwiBFlu+/e/Iq674pI1VJ4xJB93XffW7CecdPUfVdEqo7CI8Z82333zfmaylZEqozCI8bs676bkhiv7rsiUmUUHjFoX/fd+WvVfVdEqobCI0aV7b77n7nrIl2OiMQYhUcM++PpuXTNSedXL85m1qqvI12OiMQQhUcMS06M58mf5ZFVN5krns1nVdGOSJckIjFC4RHjMurU4pnLurO31LlszDS27NgT6ZJEJAYoPGqA1pl1GHVJN1Zv2slVz+eze29JpEsSkSin8KgherbK4N7zOvHZik3c8spcPYEuIoclIdIFSPgM6tyUVUU7uP/9JTRrkMqvT2oX6ZJEJEopPGqY605ow6pNO3ho4lKaNUjl3G7ZkS5JRKKQwqOGMTPuGHw0a7fs5JZX59AkPZk+rRtGuiwRiTJq86iBEuPjePSibrRsWJurnpvB0vVbI12SiESZsIaHmQ00s8VmtszMRuxne3Mzm2hmc8xskpllB9d3NrOpZjY/uO38cNYdi+qlJDJ6WHeSE+O5bMx0CrfujnRJIhJFwhYeZhYP/AM4BcgFhppZbrnd7gPGunsnYCRwZ3D9DuBn7n4UMBD4m5mlh6fy2JVdP5WnL82jaFsxVzw7nR3FeyNdkohEiXBeefQAlrn7CncvBsYDg8rtkwt8GFz+aN92d1/i7kuDy2uBDUBmWKqOcZ2y03l4aBfmrtnC5WPy2VmsZ0BE5ODCGR5NgbIzFBUE15X1BTA4uHw2kGZmGWV3MLMeQBKwvIrqrHFOys3iwfM78/mXRVz+7HQFiIgcVHVrML8R6G9ms4D+wBrg299kZtYYeA64zN1/MFGFmQ03s3wzyy8sLAxXzTFhUOem3D/kGD5bUcQVY6eza48CREQOLJzhsQZoVuZ1dnDdt9x9rbsPdvcuwK3BdZsBzKwu8BZwq7t/tr8PcPdR7p7n7nmZmbqrFaqzu2Rz33nH8OnyIq54Nl8BIiIHFM7wmA60NbOWZpYEXAC8XnYHM2toZvtqugUYHVyfBLxGoDH95TDWXOMM7prNvecew5TlG7lyrAJERPYvbOHh7nuB64B3gYXABHefb2YjzezM4G4DgMVmtgTIAm4Prh8CHAcMM7PZwZ/O4aq9pjm3Wzb3nNOJT5ZtZPhzMxQgIvIDFqsD5OXl5Xl+fn6ky4hqE6av5uZX53Bc20yeuKQbyYnxkS5JRKqYmc1w97yD7VfdGsylGhnSvRl3DT6a/y4p5OrndQUiIt9ReMiPOr97DncOPppJiwu55vkZmgtERACFh1TA0B453HH20Xy0uJBrnp+pKxARUXhIxVzYM4e/ntWRDxdtoO/dH/HwxKUUbdN4WCI1lRrMJSRTlxcxavJyPlpcSK2EOAZ3zebyvi1o0ygt0qWJSCWoaIO55vOQkPRunUHv1hks27CVpz9ZyaszCxg3bRXHH5nJ5X1bcWybDMws0mWKSBXTlYcclqJtu/nn56sYO3UlG7cV0/6INC7v25IzOzehVoK69opEm4peeYQUHsEnvQcCbYGn3H2LmbUANu8bRqS6UHiE1649Jbz+xVqe/vhLFq/fSsM6tbjpp0cypHuzgx8sItVGpYeHmeUA7xEYn6oW0M7dV5jZ34Bkd7/6cAqubAqPyHB3Plm2kb9PXMa0lZsYkpfNyEEd9YChSJSoiocEHyQwZHoDYGeZ9f8GTgitPIlVZka/tpmMG96L609ow4T8As557FNWFe2IdGkiUolCCY9+wEh3L98/80t+OC+H1HDxccZvTz6S0cPyWL1pB6f//WM+XLQ+0mWJSCUJJTxSgOL9rM8EdlVOORJrTmifxZvX9yO7fio/H5PPA+8tpqQ0NjtpiNQkoYTHp8DQMq/3/Qa4AZhcaRVJzMnJSOXVX/ThvG7ZPPzhMoY9M41N2/f37xARiRahhMfvgd+Z2QsEng+5xcw+A84A/q8qipPYkZwYzz3nduLOwUfz+YpNnPH3T/hidbXqoCciIahweLj7DKAnsJvA/OF9gSVAT3efXzXlSSwxM4b2yOHla3oDcN7jU/nn518Rq88aicQyPSQoEfH19mJueHE2k5cUBsbNGtSRuDg9mS4SaZXeVdfMmphZkzKvu5jZvWY27BBrlBqsfu0knhnWnav7t+aFz1dx8ytzKFVDukjUCGVsqxeAscBoM8sAPgTWAVeZWaa731sVBUrsio8zRpzSnloJcTw0cSkO3H1OJ+J1BSJS7YUSHkcDnweXBwMr3L2bmZ0N3AUoPOSQ/PqkdgCBAHG451wFiEh1F0p4pALfBJdPAN4ILucTGLJE5JD9+qR2mMHfPliK49x77jEKEJFqLJTw+BI4zsz+DZwE/CO4PhPYWtmFSc3zqxPbYRgPfrAEHO49TwEiUl2FEh4PAGOAbQSCZEpw/XHAvMotS2qqG05sixk88P4SQAEiUl1VODzc/SkzmwnkAO/7d318VwB/qoLapIb65U/aYsD9ChCRaiukmQTdfSYws9y61yu1IhHg+p8ErkDue28JDtynABGpVkIKDzNrT6CxvBHlnhFx9z9WYl0iXHdCW8yMe99djLtz/5DOChCRaqLC4WFm1wIPA1uADXw3MCLBZYWHVLprj28DwL3vLqa4pJS7z+lEWnJihKsSkVAGRrwFuBXIcPf27t6hzE9uRd7AzAaa2WIzW2ZmI/azvbmZTTSzOWY2ycyyy2y71MyWBn8uDaFuiXLXHt+GW0/twDvz/scpD33MtC83RbokkRovlPBIBSb4IQ6GZWbxBLr3ngLkAkPNrHzo3AeMdfdOwEjgzuCxDYDbCAzM2AO4zczqH0odEp2uPK4VL13dh/g44/xRU7n7nUUU7y2NdFkiNVYo4fE8cOZhfFYPYJm7r3D3YmA8MKjcPrkEhj0B+KjM9p8S6OG1yd2/Bt4HBh5GLRKFujWvz9u/7Mf5ec14bNJyzvrHFJas1yNGIpEQSoP5jcBrZnY8MAfYU3aju488yPFNgdVlXhcQuJIo6wsCQ588BJwNpAXH0drfsZr6tgaqXSuBu87pxAntGzHi1bmc/vdPGDGwPcP6tNCovCJhFEp4XEHgltNW4Ch+2GB+sPCoiBuBR4Ij9U4G1gAlFT3YzIYDwwFycnIqoRyprk4+6gi65NTn5lfmMPLNBXy0eAP3nnsMR9RLjnRpIjVCKLetbiMwY2C6u7dx97ZlftpV4Pg1fH8MrOzgum+5+1p3H+zuXQg0zuPumytybHDfUe6e5+55mZmZIZyaRKPMtFo8fWket5/dkfyVX/PTv03mzTlrI12WSI0QSngkAeMPtcEcmA60NbOWZpYEXAB87wFDM2toZvtqugUYHVx+FzjZzOoHG8pPDq6TGs7MuKhnc976ZV9aNKzNdS/M4jcvzmZH8d5IlyYS00IJj3EcRoO5u+8FriPwS38hgZ5b881spJnte98BwGIzWwJkAbcHj90E/IVAAE0HRgbXiQDQKrMOL1/dmxt+0pbXZq9h8KOf8lXR9kiXJRKzKjwNrZmNJPDLfxKBhu3yDeZ3VHZxh0PT0NZc/11SyC/HzcLdeXhoFwYc2SjSJYlEjYpOQxtKeHz5I5vd3VtVtLhwUHjUbKuKdjD8uXwWr9/KjScfyS8GtMZMvbFEDqZS5zA3s0QgGTjd3Vvu56daBYdITkYqr/6iD2d0asK97y7mmudnsm232kFEKkuFwsPd9xDojlvhbrMikZaalMBDF3Tm/07rwPsL13PWP6awvHBbpMsSiQmhNJg/BfyyqgoRqQpmxhX9WvHc5T3YtL2Ysx6ZwgcL1ke6LJGoF0p4NAEuMrNFZvZPMxtV9qeqChSpDH1aN+SN6wPdea8Ym8+D7y+htPRQe52LSCjh0ZrARFDrCARJ2zI/bSq/NJHK1TQ9hZeu7s3grk15aOJSrhybT+HW3ZEuSyQqVbi3VbRRbys5EHdn7NSvuP2thSQnxnHzKe0Z2j1HY2OJUMm9rURiiZlxaZ8W/OdX/TiqST1ufW0e5z7+KQvXfRPp0kSihsJDaqzWmXV44cqePDDkGFYW7eD0v3/CnW8v1NAmIhWg8JAazcwY3DWbD3/bn/O6ZfPE5BWc9MBk9cgSOQiFhwiQnprEXed04qWre1O7VjxXjM3nqufyWbdlZ6RLE6mWFB4iZXRv0YA3r+/HzQPb898lhZx4/3956uMVlKhbr8j3KDxEyklKiOOaAa15/9f96dGyAX99ayE3jJ/FnhLNmS6yj8JD5ACaNUhl9LDujDilPW/OWcfVz81g1x6N0CMCCg+RH2VmXN2/NX85qyMTF23g52Oms10DLIooPEQq4pJezXlgyDF8tqKIi5/+nC079hz8IJEYpvAQqaDBXbN59KKuzFuzhQue/IyN2zS0idRcCg+REAzs2JinLu3Olxu3MeSJqerKKzWWwkMkRP3bZTL25z3Z8M1uznt8quZKlxpJ4SFyCHq0bMALV/Zk++69nPf4VJau3xrpkkTCSuEhcog6Zafz4lW9cWDIE1OZW7Al0iWJhI3CQ+QwtMtK46WrepOalMCFT37GW3PWEavTHIiUpfAQOUwtGtbmpat706xBKte+MJPzn/iMOQWbI12WSJVSeIhUgibpKbxxfV/uOPtoVmzcxpmPTOE3L85WbyyJWQoPkUoSH2dc2DOHj24cwDUDWvPm3HUcf98kHnh/ieYIkZij8BCpZGnJidw8sD0Tf9OfEztk8fDEpQy4dxIv5a+mVKPzSowIa3iY2UAzW2xmy8xsxH6255jZR2Y2y8zmmNmpwfWJZvasmc01s4Vmdks46xY5FM0apPLIhV155Zo+NElP4Xcvz+GMRz7hsxVFkS5N5LBZuHqGmFk8sAQ4CSgApgND3X1BmX1GAbPc/TEzywXedvcWZnYhcKa7X2BmqcACYIC7rzzQ5+Xl5Xl+fn4VnpFIxZWWOm/MWcvd/1nE2i276NmyAed0y+bUoxtTp1ZCpMsT+ZaZzXD3vIPtF84rjx7AMndf4e7FwHhgULl9HKgbXK4HrC2zvraZJQApQDHwTdWXLFI54uKMQZ2b8uGNA7jllPZs2Lqbm16eQ/e/fsCvX5zNJ0s3asIpiSrh/CdPU2B1mdcFQM9y+/wJeM/MrgdqAycG179MIGjWAanAr919U5VWK1IFkhPjuap/a4Yf14qZqzbzyswC3vhiLa/NWkPjesmc3aUp53TLpnVmnUiXKvKjqtv18lBgjLvfb2a9gefMrCOBq5YSoAlQH/jYzD5w9xVlDzaz4cBwgJycnPBWLhICM6Nb8/p0a16fP56eywcL1/PKjAKemLyCRyct55hm6ZzbtSlnHtOUeqmJkS5X5AfC2ebRG/iTu/80+PoWAHe/s8w+84GB7r46+HoF0Au4DfjM3Z8Lrh8NvOPuEw70eWrzkGi0Yesu/j1rLa/MLGDR/7ZSOymei3s15/K+LWlUNznS5UkNUB3bPKYDbc2spZklARcAr5fbZxXwEwAz6wAkA4XB9ScE19cmECiLwlS3SNg0SkvmyuNa8Z8b+vHm9X05MTeLJz9eQd97PuLW1+ayqmhHpEsUAcJ45QEQ7Hr7NyAeGO3ut5vZSCDf3V8P9rB6EqhDoJH8Jnd/z8zqAM8AuYABz7j7vT/2WbrykFjxVdF2npi8gpfzCyhx54xOjblmQBuOPCIt0qVJDKrolUdYwyOcFB4Sa9Z/s4unP/mS5z/7ih3FJZzYIYtfHN+arjn1I12axBCFh8JDYtTX24t5dupKnpmyki0799C7VQbD+7fiuLaZxMdZpMuTKKfwUHhIjNu+ey/jpq1i1OQVbNi6m6y6tTirc1MGd83WLS05ZAoPhYfUELv3ljBx4QZenVnApMWF7C11jmpSl8FdsznzmCZkptWKdIkSRRQeCg+pgTZu280bX6zl1ZlrmLtmC/FxRv92mQzu2pQTO2SRnBgf6RKlmlN4KDykhluyfiuvzlzDv2at4X/f7CItOYFzu2Xz+1M7kBivAbVl/yoaHtXtCXMRqSTtstIYcUp7fvfTI5m6vIiXZqzmmSkr2Vlcwp2Dj8ZMjety6BQeIjEuPs7o27Yhfds2JKdBKn//cFlgytzj20S6NIliCg+RGuQ3J7Vj9aYd3PvuYrLrpzCoc9NIlyRRSuEhUoOYGXef24n/fbOL3700h6y6yfRqlRHpsiQKqdVMpIaplRDPExfnkZORyvCx+SzbsDXSJUkUUniI1ED1UhN5Zlh3khLiGfbMdAq37o50SRJlFB4iNVSzBqmMHpZH0bZiLn92OjuK90a6JIkiCg+RGqxTdjoPD+3CvDVb+OW42ZoKVypM4SFSw52Um8VtZxzFBwvX85c3FxCrDw5L5VJvKxHh0j4tWL1pB0998iXZ9VO4ol+rSJck1ZzCQ0QA+P2pHVizeSe3v72QpukpnHJ040iXJNWYbluJCABxccaD53emS7N0bnhxNn9+Yz6rN2naW9k/hYeIfCs5MZ6nLu3OaUc35rmpXzHgvklcP24Wcwu2RLo0qWY0qq6I7NfazTsZ8+lKXvh8Fdt276VXqwYMP64VA9o1Ik4zFsYsDcmu8BCpFN/s2sOL01YzesqXrNuyi7aN6nBlv1YM6tKEWgmaHyTWKDwUHiKVak9JKW/NWccTk1ewcN03NKxTi0t6NadJejIOuDvuUOrgOKWBlZQ6mEH/dpk0z6gd6dOQg9B8HiJSqRLj4zirS1MGdW7ClGVFjPp4BQ9+sKTCxyfEGUO6N+P6E9rQuF5KFVYq4aDwEJGQmH03P8iGrbvYvacUCPTWMiDODLPA1YZhxBls272Xpz/5knHTVvHyjAJ+1qs51wxoTUYdza8erXTbSkTCZvWmHTw0cSmvziwgJTGey/u25IrjWlE3OTHSpUmQ2jwUHiLV1rINW3nw/aW8NXcd9VISubp/a4b1aUFKkhrgI03hofAQqfbmrdnC/e8t5qPFhWSm1eK649twQY9m6sUVQdUyPMxsIPAQEA885e53ldueAzwLpAf3GeHubwe3dQKeAOoCpUB3d991oM9SeIhEj/yVm7jn3cVM+3ITackJnNQhi9M6NaZv24YKkjCrduFhZvHAEuAkoACYDgx19wVl9hkFzHL3x8wsF3jb3VuYWQIwE7jE3b8wswxgs7uXHOjzFB4i0cXdmbqiiNdmruG9BevZsnMPabUSOCk3i1OPbky/dgqScKiOXXV7AMvcfQXyC8hNAAAMIUlEQVSAmY0HBgELyuzjBK4sAOoBa4PLJwNz3P0LAHcvCkvFIhI2Zkaf1g3p07oht+8t5dPlG3lrzjreW7CeV2etIa1WAifmZnGagqRaCGd4NAVWl3ldAPQst8+fgPfM7HqgNnBicH07wM3sXSATGO/u91RtuSISKUkJcQw4shEDjmzEHSWlTFm2kbfnruPd+et5LRgk7Y5Io9SdktJyP+6UBv8sKXHaZKXxwJBjaKhuwZWquj3nMRQY4+73m1lv4Dkz60igzr5Ad2AHMDF4aTWx7MFmNhwYDpCTkxPeykWkSiTGfxckt59dyqfLi/jP3HWs2rSD+DgL/Jh9uxwXfJ0QZ2Dw9tx1nPvYp4z9eU9yMlIjfToxI5zhsQZoVuZ1dnBdWZcDAwHcfaqZJQMNCVylTHb3jQBm9jbQFfheeLj7KGAUBNo8quAcRCSCEuPj6N8uk/7tMit8zEU9m3P5s9MZ/NinjLmsOx2b1qvCCmuOcA7JPh1oa2YtzSwJuAB4vdw+q4CfAJhZByAZKATeBY42s9Rg43l/vt9WIiKyX92a1+flq3uTFG9cMOozPl22MdIlxYSwhYe77wWuIxAEC4EJ7j7fzEaa2ZnB3X4LXGlmXwDjgGEe8DXwAIEAmg3MdPe3wlW7iES3No3SePUXx9I0PYVLn5nGG1+sPfhB8qP0kKCI1BhbduzhyrH5TP9qE388PZfLjm0Z6ZKqnYp21dVMgiJSY9RLTWTs5T04qUMWf35jAXe/s4hY/Qd0VVN4iEiNkpwYz2MXd+PCnjk8Nmk5N740hz0lpZEuK+pUt666IiJVLj7OuP2sjjRKq8XfPljKpu27+cdFXUlNCvxKdHd27ilh8449wZ9iNu8MLBfvLaFnqwzaH5GGWc2djlfhISI1kpnxqxPbkZlWiz/8ax4nPziZ1KT4QFjs3EPx3h+/GmmansKJHRpxYm4WPVtmkJRQs27kKDxEpEa7qGdzstKSeXbqSlKT4qmfmkS91ETSU5JIT02kfmoi9b5dTqLUnclLCvlg4XpezF/Ns1O/ok6tBPq3y+TE3EYcf2Qj0lOTIn1aVU69rUREDtHO4hKmLNvIBwvXM3HRBgq37iY+zujWvD4ndgg8Fd+2UZ2our1V7UbVDTeFh4iEU2mpM2fNFiYuXM/7C9az6H9bAciqW4u+bTI5rl1Djm3TsNqPsaXwUHiISASt2byTT5YWMnnpRqYs28jmHXsAyG1cl37tGtKvTSZ5LeqTnFix0YHdPSxXMAoPhYeIVBMlpc78tVv4eOlGPl5ayIyvvmZPiVMrIY4eLRtQLyWRXXtK2LWnlJ17Sti1pyTwZ3EJu/aWsrO4hN17S+jeogEjB3XkyCPSqqxWhYfCQ0Sqqe279/L5l0V8vHQjU5cXUby3lOTEeJIT40hJiiclMZ5aiYE/kxPjSEmMx8yYkL+arbv2cnnfltzwk7bUrlX5fZ4UHgoPEYkxm7YXc887ixg/fTVH1E3mj2fkckrHIyr1dpaGJxERiTENaidx1zmdeOWaPtSvncQv/jmTS5+Zzpcbt4e9FoWHiEiU6da8Pm9cdyy3nZHLzK++5qcPTuaB95ewa09J2GpQeIiIRKGE+DguO7YlH/62PwM7HsHDE5dy8oOT+WjRhrB8vsJDRCSKNaqbzMNDu/DCFT1JiDcuGzOda/85k9LSqm3P1vAkIiIxoE+bhrxzw3E8+fEKdhaXEBdXtc+EKDxERGJEUkIc1x7fJiyfpdtWIiISMoWHiIiETOEhIiIhU3iIiEjIFB4iIhIyhYeIiIRM4SEiIiFTeIiISMhidkh2MysEvjqMt2gIbKykciIpVs4DdC7VVaycS6ycBxzeuTR398yD7RSz4XG4zCy/ImPaV3exch6gc6muYuVcYuU8IDznottWIiISMoWHiIiETOFxYKMiXUAliZXzAJ1LdRUr5xIr5wFhOBe1eYiISMh05SEiIiFTeJRjZgPNbLGZLTOzEZGu53CY2Uozm2tms80sP9L1hMLMRpvZBjObV2ZdAzN738yWBv+sH8kaK+oA5/InM1sT/G5mm9mpkayxIsysmZl9ZGYLzGy+md0QXB9138uPnEs0fi/JZjbNzL4Insufg+tbmtnnwd9lL5pZUqV+rm5bfcfM4oElwElAATAdGOruCyJa2CEys5VAnrtHXd91MzsO2AaMdfeOwXX3AJvc/a5gsNd395sjWWdFHOBc/gRsc/f7IllbKMysMdDY3WeaWRowAzgLGEaUfS8/ci5DiL7vxYDa7r7NzBKBT4AbgN8Ar7r7eDN7HPjC3R+rrM/Vlcf39QCWufsKdy8GxgODIlxTjeTuk4FN5VYPAp4NLj9L4C97tXeAc4k67r7O3WcGl7cCC4GmROH38iPnEnU8YFvwZWLwx4ETgJeD6yv9e1F4fF9TYHWZ1wVE6f9QQQ68Z2YzzGx4pIupBFnuvi64/D8gK5LFVILrzGxO8LZWtb/VU5aZtQC6AJ8T5d9LuXOBKPxezCzezGYDG4D3geXAZnffG9yl0n+XKTxiW1937wqcAlwbvH0SEzxwvzWa77k+BrQGOgPrgPsjW07FmVkd4BXgV+7+Tdlt0fa97OdcovJ7cfcSd+8MZBO4g9K+qj9T4fF9a4BmZV5nB9dFJXdfE/xzA/Aagf+potn64L3qffesN0S4nkPm7uuDf+FLgSeJku8meE/9FeCf7v5qcHVUfi/7O5do/V72cffNwEdAbyDdzBKCmyr9d5nC4/umA22DvRSSgAuA1yNc0yExs9rBhkDMrDZwMjDvx4+q9l4HLg0uXwr8O4K1HJZ9v2yDziYKvptgw+zTwEJ3f6DMpqj7Xg50LlH6vWSaWXpwOYVAh5+FBELk3OBulf69qLdVOcGueX8D4oHR7n57hEs6JGbWisDVBkAC8EI0nYuZjQMGEBgddD1wG/AvYAKQQ2DE5CHuXu0bog9wLgMI3BpxYCVwVZl2g2rJzPoCHwNzgdLg6t8TaCuIqu/lR85lKNH3vXQi0CAeT+CCYIK7jwz+DhgPNABmARe7++5K+1yFh4iIhEq3rUREJGQKDxERCZnCQ0REQqbwEBGRkCk8REQkZAoPkWrKzAaYmZtZdqRrESlP4SEiIiFTeIiISMgUHiIHYGbXm9kiM9sVnOjo1n1jBQUn2rrdzJ4ys2/MbKOZ3WFmcWWOTzOzJ8ys0Mx2m1m+mZ1c7jMamdkzZrY++DmLzezn5UrpYGaTzWxHcPKiU8q9x+/NbEXwMwrN7N3gMBUiVSbh4LuI1DzByZouA34FzAY6AI8DycAfgrtdT2Aom+4EBtB7nMDwIw8Ft48ObrsYWAVcDbxpZp3cfVHwF/x/gZ3ARcAKoA2B4STKug+4mcAw278HXjSz5u7+tZkNBkYEj/8ieOyAyvrvIHIgGp5EpBwzSwU2AoPd/Z0y638GPOzu6cFZGle7e78y2+8ALnH3ZmbWBlgKnObub5fZZyYw291/bmaXA/8A2rh7wX7qGEBgcLtz9o36amZZBObMGOju75rZr4FrgKPcfU/l/pcQOTDdthL5oaOAFOAVM9u27wd4AqhnZpnB/aaWO24KkG1mdYHc4LrJ5faZHHx/gG7Agv0FRzmz9y24+3qghO8mXJpAYOa4r8xsjJldsm80ZZGqpNtWIj+07x9V5xGY0768cI8YW7yfdXEQmLPFzNoDxxOYdvQPwN1m1tPdV+/nOJFKoSsPkR+aD+wCWrn7sv38lAT361XuuD7AmuCMdPOD68rP3ngc380RMQPIPdznONx9t7u/4+43AUcDqUTBPOIS3XTlIVKOu28Ltl/cYWYOfEDg78rRQBd3vzm4a+dgw/oLQB5wA8HGdHdfbmYvAY+a2VUE5rm4BugIXBg8fhxwE/C6md1EoEG8FdDQ3V+sSK3BdpM4YBqwGfgJkAYsOPT/AiIHp/AQ2Q93/4uZrQOuIzCP9U4Ct7DGlNnt70BzIB/YAzzCdz2tAK4A7gWeB+oSmHjodHdfFPyMHWbWH7iHwKQ9dQhMQHRXCKV+DdwYfI9aBHpsDXf3iSG8h0jI1NtK5BAEe1s95e5/jXQtIpGgNg8REQmZwkNEREKm21YiIhIyXXmIiEjIFB4iIhIyhYeIiIRM4SEiIiFTeIiISMgUHiIiErL/B5hl5+3N9kVuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Model Fit\n",
    "train_time = model_1m.fit(Xtr_1m, Xtst_1m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During training, we evauate the root mean squared error to have an idea of how learning is proceeding. Remember that in the RBM this is not the quantity being minimized, but plotting the rmse per epoch gives us a rough understanding of how learning is proceeding and how we should adjust the hyper parameters. Generally, we would like to see the rmse decrease monotonically as a function of the learning epochs. Even though you may be using an automated hyper parameter optimization method, I strongly suggest to spend some time to manually inspect the learning process; this will give you an idea of the value range to expect for the hyperparameters. Finally, note that most automated hyperparameters search methods are optimized for supervised learning, so they may not work as well for unsupervised tasks. \n",
    "\n",
    "The two final scores are the train/test mean average accuracies across the all set together with their difference. This has been defined as: \n",
    "\n",
    "$$ AC = \\frac{1}{m} \\sum_{\\mu=1}^{m} \\sum_{i=1}^{N_v} \\frac{1}{s_i} \\, I(v=vp)_{\\mu,i}, $$\n",
    "\n",
    "where $m$ = total number of users, $N_v$ = Total number of items $\\equiv$ number of visible units and $s_i$= the number of non-zero elements per row, i.e. the per user total number of ratings. \n",
    "Remember that for a model to generalize well, the difference between train and test metrics should not be too big. In order to visualize these online metrics, choose `with_metrics =True` in the `RBM()` model function. When evaluating metrics, the model takes a bit longer to run, but you need to do so only in the exploratory phase of your work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1.2 Model Evaluation\n",
    "\n",
    "To evaluate the model performance and compare it against the other algorithms in this repository, we use the `recommend_k_items()` method. Note that we pass 'maps' as a second argument in order to return the correct user/item IDs in a pandas dataframe format.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda/envs/reco_full/lib/python3.6/site-packages/numpy/core/fromnumeric.py:83: RuntimeWarning: invalid value encountered in reduce\n",
      "  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)\n",
      "Extracting top 10 elements\n",
      "Done recommending items, time 1.7780292\n"
     ]
    }
   ],
   "source": [
    "#number of top score elements to be recommended  \n",
    "K = 10\n",
    "\n",
    "#Model prediction on the test set Xtst. \n",
    "top_k_1m, test_time =  model_1m.recommend_k_items(Xtst_1m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "top_k returns the first K elements having the highest recommendation score. Here the recommendation score is evaluated by multiplying the predicted rating by its probability, i.e. the confidence the algorithm has about its output. So if we have two items both with predicted ratings 5, but one with probability 0.5 and the other 0.9, the latter will be considered more relevant. In order to inspect the prediction and use the evaluation metrics in this repository, we convert both top_k and Xtst to pandas dataframe format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_k_df_1m = am1m.map_back_sparse(top_k_1m, kind = 'prediction')\n",
    "test_df_1m = am1m.map_back_sparse(Xtst_1m, kind = 'ratings')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dataset</th>\n",
       "      <th>K</th>\n",
       "      <th>MAP</th>\n",
       "      <th>nDCG@k</th>\n",
       "      <th>Precision@k</th>\n",
       "      <th>Recall@k</th>\n",
       "      <th>Train time (s)</th>\n",
       "      <th>Test time (s)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mv 1m</td>\n",
       "      <td>10</td>\n",
       "      <td>0.379384</td>\n",
       "      <td>0.844124</td>\n",
       "      <td>0.747103</td>\n",
       "      <td>0.396285</td>\n",
       "      <td>10.956645</td>\n",
       "      <td>1.778029</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Dataset   K       MAP    nDCG@k  Precision@k  Recall@k  Train time (s)  \\\n",
       "0   mv 1m  10  0.379384  0.844124     0.747103  0.396285       10.956645   \n",
       "\n",
       "   Test time (s)  \n",
       "0       1.778029  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rating_1m= ranking_metrics(\n",
    "    data_size = \"mv 1m\",\n",
    "    data_true =test_df_1m,\n",
    "    data_pred =top_k_df_1m,\n",
    "    time_train=train_time,\n",
    "    time_test =test_time,\n",
    "    K =10)\n",
    "\n",
    "rating_1m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Formally, one should train the model until the cost function becomes flat but often an \"early stopping\" does the job. In the above example, we decided to train the algorithm to achieve higher ranking metrics. A faster optimization will do as well, but it will decrease the ranking metrics. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 100k Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 1.12.0\n"
     ]
    }
   ],
   "source": [
    "#100k\n",
    "model_100k = RBM(hidden_units= 600, training_epoch = 30, minibatch_size= 60,keep_prob= 0.9, with_metrics = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Creating the computational graph\n",
      "Initialize Gibbs protocol\n",
      "training epoch 0 rmse 0.943123\n",
      "training epoch 10 rmse 0.807337\n",
      "training epoch 20 rmse 0.786743\n",
      "training epoch 30 rmse 0.779240\n",
      "done training, Training time 2.4838602\n",
      "Train set accuracy 0.3806472\n",
      "Test set accuracy 0.3675472\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEOCAYAAACqzTG4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8nXXZx/HPld00SdM2aZsm6QBaumdaZhkitZSNiK2AggriI4iKIG4ffFQcj4NHEAERRaFWEAFlyh4Fmk5autKZ7nTP7Ov549wthzSkOe1ZSb7v1+u8zn3uda4fB/LlXr+fuTsiIiLRkpLoAkREpH1RsIiISFQpWEREJKoULCIiElUKFhERiSoFi4iIRJWCRUREoiquwWJmk8xsiZlVmNmtzSzva2YvmNl8M3vZzErCljWY2dzg9UTY/P5m9nawz7+ZWUa82iMiIoeyeD0gaWapwFLgbGAtMBOY6u7vha3zd+Bf7v4nM/sIcLW7Xxks2+PuOc3sdzrwD3efZmZ3A/Pc/XdxaJKIiDQjnsFyEvADd/9Y8PmbAO7+k7B1FgKT3L3SzAzY6e55wbJDgiVYpwro5e71Tb/jwxQUFHi/fv2i2DoRkfZv1qxZW9y98HDrpcWjmEAxUBn2eS1wQpN15gGXAL8BLgZyzay7u28FssysHKgHbnf3fwLdgR3uXh+2z+LDFdKvXz/Ky8uPqjEiIh2Nma1uzXrJdvH+68DpZjYHOB1YBzQEy/q6exnwKeDXZnZsJDs2s2vNrNzMyquqqqJatIiIvC+ewbIOKA37XBLMO8jd17v7Je4+Gvh2MG9H8L4ueF8BvAyMBrYC+WaW9mH7DNv3Pe5e5u5lhYWHPZITEZEjFM9gmQkMCO7iygCmAE+Er2BmBWZ2oKZvAvcH87uaWeaBdYBTgPc8dIHoJeDSYJvPAI/HvCUiIvKh4naNJbi4fj3wLJAK3O/uC83sNqDc3Z8AzgB+YmYOvAp8Kdh8MPB7M2skFIa3h91N9g1gmpn9DzAH+EO82iQiHUddXR1r166luro60aXEXFZWFiUlJaSnpx/R9nG7KyyZlJWVuS7ei0gkVq5cSW5uLt27dyd0Q2r75O5s3bqV3bt3079//w8sM7NZwbXuFiXbxXsRkaRUXV3d7kMFwMzo3r37UR2ZKVhERFqpvYfKAUfbTgVLBJ6ct56/vt2q27hFRDosBUsEnlmwkV89v5T6hsZElyIiHcyOHTu46667It5u8uTJ7NixIwYVfTgFSwTOG1HElj21vL1yW6JLEZEO5sOCpb6+vpm13/fUU0+Rn58fq7KapWCJwJmDetA5I5Un561PdCki0sHceuutLF++nFGjRjFu3DgmTJjABRdcwJAhQwC46KKLGDt2LEOHDuWee+45uF2/fv3YsmULq1atYvDgwVxzzTUMHTqUiRMnsn///pjUGs++wtq8rPRUJg7txdMLNnLbhcPISFMui3RE//3kQt5bvyuq+xzSO4/vnz/0Q5fffvvtLFiwgLlz5/Lyyy9z7rnnsmDBgoO3BN9///1069aN/fv3M27cOD7+8Y/TvXv3D+xj2bJlPPzww9x7771cdtllPProo1xxxRVRbQfoiCVi548sYuf+Ot6o2JLoUkSkAxs/fvwHnjO54447GDlyJCeeeCKVlZUsW7bskG369+/PqFGjABg7diyrVq2KSW06YonQqccV0qVTOk/OW8+Zg3okuhwRSYCWjizipXPnzgenX375Zf7zn/8wY8YMsrOzOeOMM5p9DiUzM/PgdGpqasxOhemIJUIZaSlMGtqL597bRHVdw+E3EBGJgtzcXHbv3t3ssp07d9K1a1eys7NZvHgxb731Vpyr+yAFyxE4f2Rv9tTU8/KSzYkuRUQ6iO7du3PKKacwbNgwbr755g8smzRpEvX19QwePJhbb72VE088MUFVhqivsCNQ39DICT9+gROP7c6dnxoTxcpEJFktWrSIwYMHJ7qMuGmuveorLIbSUlOYPLyIFxZtYm9Ny/eQi4h0NAqWI3T+yN5U1zXyn0WbEl2KiEhSUbAcobK+XemVl8W/5m9IdCkiEicd5dLB0bZTwXKEUlKMc0cU8cqSKnbur0t0OSISY1lZWWzdurXdh8uB8ViysrKOeB96juUonD+yN394fSXPLdzIJ8pKE12OiMRQSUkJa9eupaqqKtGlxNyBESSPlILlKIws6UJpt048OX+DgkWknUtPTz9kREVpnk6FHQUz47wRvXmjYgvb9tYmuhwRkaSgYDlK54/oTUOj8/QCXcQXEQEFy1EbXJTLsYWd1ZW+iEggrsFiZpPMbImZVZjZrc0s72tmL5jZfDN72cxKgvmjzGyGmS0Mln0ybJsHzGylmc0NXqPi3CbOG9Gbt1duY/OuQzt9ExHpaOIWLGaWCtwJnAMMAaaa2ZAmq/0C+LO7jwBuA34SzN8HfNrdhwKTgF+bWfiQaDe7+6jgNTemDWnG+SOLcId/v6vTYSIi8TxiGQ9UuPsKd68FpgEXNllnCPBiMP3SgeXuvtTdlwXT64HNQGFcqm6F43rkMrgoT6fDRESIb7AUA5Vhn9cG88LNAy4Jpi8Gcs3sA0Ogmdl4IANYHjb7R8Epsl+ZWSYJcP7IImav2cHa7fsS8fUiIkkj2S7efx043czmAKcD64CDg56YWRHwIHC1uzcGs78JDALGAd2AbzS3YzO71szKzaw8Fg84nTe8NwD/VhcvItLBxTNY1gHhTxGWBPMOcvf17n6Ju48Gvh3M2wFgZnnAv4Fvu/tbYdts8JAa4I+ETrkdwt3vcfcydy8rLIz+WbQ+3bMZWZrPk/N1OkxEOrZ4BstMYICZ9TezDGAK8ET4CmZWYGYHavomcH8wPwN4jNCF/UeabFMUvBtwEbAgpq1owfkjiliwbhcrqvYkqgQRkYSLW7C4ez1wPfAssAiY7u4Lzew2M7sgWO0MYImZLQV6Aj8K5l8GnAZc1cxtxX81s3eBd4EC4H/i06JDnTuiCEA9HotIh6YRJKPssrtnsGN/Lc999fSY7F9EJFE0gmSCnD+yiKWb9rBk4+5ElyIikhAKlig7Z3gRKYaeaRGRDkvBEmUFOZmcfGwBT85f3+4HBBIRaY6CJQYuGNWb1Vv3MXvNjkSXIiISdwqWGDhnWC+y0lN4bM7aRJciIhJ3CpYYyM1K52NDe/HkvA3U1DccfgMRkXZEwRIjl4wpYef+Ol5avDnRpYiIxJWCJUZOObY7PXIzeXT2usOvLCLSjihYYiQtNYWLRhfz0uLNbNtbm+hyRETiRsESQxePLqa+0fmXOqYUkQ5EwRJDg4vyGFyUp9NhItKhKFhi7ONjiplXuYOKzerxWEQ6BgVLjF0wqjcphp5pEZEOQ8ESYz1yszhtYCH/nLOexkZ18SIi7Z+CJQ4uHl3Muh37eXvltkSXIiIScwqWOJg4pBc5mWn8Y7ZOh4lI+6dgiYNOGalMHt6Lp97dwP5adfEiIu2bgiVOLhlTwt7aBp57b2OiSxERiSkFS5yM79eN4vxOeqZFRNo9BUucpKQYl4wp5vVlVWzeVZ3ockREYkbBEkcXjy6m0eHxueriRUTaLwVLHB1TmMOo0nwe1d1hItKOxTVYzGySmS0xswozu7WZ5X3N7AUzm29mL5tZSdiyz5jZsuD1mbD5Y83s3WCfd5iZxas9R+LjY4pZvHE3763flehSRERiIm7BYmapwJ3AOcAQYKqZDWmy2i+AP7v7COA24CfBtt2A7wMnAOOB75tZ12Cb3wHXAAOC16QYN+WonDeiN+mppmdaRKTdiucRy3igwt1XuHstMA24sMk6Q4AXg+mXwpZ/DHje3be5+3bgeWCSmRUBee7+lrs78Gfgolg35Gh07ZzBRwb14J9z11Pf0JjockREoi6ewVIMVIZ9XhvMCzcPuCSYvhjINbPuLWxbHEy3tE8AzOxaMys3s/KqqqojbkQ0XDy6hC17ani9YktC6xARiYVku3j/deB0M5sDnA6sA6LyqLq73+PuZe5eVlhYGI1dHrEzBxWSn53OP/RMi4i0Q/EMlnVAadjnkmDeQe6+3t0vcffRwLeDeTta2HZdMP2h+0xGmWmpnD+iN88u3Mju6rpElyMiElXxDJaZwAAz629mGcAU4InwFcyswMwO1PRN4P5g+llgopl1DS7aTwSedfcNwC4zOzG4G+zTwOPxaMzRumRMMTX1jTz9rrp4EZH2JW7B4u71wPWEQmIRMN3dF5rZbWZ2QbDaGcASM1sK9AR+FGy7DfghoXCaCdwWzAP4L+A+oAJYDjwdnxYdnVGl+RxT0FnPtIhIu2Ohm6k6lrKyMi8vL090Gfz2xWX84rmlPPuV0zi+V26iyxERaZGZzXL3ssOtl2wX7zuUK07sS25WGr94bkmiSxERiRoFSwLlZ2fwhdOO4fn3NjF7zfZElyMiEhUKlgS7+pT+FORk8LNnFtMRT0uKSPujYEmwzplpXH/mcby1YhuvLdMDkyLS9ilYksDUE/pQnN+Jnz+7REctItLmKViSQGZaKl87eyDvrtvJ0wv0XIuItG0KliRx0ehiBvbM4RfPLVHnlCLSpilYkkRqinHTxONZUbVXD02KSJumYEkiE4f0ZFRpPr/+zzKq66LS96aISNwpWJKImXHLpOPZsLOav7y1OtHliIgcEQVLkjn52AImDCjgzpcq1POxiLRJCpYkdPPHjmf7vjrue21loksREYmYgiUJjSjJ55xhvbjvtRVs3VOT6HJERCKiYElSN008nv11Ddz18vJElyIiEhEFS5I6rkcOl44t4cEZq1m3Y3+iyxERaTUFSxK78aMDAfjNf5YmuBIRkdZTsCSx4vxOXHFiXx6ZtZaKzXsSXY6ISKsoWJLcl848lk7pqfzyeQ0GJiJtg4IlyXXPyeTzE47hqXc3Mn/tjkSXIyJyWAqWNuDzE/rTrXMGP39WRy0ikvwULG1AblY6XzrzOF5btoU3KjQYmIgkt7gGi5lNMrMlZlZhZrc2s7yPmb1kZnPMbL6ZTQ7mX25mc8NejWY2Klj2crDPA8t6xLNN8XJ5MBiYhjAWkWQXt2Axs1TgTuAcYAgw1cyGNFntO8B0dx8NTAHuAnD3v7r7KHcfBVwJrHT3uWHbXX5gubtvjnljEiArPZWvfHQA89bu5BkNBiYiSSyeRyzjgQp3X+HutcA04MIm6ziQF0x3AdY3s5+pwbYdziVjShjQI4efazAwEUli8QyWYqAy7PPaYF64HwBXmNla4Cnghmb280ng4Sbz/hicBvuumVlzX25m15pZuZmVV1VVHVEDEi01xfj6xzQYmIgkt2S7eD8VeMDdS4DJwINmdrBGMzsB2OfuC8K2udzdhwMTgteVze3Y3e9x9zJ3LyssLIxdC2JMg4GJSLKLZ7CsA0rDPpcE88J9DpgO4O4zgCygIGz5FJocrbj7uuB9N/AQoVNu7ZaZ8Y1Jg9iws5oHZ2gwMBFJPhEFi5llmNkFZnaTmXUJ5vUzs/xWbD4TGGBm/c0sg1BIPNFknTXAWcF+BxMKlqrgcwpwGWHXV8wszcwKgul04DxgAe3cScd257SBhdz5cgW7NBiYiCSZVgeLmfUB5hM6Yvgp0D1Y9BXg9sNt7+71wPXAs8AiQnd/LTSz28zsgmC1m4BrzGxe8D1X+fv31p4GVLr7irDdZgLPmtl8YC6hI6B7W9umtuyWjx3Pjn113PvqisOvLCISR9baZyLM7FGgHvg0sAUY6e4rzOxM4PfuPjB2ZUZXWVmZl5eXJ7qMo3b9Q7N5YdFmXr3lTApzMxNdjoi0c2Y2y93LDrdeJKfCJgC3uXvTIQ1XcujdXRIHN008ntqGRn774rJElyIiclAkwdIJqG1mfiFQHZ1yJBL9CzrzyXGlPPTOGtZs3ZfockREgMiC5U1CtwMfcOAc2o3Aq1GrSCJy41kDSDHjVxoMTESSRCTB8i3gZjN7CEgDvmlmbwHnE+qKRRKgZ14WV5/Sn3/OXceiDbsSXY6ISOuDxd1nAScANcBy4FRgKXCCuy+MTXnSGl88/VhyM9P4hbrVF5EkkBbJyu7+HnB1jGqRI9QlO53rzjiWnz2zhJmrtjGuX7dElyQiHVgkz7H0NrPeYZ9Hm9nPzeyqmFQmEbn65P70yM3kp0+rW30RSaxIrrE8BEwCMLPuwIvAucAdZnZzDGqTCHTKSOXLZw2gfPV2XlnaNjvZFJH2IZJgGQ68HUxfAqxw9yHAZ4DPR7swidwnx5VSkJPJX95ak+hSRKQDiyRYsoEDtx19BHgymC7ng51LSoKkp6Zw6dgSXlqymU279GiRiCRGJMGyEjjNzHKAs4H/BPMLgd3RLkyOzCfHldLQ6DwyS+O1iEhiRBIsvwQeIDRY1xrgjWD+aXSAHoXbiv4FnTnxmG5Mm7mGxkZdxBeR+IvkOZb7CD3HcjUwIazX4RWERn6UJDF1fB8qt+1nxoqtiS5FRDqgiMZjcffZ7v5Pd98bNu8Jd38t+qXJkfrY0F506ZTOw+/oIr6IxF9ED0ia2SBCF+570CSU3P17UaxLjkJWeioXjy7mobfXsG1vLd06ZyS6JBHpQCJ5QPJLwELgfwiN/viJsNelMalOjtiU8aXUNjTyj9m6iC8i8RXJqbBvAt8Gurv7IHcfHPYaEqP65AgN6pXHqNJ8ps2s1JP4IhJXkT7HMt31V6rNmDq+lIrNe5i9ZnuiSxGRDiSSYPkLcMFh15Kkcd6I3nTOSOXhdyoTXYqIdCCRXLz/OvBYMMb9fKAufKG73xbNwuTodc5M44JRvfnnnPV87/wh5GWlJ7okEekAIgmWzwPnEHrKfijvjyBJMK1gSUJTxvXh4XcqeWLueq44sW+iyxGRDiCSU2HfJzRSZL67H+fuA8JeA1uzAzObZGZLzKzCzG5tZnkfM3vJzOaY2XwzmxzM72dm+81sbvC6O2ybsWb2brDPO8zMImhTuzeipAuDeuUybaaeaRGR+IgkWDKAaUd68d7MUoE7CR31DAGmmlnTu8m+Q+gGgdGEbmm+K2zZcncfFbyuC5v/O+AaYEDwmnQk9bVXZsbU8X1YsG4XC9btTHQ5ItIBRBIsD3N0F+/HAxXuvsLda4FpwIVN1nEgL5juAqxvaYdmVgTkuftbQeD9GbjoKGpsly4aVUxmWoqOWkQkLiK5xrIF+J6ZnQbM49CL9z8+zPbFhDqwPGAtob7Hwv0AeM7MbgA6Ax8NW9bfzOYQ6rr/O0E3MsXBfsL3Wdyq1nQgXbLTmTy8iMfnrOdbkweTnRFRhwsiIhGJ5IjlSmAnMBq4itDppwOvaA30NRV4wN1LgMnAg2aWAmwA+gSnyL4GPGRmeS3s5xBmdq2ZlZtZeVVVxxthccq4UnbX1PPUuxsTXYqItHOtChYzSweygPPcvX8zr2NasZt1fHBAsJJgXrjPAdMB3H1G8J0F7l7j7luD+bOA5cDAYPuSw+yTYLt73L3M3csKCwtbUW77Mr5/N44p6Mw0dUwpIjHWqmBx9zpC1z8ajuK7ZgIDzKy/mWUQujj/RJN11gBnAZjZYELBUmVmhcHFf8zsGEIX6Ve4+wZgl5mdGNwN9mng8aOosd0yMz45rpTy1dup2Kxx2UQkdiI5FXYf8OUj/SJ3rweuB54FFhG6+2uhmd1mZgduCrgJuMbM5hG6WeCq4KL8acB8M5sLPAJc5+7bgm3+K6itgtCRzNNHWmN79/GxJaSlGNP0JL6IxJC19u5hM7uPUE/GG4BZwN7w5e5+bdSri5GysjIvLy9PdBkJ8cW/zOKtFVt561tnkZmWmuhyRKQNMbNZ7l52uPUiOWI5FphNKFh68/5zIwOA446kSIm/KeP7sH1fHc+/tynRpYhIO9Xq+07d/cxYFiLxMeG4AorzOzHtnUrOG9E70eWISDsU0dDE0valpBiXlZXyesUWKrftS3Q5ItIOKVg6oMvGlZBicPMj89i8uzrR5YhIO6Ng6YCKunTiZ5eOZG7lDib/5nVeX7Yl0SWJSDuiYOmgLh1bwuNfOpWu2elcef/b/O9zS6hvaEx0WSLSDihYOrDje+Xy+PWn8ImxJfzfixV86t632bBzf6LLEpE2TsHSwWVnpPGzS0fyq0+OZMH6nUz+zWu8uFi3IovIkVOwCAAXjy7hXzecSq8unfjsA+X8+KlF1Nbr1JiIRE7BIgcdU5jDY/91Mlee2Jd7Xl3BZb+foVuSRSRiChb5gKz0VH540TDuunwMyzfvYfIdr/Hq0o43zICIHDkFizRr8vAinrpxAr3ysrjlkflU1x1Nx9Yi0pEoWORDlXbL5r8vGMrGXdUax0VEWk3BIi066djunNC/G3e+vFxHLSLSKgoWaZGZ8bWzB1K1u4a/vLU60eWISBugYJHDOuGY7px6XAF3v7KcfbX1iS5HRJKcgkVa5atnD2DLnloenKGjFhFpmYJFWmVs326cPrCQu19Zzp4aHbWIyIdTsEirffXsgWzfV8ef3lyV6FJEJIkpWKTVRpXmc9agHtzz6gp2VdcluhwRSVIKFonIV88eyM79dfzx9VWJLkVEklRcg8XMJpnZEjOrMLNbm1nex8xeMrM5ZjbfzCYH8882s1lm9m7w/pGwbV4O9jk3ePWIZ5s6mmHFXZg4pCf3vb6Cnft01CIih4pbsJhZKnAncA4wBJhqZkOarPYdYLq7jwamAHcF87cA57v7cOAzwINNtrvc3UcFr80xa4QAoaOW3dX1/OH1FYkuRUSSUDyPWMYDFe6+wt1rgWnAhU3WcSAvmO4CrAdw9znuvj6YvxDoZGaZcahZmjG4KI9zhxdx/xur2L63NtHliEiSiWewFAOVYZ/XBvPC/QC4wszWAk8BNzSzn48Ds929JmzeH4PTYN81M4tizfIhbvzoAPbW1nPvazpqEZEPSraL91OBB9y9BJgMPGhmB2s0s6HAT4EvhG1zeXCKbELwurK5HZvZtWZWbmblVVXqBv5oDeyZy/kjevPAm6vYuqfm8BuISIcRz2BZB5SGfS4J5oX7HDAdwN1nAFlAAYCZlQCPAZ929+UHNnD3dcH7buAhQqfcDuHu97h7mbuXFRYWRqVBHd2XzxpAdV0D97yqoxYReV88g2UmMMDM+ptZBqGL8080WWcNcBaAmQ0mFCxVZpYP/Bu41d3fOLCymaWZ2YHgSQfOAxbEvCUCwHE9crhoVDF/mrGKqt06ahGRkLgFi7vXA9cDzwKLCN39tdDMbjOzC4LVbgKuMbN5wMPAVe7uwXbHAd9rcltxJvCsmc0H5hI6Aro3Xm0SuOGsAdQ1OHe/svzwK4tIh2Chv9sdS1lZmZeXlye6jHbj5r/P44l563n1ljPpmZeV6HJEJEbMbJa7lx1uvWS7eC9t0A0fGUBDo/PL55bSEf9HRUQ+SMEiR61P92yuOrkffyuv5FuPLaC+oTHRJYlIAqUlugBpH741eTCZ6Snc+dJy1u/Yz52XjyEnU/96iXREOmKRqEhJMW7+2CB+cslwXq/YwmV3z2DTrupElyUiCaBgkaiaOr4Pf/hMGau37uWiO99g8cZdiS5JROJMwSJRd8bxPZh+3Uk0uvOJ383g9WVbEl2SiMSRgkViYmjvLjz2X6dQ3LUTV/3xHaaXVx5+IxFpFxQsEjO98zvx9+tO4qRju3PLI/P55XNLdDuySAegYJGYys1K5/6rxnFZWQl3vFjBTdPnUVuv25FF2jPdDyoxl56awk8/PoLSrtn87/NLWbtjP7+ZMoqiLp0SXZqIxICOWCQuzIwbzhrAb6aM4t21O5n069f41/z1h99QRNocBYvE1YWjinnqxgn0L+jM9Q/N4at/m8uu6rpElyUiUaRgkbjrX9CZR647ia98dABPzFvPOb9+jbdXbE10WSISJQoWSYi01BS+8tGBPHLdSaSnGlPufYufPL2ImvqGRJcmIkdJwSIJNbpPV/795QlMGVfK719ZwcV3vsnSTbsTXZaIHAWNxyJJ4/n3NnHro/PZXVPPrZMGcdXJ/UhJMQDqGxrZsqeWjbuq2RS8Nu6sZtOuGmrqG/ja2QM5pjAnwS0Qad9aOx6LgkWSStXuGr7x6HxeXLyZIUV5pKUaG3dWs2VPDY1N/lVNSzF65Gayu7qezPRU/vr5Ezi+V25iChfpABQsLVCwJDd356F31vC3mZXkZ2fQMzeTXl2y6JmXRa+80HvPLpkUdM4kJcWo2LyHy+97i9r6Rh783AkMK+6S6CaItEsKlhYoWNqf1Vv38ql732ZXdR0PXD2esX27JrokkXZHQxNLh9K3e2emX3cS3TtncOUf3mbGct2+LJIoChZpN4rzOzH9CydRnB/qUfmVpVWJLkmkQ4prsJjZJDNbYmYVZnZrM8v7mNlLZjbHzOab2eSwZd8MtltiZh9r7T6lY+mRl8W0a0/k2MIcrvlTOc8t3JjokkQ6nLgFi5mlAncC5wBDgKlmNqTJat8Bprv7aGAKcFew7ZDg81BgEnCXmaW2cp/SwXTPyeTha05kSO88vvjX2Tw5T32SicRTPI9YxgMV7r7C3WuBacCFTdZxIC+Y7gIc+ItwITDN3WvcfSVQEeyvNfuUDqhLdjp/+fwJjO3blRunzeHvGmhMJG7iGSzFQPh/3WuDeeF+AFxhZmuBp4AbDrNta/YpHVROZhp/uno8pxxXwM2PzOfBGasSXZJIh5Bs47FMBR5w9/81s5OAB81sWDR2bGbXAtcC9OnTJxq7lDagU0Yq9366jOsfms13H1/ID/+1CDNIMSPFICXF3p82w8xISzEuHNWbmyYeT0aa7m8RiVQ8g2UdUBr2uSSYF+5zhK6h4O4zzCwLKDjMtofbJ8H+7gHugdBzLEfWBGmLstJT+d0VY/nzjNWhJ/gbnUZ3Gh0a3fHg/cC8Lbtr+P2rK3hr5TZ+O3U0pd2yE90EkTYlnsEyExhgZv0J/fGfAnyqyTprgLOAB8xsMJAFVAFPAA+Z2S+B3sAA4B3AWrFPEdJTU/jcqf1bvf7T727glkfnM/mO1/j5pSOYNKwohtWJtC9xO85393rgeuBZYBGhu78WmtltZnZBsNpNwDVmNg94GLjKQxYC04H3gGeAL7l7w4ftM15tkvbrnOFF/PuGCRxT0Jnr/jKb7z++QF36i7SSunSFcTIcAAAPiUlEQVQRaUFtfSO3P72Y+99YybDiPH47dQz9CjonuiyRhFCXLiJRkJGWwvfOH8K9ny6jctt+zvu/1/nX/Miei2lodBqbds0s0o4l211hIknp7CE9+feXT+XLD8/h+ofm8ObyrXzvvCFkpaceXKex0Vm3Yz9LN+1m6aY9wftuKjbvISczjctP7MsVJ/ahR25WAlsiEns6FSYSgbqGRn7x3BJ+/8oKBvXK5eLRxVRsDoXIss172Ff7/nWYXnlZDOyVy8AeOazaupcXFm8mPSWF80f25rOn9mNob3XvL22Lus1vgYJFjtZLizfztelz2b6vjoKcTAb2zGFgz1wG9szl+F45HNcjly6d0j+wzcote/nTm6uYXl7JvtoGTujfjc+e2p+PDu5JajBSZjJpbHT21tazr7aBPTX17KtpYH9dA4OLcsnNSj/8DqTdUbC0QMEi0bC/NvSHtlvnjIi227m/jukzK3ngzVWs27Gf0m6duOrk/lxWVvKBP9h7a+qp3L6Pym37qdy2L5gOfd5VXceIki6M69eNE/p3Z3BRLmmpkV0y3ba3ltmrtzNrzXbmr93Btr117KutZ29NA3tr6tlf1/xdcIW5mXz//CGcO7wIs+QLRIkdBUsLFCySDOobGnn+vU384fWVlK/eTk5mGicd253Nu2tYu20fW/fWfmD97IxUSrtmU9otm+yMVOZUbqdy234g1H3NmL5dOaF/N8b168aIki4fuP7T0Ogs3bSb2Wu2M2v1duas2cHKLXuB0BDPg4vy6JmXSefMNLIz0sjJTCU7I43Omal0zkyjc0YanTPTaHTn/15cxoJ1uzjz+EJ+eNEwSrrqAdKOQsHSAgWLJJt5lTv44xsrmb92J73zO1HarROl3bIPBklp105065xxyBHChp37eWflNmau2sY7K7exdNMeIHQ326iSfIb0zqNi8x7mVu5gT009AAU5GYzp05Uxfbsypk/XQ0LocOobGnngzVX88vmluMPXzh7I1af0i/iISdoeBUsLFCzSXm3fW0v56u28s3Ir76zazuINuzi2MIexfbsypm8+Y/t0o7Rbp6icwlq7fR/ff3whLyzezNDeefzkkuGMKMmPQiskWSlYWqBgEYkOd+fpBRv5wRML2bKnhs+c3I+bJh5PTmbzTzLs3FfH/HU7mFe5g3lrdzKvcgcOTBzSk3OGFXHCMd1I15FP0lKwtEDBIhJdu6rr+PkzS/jL26vplZfFbRcOY8KAAhau38ncyp3MXxsKk1Vb9x3c5pjCzowqyaemvpEXF29mf10D+dnpnD24J5OHF3Hycd3JTGv9KTqJPQVLCxQsIrExa/V2vv3YuyzeuJvUFKMh6HGgV14WI0u7MKIkn1Gl+Qwr7vKB27H31zbwytIqnlmwgRcWbWZ3TT25mWmcNbgHk4YVccbxhQevA9U1NLJxZzWV2/axdvt+KrcH78HnPTX1nNC/G2cO6sGZg3pQnN8pIf8s2iMFSwsULCKxU9fQyENvr2Hz7mpGluQzsjSfnnmt722gpr6BNyu28tS7G3h+0SZ27KujU3oqg4ty2bSrho27qg8GFkCKQVGXTpR07URJ12wy01N4fdkW1mwLHR0N6pXLGcf34CODejCmT75uMjgKCpYWKFhE2oa6hkbeXrGNpxdsYNmmPfTOz6K0WzYlXTtR2jWbkq7ZFOVnHXJdxt1ZXrWXlxZv5sXFm5m5ahv1jU6XTumcNrCQjwwqZMKAQrIzUqlrcBoanfqGRuoanYYGp66xkfoGp76xEXc4rkdORHfOtVcKlhYoWEQ6ll3Vdby+bAsvLt7My0uq2LKnJqLtczLT+OjgHpw7ojcTBhR02JBRsLRAwSLScTU2OgvW7+SdldtoaHRSU4z01JTg3UhLSSEteD9wnejVpVU8s3AjO/fXkZuZxkeH9OTc4UVMGFjQoW4wULC0QMEiIpGqa2jkzeVb+ff89Ty7cNPBkDl7aChkTh3wwZCpa2hkb009u6vr2VMTvKrr2V1TT01dAw4Q/Pl1QkNkOwTvoc/52emM7tOV3l2ykqL7HAVLCxQsInI0ausbeXP5Fv49fwPPvReETFYaBTmZQZDUUV3XGLXv65mXyejS0EOuY/p0ZVhxZL0lRIuCpQUKFhGJltr6Rt5YvoXnFm5kT00DOZlp5GalkZMZvLLSyMtKIycznZysUD9s4Uc2ZmBm2IFpLHiHTbtqmFO5ndmrtzN7zY6Dd7qlpxpDendhTJ98RvfpysiSLhTnd4r5HW8KlhYoWESkLaraXcOcNaGQmbNmO/PX7jzYC3VqilHUJYs+B/uYC/qbCz4X5Bza11ykWhssGkFSRKSNKMzNZOLQXkwc2gsIXcdZsnE3763fReX2fazZFhpa4cUlm6na/cE73zqlp1LStRO/v3IsxxTmxLROBYuISBuVnprCsOIuDCs+dDTS/bUNrN0eGsdnzdZ9VAa9E+RnRzZ+0JGIa7CY2STgN0AqcJ+7395k+a+AM4OP2UAPd883szOBX4WtOgiY4u7/NLMHgNOBncGyq9x9bgybISKS9DplpDKgZy4DeubG/bvjFixmlgrcCZwNrAVmmtkT7v7egXXc/ath698AjA7mvwSMCuZ3AyqA58J2f7O7PxLzRoiIyGHFs9Oc8UCFu69w91pgGnBhC+tPBR5uZv6lwNPuvq+ZZSIikmDxDJZioDLs89pg3iHMrC/QH3ixmcVTODRwfmRm883sV2aWGY1iRUTkyCRrN59TgEfcvSF8ppkVAcOBZ8Nmf5PQNZdxQDfgG83t0MyuNbNyMyuvqqqKTdUiIhLXYFkHlIZ9LgnmNae5oxKAy4DH3L3uwAx33+AhNcAfCZ1yO4S73+PuZe5eVlhYeEQNEBGRw4tnsMwEBphZfzPLIBQeTzRdycwGAV2BGc3s45DrLsFRDBZ68uciYEGU6xYRkQjE7a4wd683s+sJncZKBe5394VmdhtQ7u4HQmYKMM2bdAlgZv0IHfG80mTXfzWzQkI9IMwFrotdK0RE5HDUpYuIiLSK+gprgZlVAauPcPMCYEsUy0mk9tKW9tIOUFuSVXtpy9G2o6+7H/YidYcMlqNhZuWtSey2oL20pb20A9SWZNVe2hKvdiTr7cYiItJGKVhERCSqFCyRuyfRBURRe2lLe2kHqC3Jqr20JS7t0DUWERGJKh2xiIhIVClYImBmk8xsiZlVmNmtia7nSJnZKjN718zmmlmbeqDHzO43s81mtiBsXjcze97MlgXvXRNZY2t9SFt+YGbrgt9mrplNTmSNrWFmpWb2kpm9Z2YLzezGYH6b+11aaEtb/F2yzOwdM5sXtOW/g/n9zezt4O/Y34KeUKL73ToV1jrBeDJLCRtPBpgaPp5MW2Fmq4Ayd29z9+Wb2WnAHuDP7j4smPczYJu73x4Efld3b7Yz0mTyIW35AbDH3X+RyNoiEXSrVOTus80sF5hFqHulq2hjv0sLbbmMtve7GNDZ3feYWTrwOnAj8DXgH+4+zczuBua5+++i+d06Ymm9SMeTkRhw91eBbU1mXwj8KZj+E6E/BEnvQ9rS5gQdwc4OpncDiwgNidHmfpcW2tLmBJ3z7gk+pgcvBz4CHBgYMSa/i4Kl9Vo9nkwb4MBzZjbLzK5NdDFR0NPdNwTTG4GeiSwmCq4Pxhe6vy2cPgoX9Ok3GnibNv67NGkLtMHfxcxSzWwusBl4HlgO7HD3+mCVmPwdU7B0TKe6+xjgHOBLwSmZdiHovLQtn9/9HXAsoaG4NwD/m9hyWs/McoBHga+4+67wZW3td2mmLW3yd3H3BncfRWiYkvGExq6KOQVL60UynkxSc/d1wftm4DE+ZAybNmRT2PAJRYT+76xNcvdNwR+DRuBe2shvE5zDfxT4q7v/I5jdJn+X5trSVn+XA9x9B/AScBKQb2YHeraPyd8xBUvrtWo8mWRnZp2Di5KYWWdgIm1/DJsngM8E058BHk9gLUflwB/iwMW0gd8muEj8B2CRu/8ybFGb+10+rC1t9HcpNLP8YLoToRuPFhEKmEuD1WLyu+iusAgEtxj+mvfHk/lRgkuKmJkdQ+goBULj8TzUltphZg8DZxDqpXUT8H3gn8B0oA+hXqsvc/ekvyj+IW05g9DpFgdWAV8Iu06RlMzsVOA14F2gMZj9LULXJtrU79JCW6bS9n6XEYQuzqcSOoiY7u63BX8DphEayn0OcEUwAm/0vlvBIiIi0aRTYSIiElUKFhERiSoFi4iIRJWCRUREokrBIiIiUaVgEWljzOwMM3MzK0l0LSLNUbCIiEhUKVhERCSqFCwiETKzG8xssZlVB4NYfftA30vBIGo/MrP7zGyXmW0xsx+bWUrY9rlm9nszqzKzGjMrN7OJTb6jh5n90cw2Bd+zxMw+26SUwWb2qpntCwamOqfJPr5lZiuC76gys2eDrj1EYirt8KuIyAHBQFxXA18B5gKDgbuBLOC7wWo3EOr6ZxyhzgrvJtRly2+C5fcHy64A1gDXAf8ysxHuvjj44/8KsB+4HFgBHEeoC45wvwC+Qagr9G8BfzOzvu6+3cwuAW4Ntp8XbHtGtP45iLREXbqItJKZZQNbgEvc/Zmw+Z8G7nD3/GB0zkp3nxC2/MfAle5eambHAcuAc939qbB1ZgNz3f2zZvY54E7gOHdf20wdZxDqSPDjB3rfNbOehMY8meTuz5rZV4EvAkPdvS66/yREWqZTYSKtNxToBDxqZnsOvIDfA13MrDBYb0aT7d4ASswsDxgSzHu1yTqvBvsHGAu811yoNDH3wIS7bwIaeH8wremERgxcbWYPmNmVB3q1Fok1nQoTab0D/yP2CWBpM8vj3XNvbTPzUiA05o6ZDQLOJDQU7XeBn5rZCe5e2cx2IlGjIxaR1lsIVAPHuHtFM6+GYL0Tm2x3MrAuGIlwYTCv6aidp/H+GB+zgCFH+5yKu9e4+zPufgswHMimDYw7L22fjlhEWsnd9wTXS35sZg78h9B/Q8OB0e7+jWDVUcFF/oeAMuBGggv77r7czP4O3GVmXyA0TskXgWHAp4LtHwZuAZ4ws1sIXZw/Bihw97+1ptbgOk0K8A6wAzgLyAXeO/J/AiKto2ARiYC7/9DMNgDXExr3fD+h02IPhK32f0BfoByoA37L+3eEAXwe+DnwFyCP0KBS57n74uA79pnZ6cDPCA3IlENocKnbIyh1O/D1YB+ZhO4su9bdX4hgHyJHRHeFiURRcFfYfe7+P4muRSRRdI1FRESiSsEiIiJRpVNhIiISVTpiERGRqFKwiIhIVClYREQkqhQsIiISVQoWERGJKgWLiIhE1f8DWJtknTJ6aFUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_time = model_100k.fit(Xtr_100k, Xtst_100k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Extracting top 10 elements\n",
      "Done recommending items, time 0.1804592\n"
     ]
    }
   ],
   "source": [
    "#Model prediction on the test set Xtst. \n",
    "top_k_100k, test_time =  model_100k.recommend_k_items(Xtst_100k)\n",
    "\n",
    "#to df\n",
    "top_k_df_100k = am100k.map_back_sparse(top_k_100k, kind = 'prediction')\n",
    "test_df_100k = am100k.map_back_sparse(Xtst_100k, kind = 'ratings')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.1 Model evaluation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dataset</th>\n",
       "      <th>K</th>\n",
       "      <th>MAP</th>\n",
       "      <th>nDCG@k</th>\n",
       "      <th>Precision@k</th>\n",
       "      <th>Recall@k</th>\n",
       "      <th>Train time (s)</th>\n",
       "      <th>Test time (s)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mv 100k</td>\n",
       "      <td>10</td>\n",
       "      <td>0.40663</td>\n",
       "      <td>0.818305</td>\n",
       "      <td>0.698621</td>\n",
       "      <td>0.432109</td>\n",
       "      <td>2.48386</td>\n",
       "      <td>0.180459</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Dataset   K      MAP    nDCG@k  Precision@k  Recall@k  Train time (s)  \\\n",
       "0  mv 100k  10  0.40663  0.818305     0.698621  0.432109         2.48386   \n",
       "\n",
       "   Test time (s)  \n",
       "0       0.180459  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_100k= ranking_metrics(\n",
    "    data_size = \"mv 100k\",\n",
    "    data_true =test_df_100k,\n",
    "    data_pred =top_k_df_100k,\n",
    "    time_train=train_time,\n",
    "    time_test =test_time,\n",
    "    K=10) \n",
    "\n",
    "eval_100k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python (reco_full)",
   "language": "python",
   "name": "reco_full"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "nteract": {
   "version": "0.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}