notebooks/02_model/baseline_deep_dive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Estimating Baseline Performance\n",
    "<br>\n",
    "Estimating baseline performance is as important as choosing right metrics for model evaluation. In this notebook, we briefly discuss about why do we care about baseline performance and how to measure it.\n",
    "\n",
    "The notebook covers two example scenarios under the context of movie recommendation: 1) rating prediction and 2) top-k recommendation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why does baseline performance matter? \n",
    "<br>\n",
    "Before we go deep dive into baseline performance estimation, it is worth to think about why we need that.\n",
    "\n",
    "As we can simply see from the definition of the word 'baseline', <b>baseline performance</b> is a minimum performance we expect to achieve by a model or starting point used for model comparisons.\n",
    "\n",
    "Once we train a model and get results from evaluation metrics we choose, we will wonder how should we interpret the metrics or even wonder if the trained model is better than a simple rule-based model. Baseline results help us to understand those.\n",
    "\n",
    "Let's say we are building a food recommender. We evaluated the model on the test set and got nDCG (at 10) = 0.3. At that moment, we would not know if the model is good or bad. But once we find out that a simple rule of <i>'recommending top-10 most popular foods to all users'</i> can achieve nDCG = 0.4, we see that our model is not good enough. Maybe the model is not trained well, or maybe we should think about if nDCG is the right metric for prediction of user behaviors in the given problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How can we estimate the baseline performance?\n",
    "<br>\n",
    "To estimate the baseline performance, we first pick a baseline model and evaluate it by using the same evaluation metrics we will use for our main model. In general, a very simple rule or even <b>zero rule</b>--<i>predicts the mean for regression or the mode for classification</i>--will be a enough as a baseline model (Random-prediction might be okay for certain problems, but usually it performs poor than the zero rule). If we already have a running model in hand and now trying to improve that, we can use the previous results as a baseline performance for sure.\n",
    "\n",
    "Most importantly, <b>different baseline approaches should be taken for different problems and business goals</b>. For example, recommending the previously purchased items could be used as a baseline model for food or restaurant recommendation since people tend to eat the same foods repeatedly. For TV show and/or movie recommendation, on the other hand, recommending previously watched items does not make sense. Probably recommending the most popular (most watched or highly rated) items is more likely useful as a baseline.\n",
    "\n",
    "In this notebook, we demonstrate how to estimate the baseline performance for the movie recommendation with MovieLens dataset. We use the mean for rating prediction, i.e. our baseline model will predict a user's rating of a movie by averaging the ratings the user previously submitted for other movies. For the top-k recommendation problem, we use top-k most-rated movies as the baseline model. We choose the number of ratings here because we regard the binary signal of 'rated vs. not-rated' as user's implicit preference when evaluating ranking metrics.\n",
    "\n",
    "Now, let's jump into the implementation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.8 |Anaconda, Inc.| (default, Feb 11 2019, 15:03:47) [MSC v.1915 64 bit (AMD64)]\n",
      "Pandas version: 0.24.1\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "\n",
    "import itertools\n",
    "import pandas as pd\n",
    "\n",
    "from reco_utils.common.notebook_utils import is_jupyter\n",
    "from reco_utils.dataset import movielens\n",
    "from reco_utils.dataset.python_splitters import python_random_split\n",
    "from reco_utils.dataset.pandas_df_utils import filter_by\n",
    "from reco_utils.evaluation.python_evaluation import (\n",
    "    rmse, mae, rsquared, exp_var,\n",
    "    map_at_k, ndcg_at_k, precision_at_k, recall_at_k\n",
    ")\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Pandas version: {}\".format(pd.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's prepare training and test data sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "MOVIELENS_DATA_SIZE = '100k'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserId</th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3.0</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3.0</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1.0</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2.0</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1.0</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserId  MovieId  Rating  Timestamp\n",
       "0     196      242     3.0  881250949\n",
       "1     186      302     3.0  891717742\n",
       "2      22      377     1.0  878887116\n",
       "3     244       51     2.0  880606923\n",
       "4     166      346     1.0  886397596"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = movielens.load_pandas_df(\n",
    "    size=MOVIELENS_DATA_SIZE,\n",
    "    header=['UserId', 'MovieId', 'Rating', 'Timestamp']\n",
    ")\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = python_random_split(data, ratio=0.75, seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Rating prediction baseline\n",
    "\n",
    "As we discussed earlier, we use each user's **mean rating** as the baseline prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserId</th>\n",
       "      <th>AvgRating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>3.696970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3.837209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>2.744186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>4.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2.868217</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserId  AvgRating\n",
       "0       1   3.696970\n",
       "1       2   3.837209\n",
       "2       3   2.744186\n",
       "3       4   4.500000\n",
       "4       5   2.868217"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Calculate avg ratings from the training set\n",
    "users_ratings = train.groupby(['UserId'])['Rating'].mean()\n",
    "users_ratings = users_ratings.to_frame().reset_index()\n",
    "users_ratings.rename(columns = {'Rating': 'AvgRating'}, inplace = True)\n",
    "\n",
    "users_ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserId</th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>AvgRating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12215</th>\n",
       "      <td>1</td>\n",
       "      <td>233</td>\n",
       "      <td>2.0</td>\n",
       "      <td>878542552</td>\n",
       "      <td>3.69697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12216</th>\n",
       "      <td>1</td>\n",
       "      <td>159</td>\n",
       "      <td>3.0</td>\n",
       "      <td>875073180</td>\n",
       "      <td>3.69697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12217</th>\n",
       "      <td>1</td>\n",
       "      <td>238</td>\n",
       "      <td>4.0</td>\n",
       "      <td>875072235</td>\n",
       "      <td>3.69697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12218</th>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>5.0</td>\n",
       "      <td>878543541</td>\n",
       "      <td>3.69697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12219</th>\n",
       "      <td>1</td>\n",
       "      <td>63</td>\n",
       "      <td>2.0</td>\n",
       "      <td>878543196</td>\n",
       "      <td>3.69697</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       UserId  MovieId  Rating  Timestamp  AvgRating\n",
       "12215       1      233     2.0  878542552    3.69697\n",
       "12216       1      159     3.0  875073180    3.69697\n",
       "12217       1      238     4.0  875072235    3.69697\n",
       "12218       1      100     5.0  878543541    3.69697\n",
       "12219       1       63     2.0  878543196    3.69697"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Generate prediction for the test set\n",
    "baseline_predictions = pd.merge(test, users_ratings, on=['UserId'], how='inner')\n",
    "\n",
    "baseline_predictions.loc[baseline_predictions['UserId'] == 1].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's evaluate how our baseline model will perform on regression metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE:\t\t1.044885\n",
      "MAE:\t\t0.836925\n",
      "rsquared:\t0.136491\n",
      "exp var:\t0.136496\n"
     ]
    }
   ],
   "source": [
    "baseline_predictions = baseline_predictions[['UserId', 'MovieId', 'AvgRating']]\n",
    "\n",
    "cols = {\n",
    "    'col_user': 'UserId',\n",
    "    'col_item': 'MovieId',\n",
    "    'col_rating': 'Rating',\n",
    "    'col_prediction': 'AvgRating',\n",
    "}\n",
    "\n",
    "eval_rmse = rmse(test, baseline_predictions, **cols)\n",
    "eval_mae = mae(test, baseline_predictions, **cols)\n",
    "eval_rsquared = rsquared(test, baseline_predictions, **cols)\n",
    "eval_exp_var = exp_var(test, baseline_predictions, **cols)\n",
    "\n",
    "print(\"RMSE:\\t\\t%f\" % eval_rmse,\n",
    "      \"MAE:\\t\\t%f\" % eval_mae,\n",
    "      \"rsquared:\\t%f\" % eval_rsquared,\n",
    "      \"exp var:\\t%f\" % eval_exp_var, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, our baseline model actually performed quite well on the metrics. E.g. MAE (Mean Absolute Error) was around 0.84 on MovieLens 100k data, saying that users actual ratings would be within +-0.84 of their mean ratings. This also gives us an insight that users' rating could be biased where some users tend to give high ratings for all movies while others give low ratings.\n",
    "\n",
    "Now, next time we build our machine-learning model, we will want to make the model performs better than this baseline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Top-k recommendation baseline\n",
    "\n",
    "Recommending the **most popular items** is intuitive and simple approach that works for many of recommendation scenarios. Here, we use top-k most-rated movies as the baseline model as we discussed earlier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>50</td>\n",
       "      <td>419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>181</td>\n",
       "      <td>382</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100</td>\n",
       "      <td>381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>258</td>\n",
       "      <td>377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>288</td>\n",
       "      <td>371</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieId  Count\n",
       "0       50    419\n",
       "1      181    382\n",
       "2      100    381\n",
       "3      258    377\n",
       "4      288    371"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "item_counts = train['MovieId'].value_counts().to_frame().reset_index()\n",
    "item_counts.columns = ['MovieId', 'Count']\n",
    "item_counts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of user-item pairs: 1546764\n",
      "After remove seen items: 1471784\n"
     ]
    }
   ],
   "source": [
    "user_item_col = ['UserId', 'MovieId']\n",
    "\n",
    "# Cross join users and items\n",
    "test_users = test['UserId'].unique()\n",
    "user_item_list = list(itertools.product(test_users, item_counts['MovieId']))\n",
    "users_items = pd.DataFrame(user_item_list, columns=user_item_col)\n",
    "\n",
    "print(\"Number of user-item pairs:\", len(users_items))\n",
    "\n",
    "# Remove seen items (items in the train set) as we will not recommend those again to the users\n",
    "users_items_remove_seen = filter_by(users_items, train, user_item_col)\n",
    "\n",
    "print(\"After remove seen items:\", len(users_items_remove_seen))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Count</th>\n",
       "      <th>UserId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>50</td>\n",
       "      <td>419</td>\n",
       "      <td>877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>419</td>\n",
       "      <td>815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>50</td>\n",
       "      <td>419</td>\n",
       "      <td>416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>50</td>\n",
       "      <td>419</td>\n",
       "      <td>259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>50</td>\n",
       "      <td>419</td>\n",
       "      <td>598</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieId  Count  UserId\n",
       "0       50    419     877\n",
       "1       50    419     815\n",
       "2       50    419     416\n",
       "3       50    419     259\n",
       "4       50    419     598"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Generate recommendations\n",
    "baseline_recommendations = pd.merge(item_counts, users_items_remove_seen, on=['MovieId'], how='inner')\n",
    "baseline_recommendations.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAP:\t0.055007\n",
      "NDCG@K:\t0.252864\n",
      "Precision@K:\t0.224628\n",
      "Recall@K:\t0.111736\n"
     ]
    }
   ],
   "source": [
    "k = 10\n",
    "\n",
    "cols['col_prediction'] = 'Count'\n",
    "\n",
    "eval_map = map_at_k(test, baseline_recommendations, k=k, **cols)\n",
    "eval_ndcg = ndcg_at_k(test, baseline_recommendations, k=k, **cols)\n",
    "eval_precision = precision_at_k(test, baseline_recommendations, k=k, **cols)\n",
    "eval_recall = recall_at_k(test, baseline_recommendations, k=k, **cols)\n",
    "\n",
    "print(\"MAP:\\t%f\" % eval_map,\n",
    "      \"NDCG@K:\\t%f\" % eval_ndcg,\n",
    "      \"Precision@K:\\t%f\" % eval_precision,\n",
    "      \"Recall@K:\\t%f\" % eval_recall, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, the baseline is quite high, nDCG = 0.25 and Precision = 0.22.\n",
    "\n",
    "<br>\n",
    "\n",
    "### Concluding remarks\n",
    "\n",
    "In this notebook, we discussed how to measure baseline performance for the movie recommendation example.\n",
    "We covered very naive approaches as baselines, but still they are useful in a sense that they can provide reference numbers to estimate the complexity of the given problem as well as the relative performance of the recommender models we are building."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/papermill.record+json": {
       "map": 0.055007342636635974
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "ndcg": 0.25286402361020544
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "precision": 0.22462845010615715
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "recall": 0.1117356507425933
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "rmse": 1.044885130655045
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "mae": 0.8369250150730534
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "exp_var": 0.1364955485850292
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "rsquared": 0.13649128638749664
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if is_jupyter():\n",
    "    # Record results with papermill for unit-tests\n",
    "    import papermill as pm\n",
    "    pm.record(\"map\", eval_map)\n",
    "    pm.record(\"ndcg\", eval_ndcg)\n",
    "    pm.record(\"precision\", eval_precision)\n",
    "    pm.record(\"recall\", eval_recall)\n",
    "    pm.record(\"rmse\", eval_rmse)\n",
    "    pm.record(\"mae\", eval_mae)\n",
    "    pm.record(\"exp_var\", eval_exp_var)\n",
    "    pm.record(\"rsquared\", eval_rsquared)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "[[1](https://dl.acm.org/citation.cfm?id=1401944)] Yehuda Koren,\tFactorization meets the neighborhood: a multifaceted collaborative filtering model, KDD '08 pp. 426-434 2008.  \n",
    "[[2](https://surprise.readthedocs.io/en/stable/basic_algorithms.html)] Surprise lib, Basic algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "reco_base",
   "language": "python",
   "name": "reco_base"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}