Movie_Recommender_System.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Movies Recommender System\n",
    "\n",
    "In this notebook, I will attempt to implement a few recommendation algorithms (content based, popularity based and collaborative filtering) and try to build an ensemble of these models to come up with our final recommendation system. Below are details of the dataset-\n",
    "\n",
    "* The Full Dataset: Consists of 26,000,000 ratings and 750,000 tag applications applied to 45,000 movies by 270,000 users.    Includes tag genome data with 12 million relevance scores across 1,100 tags.\n",
    "* The Small Dataset: Comprises of 100,000 ratings and 1,300 tag applications applied to 9,000 movies by 700 users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from scipy import stats\n",
    "from ast import literal_eval\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "from sklearn.metrics.pairwise import linear_kernel, cosine_similarity\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk.corpus import wordnet\n",
    "from surprise import Reader, Dataset, SVD\n",
    "from surprise.model_selection import cross_validate\n",
    "\n",
    "import warnings; warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple Recommender\n",
    "\n",
    "The Simple Recommender offers generalized recommnendations to every user based on movie popularity and (sometimes) genre. The basic idea behind this recommender is that movies that are more popular and more critically acclaimed will have a higher probability of being liked by the average audience. This model does not give personalized recommendations based on the user.\n",
    "\n",
    "The implementation of this model is extremely trivial. All we have to do is sort our movies based on ratings and popularity and display the top movies of our list. As an added step, we can pass in a genre argument to get the top movies of a particular genre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>adult</th>\n",
       "      <th>belongs_to_collection</th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>...</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>video</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 10194, 'name': 'Toy Story Collection', ...</td>\n",
       "      <td>30000000</td>\n",
       "      <td>[{'id': 16, 'name': 'Animation'}, {'id': 35, '...</td>\n",
       "      <td>http://toystory.disney.com/toy-story</td>\n",
       "      <td>862</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>en</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>...</td>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>373554033.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>False</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>65000000</td>\n",
       "      <td>[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8844</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>en</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "      <td>...</td>\n",
       "      <td>1995-12-15</td>\n",
       "      <td>262797249.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}, {'iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Roll the dice and unleash the excitement!</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>False</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 119050, 'name': 'Grumpy Old Men Collect...</td>\n",
       "      <td>0</td>\n",
       "      <td>[{'id': 10749, 'name': 'Romance'}, {'id': 35, ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15602</td>\n",
       "      <td>tt0113228</td>\n",
       "      <td>en</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>A family wedding reignites the ancient feud be...</td>\n",
       "      <td>...</td>\n",
       "      <td>1995-12-22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Still Yelling. Still Fighting. Still Ready for...</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>False</td>\n",
       "      <td>6.5</td>\n",
       "      <td>92.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16000000</td>\n",
       "      <td>[{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31357</td>\n",
       "      <td>tt0114885</td>\n",
       "      <td>en</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Cheated on, mistreated and stepped on, the wom...</td>\n",
       "      <td>...</td>\n",
       "      <td>1995-12-22</td>\n",
       "      <td>81452156.0</td>\n",
       "      <td>127.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Friends are the people who let you be yourself...</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>False</td>\n",
       "      <td>6.1</td>\n",
       "      <td>34.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 96871, 'name': 'Father of the Bride Col...</td>\n",
       "      <td>0</td>\n",
       "      <td>[{'id': 35, 'name': 'Comedy'}]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11862</td>\n",
       "      <td>tt0113041</td>\n",
       "      <td>en</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Just when George Banks has recovered from his ...</td>\n",
       "      <td>...</td>\n",
       "      <td>1995-02-10</td>\n",
       "      <td>76578911.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Just When His World Is Back To Normal... He's ...</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>False</td>\n",
       "      <td>5.7</td>\n",
       "      <td>173.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   adult                              belongs_to_collection    budget  \\\n",
       "0  False  {'id': 10194, 'name': 'Toy Story Collection', ...  30000000   \n",
       "1  False                                                NaN  65000000   \n",
       "2  False  {'id': 119050, 'name': 'Grumpy Old Men Collect...         0   \n",
       "3  False                                                NaN  16000000   \n",
       "4  False  {'id': 96871, 'name': 'Father of the Bride Col...         0   \n",
       "\n",
       "                                              genres  \\\n",
       "0  [{'id': 16, 'name': 'Animation'}, {'id': 35, '...   \n",
       "1  [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...   \n",
       "2  [{'id': 10749, 'name': 'Romance'}, {'id': 35, ...   \n",
       "3  [{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam...   \n",
       "4                     [{'id': 35, 'name': 'Comedy'}]   \n",
       "\n",
       "                               homepage     id    imdb_id original_language  \\\n",
       "0  http://toystory.disney.com/toy-story    862  tt0114709                en   \n",
       "1                                   NaN   8844  tt0113497                en   \n",
       "2                                   NaN  15602  tt0113228                en   \n",
       "3                                   NaN  31357  tt0114885                en   \n",
       "4                                   NaN  11862  tt0113041                en   \n",
       "\n",
       "                original_title  \\\n",
       "0                    Toy Story   \n",
       "1                      Jumanji   \n",
       "2             Grumpier Old Men   \n",
       "3            Waiting to Exhale   \n",
       "4  Father of the Bride Part II   \n",
       "\n",
       "                                            overview  ... release_date  \\\n",
       "0  Led by Woody, Andy's toys live happily in his ...  ...   1995-10-30   \n",
       "1  When siblings Judy and Peter discover an encha...  ...   1995-12-15   \n",
       "2  A family wedding reignites the ancient feud be...  ...   1995-12-22   \n",
       "3  Cheated on, mistreated and stepped on, the wom...  ...   1995-12-22   \n",
       "4  Just when George Banks has recovered from his ...  ...   1995-02-10   \n",
       "\n",
       "       revenue runtime                                   spoken_languages  \\\n",
       "0  373554033.0    81.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "1  262797249.0   104.0  [{'iso_639_1': 'en', 'name': 'English'}, {'iso...   \n",
       "2          0.0   101.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "3   81452156.0   127.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "4   76578911.0   106.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "\n",
       "     status                                            tagline  \\\n",
       "0  Released                                                NaN   \n",
       "1  Released          Roll the dice and unleash the excitement!   \n",
       "2  Released  Still Yelling. Still Fighting. Still Ready for...   \n",
       "3  Released  Friends are the people who let you be yourself...   \n",
       "4  Released  Just When His World Is Back To Normal... He's ...   \n",
       "\n",
       "                         title  video vote_average vote_count  \n",
       "0                    Toy Story  False          7.7     5415.0  \n",
       "1                      Jumanji  False          6.9     2413.0  \n",
       "2             Grumpier Old Men  False          6.5       92.0  \n",
       "3            Waiting to Exhale  False          6.1       34.0  \n",
       "4  Father of the Bride Part II  False          5.7      173.0  \n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md = pd. read_csv('./the-movies-dataset/movies_metadata.csv')\n",
    "md.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "md['genres'] = md['genres'].fillna('[]').apply(literal_eval). apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top Movies\n",
    "\n",
    "I have used the TMDB Ratings to come up with a Top Movies Chart. I will use IMDB's weighted rating formula to construct my chart. Mathematically, it is represented as follows:\n",
    "\n",
    "Weighted Rating (WR) =  (vv+m.R)+(mv+m.C) \n",
    "where,\n",
    "\n",
    "v is the number of votes for the movie\n",
    "m is the minimum votes required to be listed in the chart\n",
    "R is the average rating of the movie\n",
    "C is the mean vote across the whole report\n",
    "\n",
    "The next step is to determine an appropriate value for m, the minimum votes required to be listed in the chart. I have used 95th percentile as our cutoff. In other words, for a movie to feature in the charts, it must have more votes than at least 95% of the movies in the list.\n",
    "\n",
    "I will build overall Top 250 Chart and will define a function to build charts for a particular genre. Let's begin!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.244896612406511"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vote_counts = md[md['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "vote_averages = md[md['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "C = vote_averages.mean()\n",
    "C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "434.0"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = vote_counts.quantile(0.95)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "md['year'] = pd.to_datetime(md['release_date'], errors='coerce').apply(lambda x: str(x).split('-')[0] if x != np.nan else np.nan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2274, 6)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qualified = md[(md['vote_count'] >= m) & (md['vote_count'].notnull()) & (md['vote_average'].notnull())][['title', 'year', 'vote_count', 'vote_average', 'popularity', 'genres']]\n",
    "qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "qualified.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, to qualify to be considered for the chart, a movie has to have at least 434 votes on TMDB. Also, notice that the average rating for a movie on TMDB is 5.244 on a scale of 10. 2274 Movies qualify to be on our chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weighted_rating(x):\n",
    "    v = x['vote_count']\n",
    "    R = x['vote_average']\n",
    "    return (v/(v+m) * R) + (m/(m+v) * C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "qualified['wr'] = qualified.apply(weighted_rating, axis=1)\n",
    "qualified = qualified.sort_values('wr', ascending=False).head(250)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Top Movies__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>popularity</th>\n",
       "      <th>genres</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15480</th>\n",
       "      <td>Inception</td>\n",
       "      <td>2010</td>\n",
       "      <td>14075</td>\n",
       "      <td>8</td>\n",
       "      <td>29.1081</td>\n",
       "      <td>[Action, Thriller, Science Fiction, Mystery, A...</td>\n",
       "      <td>7.917588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12481</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>2008</td>\n",
       "      <td>12269</td>\n",
       "      <td>8</td>\n",
       "      <td>123.167</td>\n",
       "      <td>[Drama, Action, Crime, Thriller]</td>\n",
       "      <td>7.905871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22879</th>\n",
       "      <td>Interstellar</td>\n",
       "      <td>2014</td>\n",
       "      <td>11187</td>\n",
       "      <td>8</td>\n",
       "      <td>32.2135</td>\n",
       "      <td>[Adventure, Drama, Science Fiction]</td>\n",
       "      <td>7.897107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2843</th>\n",
       "      <td>Fight Club</td>\n",
       "      <td>1999</td>\n",
       "      <td>9678</td>\n",
       "      <td>8</td>\n",
       "      <td>63.8696</td>\n",
       "      <td>[Drama]</td>\n",
       "      <td>7.881753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4863</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>2001</td>\n",
       "      <td>8892</td>\n",
       "      <td>8</td>\n",
       "      <td>32.0707</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.871787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>1994</td>\n",
       "      <td>8670</td>\n",
       "      <td>8</td>\n",
       "      <td>140.95</td>\n",
       "      <td>[Thriller, Crime]</td>\n",
       "      <td>7.868660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>314</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>1994</td>\n",
       "      <td>8358</td>\n",
       "      <td>8</td>\n",
       "      <td>51.6454</td>\n",
       "      <td>[Drama, Crime]</td>\n",
       "      <td>7.864000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7000</th>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>2003</td>\n",
       "      <td>8226</td>\n",
       "      <td>8</td>\n",
       "      <td>29.3244</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.861927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>1994</td>\n",
       "      <td>8147</td>\n",
       "      <td>8</td>\n",
       "      <td>48.3072</td>\n",
       "      <td>[Comedy, Drama, Romance]</td>\n",
       "      <td>7.860656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5814</th>\n",
       "      <td>The Lord of the Rings: The Two Towers</td>\n",
       "      <td>2002</td>\n",
       "      <td>7641</td>\n",
       "      <td>8</td>\n",
       "      <td>29.4235</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.851924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>256</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>1977</td>\n",
       "      <td>6778</td>\n",
       "      <td>8</td>\n",
       "      <td>42.1497</td>\n",
       "      <td>[Adventure, Action, Science Fiction]</td>\n",
       "      <td>7.834205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1225</th>\n",
       "      <td>Back to the Future</td>\n",
       "      <td>1985</td>\n",
       "      <td>6239</td>\n",
       "      <td>8</td>\n",
       "      <td>25.7785</td>\n",
       "      <td>[Adventure, Comedy, Science Fiction, Family]</td>\n",
       "      <td>7.820813</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>834</th>\n",
       "      <td>The Godfather</td>\n",
       "      <td>1972</td>\n",
       "      <td>6024</td>\n",
       "      <td>8</td>\n",
       "      <td>41.1093</td>\n",
       "      <td>[Drama, Crime]</td>\n",
       "      <td>7.814847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1154</th>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>1980</td>\n",
       "      <td>5998</td>\n",
       "      <td>8</td>\n",
       "      <td>19.471</td>\n",
       "      <td>[Adventure, Action, Science Fiction]</td>\n",
       "      <td>7.814099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Se7en</td>\n",
       "      <td>1995</td>\n",
       "      <td>5915</td>\n",
       "      <td>8</td>\n",
       "      <td>18.4574</td>\n",
       "      <td>[Crime, Mystery, Thriller]</td>\n",
       "      <td>7.811669</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   title  year  vote_count  \\\n",
       "15480                                          Inception  2010       14075   \n",
       "12481                                    The Dark Knight  2008       12269   \n",
       "22879                                       Interstellar  2014       11187   \n",
       "2843                                          Fight Club  1999        9678   \n",
       "4863   The Lord of the Rings: The Fellowship of the Ring  2001        8892   \n",
       "292                                         Pulp Fiction  1994        8670   \n",
       "314                             The Shawshank Redemption  1994        8358   \n",
       "7000       The Lord of the Rings: The Return of the King  2003        8226   \n",
       "351                                         Forrest Gump  1994        8147   \n",
       "5814               The Lord of the Rings: The Two Towers  2002        7641   \n",
       "256                                            Star Wars  1977        6778   \n",
       "1225                                  Back to the Future  1985        6239   \n",
       "834                                        The Godfather  1972        6024   \n",
       "1154                             The Empire Strikes Back  1980        5998   \n",
       "46                                                 Se7en  1995        5915   \n",
       "\n",
       "       vote_average popularity  \\\n",
       "15480             8    29.1081   \n",
       "12481             8    123.167   \n",
       "22879             8    32.2135   \n",
       "2843              8    63.8696   \n",
       "4863              8    32.0707   \n",
       "292               8     140.95   \n",
       "314               8    51.6454   \n",
       "7000              8    29.3244   \n",
       "351               8    48.3072   \n",
       "5814              8    29.4235   \n",
       "256               8    42.1497   \n",
       "1225              8    25.7785   \n",
       "834               8    41.1093   \n",
       "1154              8     19.471   \n",
       "46                8    18.4574   \n",
       "\n",
       "                                                  genres        wr  \n",
       "15480  [Action, Thriller, Science Fiction, Mystery, A...  7.917588  \n",
       "12481                   [Drama, Action, Crime, Thriller]  7.905871  \n",
       "22879                [Adventure, Drama, Science Fiction]  7.897107  \n",
       "2843                                             [Drama]  7.881753  \n",
       "4863                        [Adventure, Fantasy, Action]  7.871787  \n",
       "292                                    [Thriller, Crime]  7.868660  \n",
       "314                                       [Drama, Crime]  7.864000  \n",
       "7000                        [Adventure, Fantasy, Action]  7.861927  \n",
       "351                             [Comedy, Drama, Romance]  7.860656  \n",
       "5814                        [Adventure, Fantasy, Action]  7.851924  \n",
       "256                 [Adventure, Action, Science Fiction]  7.834205  \n",
       "1225        [Adventure, Comedy, Science Fiction, Family]  7.820813  \n",
       "834                                       [Drama, Crime]  7.814847  \n",
       "1154                [Adventure, Action, Science Fiction]  7.814099  \n",
       "46                            [Crime, Mystery, Thriller]  7.811669  "
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qualified.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, three Christopher Nolan Films, Inception, The Dark Knight and Interstellar occur at the very top of our chart. The chart also indicates a strong bias of TMDB Users towards particular genres and directors.\n",
    "\n",
    "Let us now construct our function that builds charts for particular genres. For this, I will reduce the default conditions to 85th percentile instead of 95."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = md.apply(lambda x: pd.Series(x['genres']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "s.name = 'genre'\n",
    "gen_md = md.drop('genres', axis=1).join(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_chart(genre, percentile=0.85):\n",
    "    df = gen_md[gen_md['genre'] == genre]\n",
    "    vote_counts = df[df['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "    vote_averages = df[df['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "    C = vote_averages.mean()\n",
    "    m = vote_counts.quantile(percentile)\n",
    "    \n",
    "    qualified = df[(df['vote_count'] >= m) & (df['vote_count'].notnull()) & (df['vote_average'].notnull())][['title', 'year', 'vote_count', 'vote_average', 'popularity']]\n",
    "    qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "    qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "    \n",
    "    qualified['wr'] = qualified.apply(lambda x: (x['vote_count']/(x['vote_count']+m) * x['vote_average']) + (m/(m+x['vote_count']) * C), axis=1)\n",
    "    qualified = qualified.sort_values('wr', ascending=False).head(250)\n",
    "    \n",
    "    return qualified"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see our method in action by displaying the Top 15 Romance Movies. Romance almost didn't feature at all in our Generic Top Chart despite being one of the most popular movie genres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>popularity</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10309</th>\n",
       "      <td>Dilwale Dulhania Le Jayenge</td>\n",
       "      <td>1995</td>\n",
       "      <td>661</td>\n",
       "      <td>9</td>\n",
       "      <td>34.457</td>\n",
       "      <td>8.565285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>1994</td>\n",
       "      <td>8147</td>\n",
       "      <td>8</td>\n",
       "      <td>48.3072</td>\n",
       "      <td>7.971357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>876</th>\n",
       "      <td>Vertigo</td>\n",
       "      <td>1958</td>\n",
       "      <td>1162</td>\n",
       "      <td>8</td>\n",
       "      <td>18.2082</td>\n",
       "      <td>7.811667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40251</th>\n",
       "      <td>Your Name.</td>\n",
       "      <td>2016</td>\n",
       "      <td>1030</td>\n",
       "      <td>8</td>\n",
       "      <td>34.461252</td>\n",
       "      <td>7.789489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>883</th>\n",
       "      <td>Some Like It Hot</td>\n",
       "      <td>1959</td>\n",
       "      <td>835</td>\n",
       "      <td>8</td>\n",
       "      <td>11.8451</td>\n",
       "      <td>7.745154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1132</th>\n",
       "      <td>Cinema Paradiso</td>\n",
       "      <td>1988</td>\n",
       "      <td>834</td>\n",
       "      <td>8</td>\n",
       "      <td>14.177</td>\n",
       "      <td>7.744878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19901</th>\n",
       "      <td>Paperman</td>\n",
       "      <td>2012</td>\n",
       "      <td>734</td>\n",
       "      <td>8</td>\n",
       "      <td>7.19863</td>\n",
       "      <td>7.713951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37863</th>\n",
       "      <td>Sing Street</td>\n",
       "      <td>2016</td>\n",
       "      <td>669</td>\n",
       "      <td>8</td>\n",
       "      <td>10.672862</td>\n",
       "      <td>7.689483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>882</th>\n",
       "      <td>The Apartment</td>\n",
       "      <td>1960</td>\n",
       "      <td>498</td>\n",
       "      <td>8</td>\n",
       "      <td>11.9943</td>\n",
       "      <td>7.599317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38718</th>\n",
       "      <td>The Handmaiden</td>\n",
       "      <td>2016</td>\n",
       "      <td>453</td>\n",
       "      <td>8</td>\n",
       "      <td>16.727405</td>\n",
       "      <td>7.566166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3189</th>\n",
       "      <td>City Lights</td>\n",
       "      <td>1931</td>\n",
       "      <td>444</td>\n",
       "      <td>8</td>\n",
       "      <td>10.8915</td>\n",
       "      <td>7.558867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24886</th>\n",
       "      <td>The Way He Looks</td>\n",
       "      <td>2014</td>\n",
       "      <td>262</td>\n",
       "      <td>8</td>\n",
       "      <td>5.71127</td>\n",
       "      <td>7.331363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45437</th>\n",
       "      <td>In a Heartbeat</td>\n",
       "      <td>2017</td>\n",
       "      <td>146</td>\n",
       "      <td>8</td>\n",
       "      <td>20.82178</td>\n",
       "      <td>7.003959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1639</th>\n",
       "      <td>Titanic</td>\n",
       "      <td>1997</td>\n",
       "      <td>7770</td>\n",
       "      <td>7</td>\n",
       "      <td>26.8891</td>\n",
       "      <td>6.981546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19731</th>\n",
       "      <td>Silver Linings Playbook</td>\n",
       "      <td>2012</td>\n",
       "      <td>4840</td>\n",
       "      <td>7</td>\n",
       "      <td>14.4881</td>\n",
       "      <td>6.970581</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             title  year  vote_count  vote_average popularity  \\\n",
       "10309  Dilwale Dulhania Le Jayenge  1995         661             9     34.457   \n",
       "351                   Forrest Gump  1994        8147             8    48.3072   \n",
       "876                        Vertigo  1958        1162             8    18.2082   \n",
       "40251                   Your Name.  2016        1030             8  34.461252   \n",
       "883               Some Like It Hot  1959         835             8    11.8451   \n",
       "1132               Cinema Paradiso  1988         834             8     14.177   \n",
       "19901                     Paperman  2012         734             8    7.19863   \n",
       "37863                  Sing Street  2016         669             8  10.672862   \n",
       "882                  The Apartment  1960         498             8    11.9943   \n",
       "38718               The Handmaiden  2016         453             8  16.727405   \n",
       "3189                   City Lights  1931         444             8    10.8915   \n",
       "24886             The Way He Looks  2014         262             8    5.71127   \n",
       "45437               In a Heartbeat  2017         146             8   20.82178   \n",
       "1639                       Titanic  1997        7770             7    26.8891   \n",
       "19731      Silver Linings Playbook  2012        4840             7    14.4881   \n",
       "\n",
       "             wr  \n",
       "10309  8.565285  \n",
       "351    7.971357  \n",
       "876    7.811667  \n",
       "40251  7.789489  \n",
       "883    7.745154  \n",
       "1132   7.744878  \n",
       "19901  7.713951  \n",
       "37863  7.689483  \n",
       "882    7.599317  \n",
       "38718  7.566166  \n",
       "3189   7.558867  \n",
       "24886  7.331363  \n",
       "45437  7.003959  \n",
       "1639   6.981546  \n",
       "19731  6.970581  "
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build_chart('Romance').head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content Based Recommender\n",
    "\n",
    "The recommender I built in the previous section suffers some severe limitations. For one, it gives the same recommendation to everyone, regardless of the user's personal taste. If a person who loves romantic movies (and hates action) were to look at our Top 15 Chart, the user wouldn't probably like most of the movies. If a user were to go one step further and look at our charts by genre, they wouldn't still be getting the best recommendations.\n",
    "\n",
    "To personalise our recommendations more, I am going to build an engine that computes similarity between movies based on cosine similarity metrics and suggests movies that are most similar to a particular movie that a user liked. Since I will be using movie metadata or content to build this engine, this also known as Content Based Filtering.\n",
    "\n",
    "I will build two Content Based Recommenders based on:\n",
    "\n",
    "* Movie Overviews and Taglines\n",
    "* Movie Cast, Crew, Keywords and Genre\n",
    "\n",
    "I will be using a subset of all the movies available due to limited computing power."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "links_small = pd.read_csv('./the-movies-dataset/links_small.csv')\n",
    "links_small = links_small[links_small['tmdbId'].notnull()]['tmdbId'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>adult</th>\n",
       "      <th>belongs_to_collection</th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>...</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>video</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19730</th>\n",
       "      <td>- Written by Ørnås</td>\n",
       "      <td>0.065736</td>\n",
       "      <td>/ff9qCepilowshEtG2GYWwzt2bs4.jpg</td>\n",
       "      <td>[Carousel Productions, Vision View Entertainme...</td>\n",
       "      <td>[{'iso_3166_1': 'CA', 'name': 'Canada'}, {'iso...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29503</th>\n",
       "      <td>Rune Balot goes to a casino connected to the ...</td>\n",
       "      <td>1.931659</td>\n",
       "      <td>/zV8bHuSL6WXoD6FWogP9j4x80bL.jpg</td>\n",
       "      <td>[Aniplex, GoHands, BROSTA TV, Mardock Scramble...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>[{'iso_639_1': 'ja', 'name': '日本語'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35587</th>\n",
       "      <td>Avalanche Sharks tells the story of a bikini ...</td>\n",
       "      <td>2.185485</td>\n",
       "      <td>/zaSf5OG7V8X8gqFvly88zDdRm46.jpg</td>\n",
       "      <td>[Odyssey Media, Pulser Productions, Rogue Stat...</td>\n",
       "      <td>[{'iso_3166_1': 'CA', 'name': 'Canada'}]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   adult  \\\n",
       "19730                                 - Written by Ørnås   \n",
       "29503   Rune Balot goes to a casino connected to the ...   \n",
       "35587   Avalanche Sharks tells the story of a bikini ...   \n",
       "\n",
       "      belongs_to_collection                            budget  \\\n",
       "19730              0.065736  /ff9qCepilowshEtG2GYWwzt2bs4.jpg   \n",
       "29503              1.931659  /zV8bHuSL6WXoD6FWogP9j4x80bL.jpg   \n",
       "35587              2.185485  /zaSf5OG7V8X8gqFvly88zDdRm46.jpg   \n",
       "\n",
       "                                                  genres  \\\n",
       "19730  [Carousel Productions, Vision View Entertainme...   \n",
       "29503  [Aniplex, GoHands, BROSTA TV, Mardock Scramble...   \n",
       "35587  [Odyssey Media, Pulser Productions, Rogue Stat...   \n",
       "\n",
       "                                                homepage  id imdb_id  \\\n",
       "19730  [{'iso_3166_1': 'CA', 'name': 'Canada'}, {'iso... NaN       0   \n",
       "29503  [{'iso_3166_1': 'US', 'name': 'United States o... NaN       0   \n",
       "35587           [{'iso_3166_1': 'CA', 'name': 'Canada'}] NaN       0   \n",
       "\n",
       "      original_language                            original_title  overview  \\\n",
       "19730             104.0  [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "29503              68.0      [{'iso_639_1': 'ja', 'name': '日本語'}]  Released   \n",
       "35587              82.0  [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "\n",
       "       ... revenue runtime spoken_languages status tagline  title  video  \\\n",
       "19730  ...     NaN     NaN              NaN    NaN     NaN    NaN    NaN   \n",
       "29503  ...     NaN     NaN              NaN    NaN     NaN    NaN    NaN   \n",
       "35587  ...     NaN     NaN              NaN    NaN     NaN    NaN    NaN   \n",
       "\n",
       "      vote_average vote_count year  \n",
       "19730          NaN        NaN  NaT  \n",
       "29503          NaN        NaN  NaT  \n",
       "35587          NaN        NaN  NaT  \n",
       "\n",
       "[3 rows x 25 columns]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def convert_int(x):\n",
    "    try:\n",
    "        return int(x)\n",
    "    except:\n",
    "        return np.nan\n",
    "\n",
    "md['id'] = md['id'].apply(convert_int)\n",
    "md[md['id'].isnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9099, 25)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md = md.drop([19730, 29503, 35587])\n",
    "md['id'] = md['id'].astype('int')\n",
    "smd = md[md['id'].isin(links_small)]\n",
    "smd.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have 9099 movies avaiable in the small movies metadata dataset which is 5 times smaller than our original dataset of 45000 movies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Movie Description Based Recommender__\n",
    "\n",
    "Let us first try to build a recommender using movie descriptions and taglines. We do not have a quantitative metric to judge our machine's performance so this will have to be done qualitatively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['tagline'] = smd['tagline'].fillna('')\n",
    "smd['description'] = smd['overview'] + smd['tagline']\n",
    "smd['description'] = smd['description'].fillna('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf = TfidfVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')\n",
    "tfidf_matrix = tf.fit_transform(smd['description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9099, 268124)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Cosine Similarity__\n",
    "\n",
    "I will be using the Cosine Similarity to calculate a numeric quantity that denotes the similarity between two movies. Mathematically, it is defined as follows:\n",
    "\n",
    "cosine(x,y)=x.y⊺/|x||.||y|| \n",
    "\n",
    "Since we have used the TF-IDF Vectorizer, calculating the Dot Product will directly give us the Cosine Similarity Score. Therefore, we will use sklearn's linear_kernel instead of cosine_similarities since it is much faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 0.00680476, 0.        , ..., 0.        , 0.00344913,\n",
       "       0.        ])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)\n",
    "cosine_sim[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a pairwise cosine similarity matrix for all the movies in our dataset. The next step is to write a function that returns the 30 most similar movies based on the cosine similarity score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd = smd.reset_index()\n",
    "titles = smd['title']\n",
    "indices = pd.Series(smd.index, index=smd['title'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_recommendations(title):\n",
    "    idx = indices[title]\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:31]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    return titles.iloc[movie_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're all set. Let us now try and get the top recommendations for a few movies and see how good the recommendations are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "973      The Godfather: Part II\n",
       "8387                 The Family\n",
       "3509                       Made\n",
       "4196         Johnny Dangerously\n",
       "29               Shanghai Triad\n",
       "5667                       Fury\n",
       "2412             American Movie\n",
       "1582    The Godfather: Part III\n",
       "4221                    8 Women\n",
       "2159              Summer of Sam\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Godfather').head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7931                      The Dark Knight Rises\n",
       "132                              Batman Forever\n",
       "1113                             Batman Returns\n",
       "8227    Batman: The Dark Knight Returns, Part 2\n",
       "7565                 Batman: Under the Red Hood\n",
       "524                                      Batman\n",
       "7901                           Batman: Year One\n",
       "2579               Batman: Mask of the Phantasm\n",
       "2696                                        JFK\n",
       "8165    Batman: The Dark Knight Returns, Part 1\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that for The Dark Knight, our system is able to identify it as a Batman film and subsequently recommend other Batman films as its top recommendations. But unfortunately, that is all this system can do at the moment. This is not of much use to most people as it doesn't take into considerations very important features such as cast, crew, director and genre, which determine the rating and the popularity of a movie. Someone who liked The Dark Knight probably likes it more because of Nolan and would hate Batman Forever and every other substandard movie in the Batman Franchise.\n",
    "\n",
    "Therefore, we are going to use much more suggestive metadata than Overview and Tagline. In the next subsection, I will build a more sophisticated recommender that takes genre, keywords, cast and crew into consideration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Metadata Based Recommender__\n",
    "\n",
    "To build our standard metadata based content recommender, I will have to merge our current dataset with the crew and the keyword datasets. Let us prepare this data to proceed further."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "credits = pd.read_csv('./the-movies-dataset/credits.csv')\n",
    "keywords = pd.read_csv('./the-movies-dataset/keywords.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45463, 25)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keywords['id'] = keywords['id'].astype('int')\n",
    "credits['id'] = credits['id'].astype('int')\n",
    "md['id'] = md['id'].astype('int')\n",
    "md.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9219, 28)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md = md.merge(credits, on='id')\n",
    "md = md.merge(keywords, on='id')\n",
    "smd = md[md['id'].isin(links_small)]\n",
    "smd.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have our cast, crew, genres and credits, all in one dataframe. Let us wrangle this a little more using the following intuitions:\n",
    "\n",
    "* Crew: From the crew, we will only pick the director as our feature since the others don't contribute that much to the feel of   the movie.\n",
    "* Cast: Choosing Cast is a little more tricky. Lesser known actors and minor roles do not really affect people's opinion of a     movie. Therefore, we must only select the major characters and their respective actors. Arbitrarily we will choose the top 3   actors that appear in the credits list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['cast'] = smd['cast'].apply(literal_eval)\n",
    "smd['crew'] = smd['crew'].apply(literal_eval)\n",
    "smd['keywords'] = smd['keywords'].apply(literal_eval)\n",
    "smd['cast_size'] = smd['cast'].apply(lambda x: len(x))\n",
    "smd['crew_size'] = smd['crew'].apply(lambda x: len(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_director(x):\n",
    "    for i in x:\n",
    "        if i['job'] == 'Director':\n",
    "            return i['name']\n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['director'] = smd['crew'].apply(get_director)\n",
    "smd['cast'] = smd['cast'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
    "smd['cast'] = smd['cast'].apply(lambda x: x[:3] if len(x) >=3 else x)\n",
    "smd['keywords'] = smd['keywords'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My approach to building the recommender is going to be extremely hacky. What I plan on doing is creating a metadata dump for every movie which consists of genres, director, main actors and keywords. I then use a Count Vectorizer to create our count matrix as we did in the Description Recommender. The remaining steps are similar to what we did earlier: calculate the cosine similarities and return movies that are most similar.\n",
    "\n",
    "These are steps I follow in the preparation of my genres and credits data:\n",
    "\n",
    "   * Strip Spaces and Convert to Lowercase from all our features. This way, our engine will not confuse between Johnny Depp and      Johnny Galecki.\n",
    "   * Mention Director 3 times to give it more weight relative to the entire cast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['cast'] = smd['cast'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])\n",
    "smd['director'] = smd['director'].astype('str').apply(lambda x: str.lower(x.replace(\" \", \"\")))\n",
    "smd['director'] = smd['director'].apply(lambda x: [x,x, x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Keywords__\n",
    "\n",
    "I will do a small amount of pre-processing of our keywords before putting them to any use. As a first step, we need to calculate the frequenct counts of every keyword that appears in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = smd.apply(lambda x: pd.Series(x['keywords']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "s.name = 'keyword'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "independent film        610\n",
       "woman director          550\n",
       "murder                  399\n",
       "duringcreditsstinger    327\n",
       "based on novel          318\n",
       "Name: keyword, dtype: int64"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = s.value_counts()\n",
    "s[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keywords occur in frequencies ranging from 1 to 610. We do not have any use for keywords that occur only once. Therefore, these can be safely removed. Finally, I will convert every word to its stem so that words such as Dogs and Dog are considered the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'dog'"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = s[s > 1]\n",
    "stemmer = SnowballStemmer('english')\n",
    "stemmer.stem('dogs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_keywords(x):\n",
    "    words = []\n",
    "    for i in x:\n",
    "        if i in s:\n",
    "            words.append(i)\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['keywords'] = smd['keywords'].apply(filter_keywords)\n",
    "smd['keywords'] = smd['keywords'].apply(lambda x: [stemmer.stem(i) for i in x])\n",
    "smd['keywords'] = smd['keywords'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])\n",
    "smd['soup'] = smd['keywords'] + smd['cast'] + smd['director'] + smd['genres']\n",
    "smd['soup'] = smd['soup'].apply(lambda x: ' '.join(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "count = CountVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')\n",
    "count_matrix = count.fit_transform(smd['soup'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "cosine_sim = cosine_similarity(count_matrix, count_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd = smd.reset_index()\n",
    "titles = smd['title']\n",
    "indices = pd.Series(smd.index, index=smd['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will reuse the get_recommendations function that we had written earlier. Since our cosine similarity scores have changed, we expect it to give us different (and probably better) results. Let us check for The Dark Knight again and see what recommendations I get this time around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8031         The Dark Knight Rises\n",
       "6218                 Batman Begins\n",
       "6623                  The Prestige\n",
       "2085                     Following\n",
       "7648                     Inception\n",
       "4145                      Insomnia\n",
       "3381                       Memento\n",
       "8613                  Interstellar\n",
       "7659    Batman: Under the Red Hood\n",
       "1134                Batman Returns\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommendations seem to have recognized other Christopher Nolan movies (due to the high weightage given to director) and put them as top recommendations. I enjoyed watching The Dark Knight as well as some of the other ones in the list including Batman Begins, The Prestige and The Dark Knight Rises.\n",
    "\n",
    "We can of course experiment on this engine by trying out different weights for our features (directors, actors, genres), limiting the number of keywords that can be used in the soup, weighing genres based on their frequency, only showing movies with the same languages, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Popularity and Ratings__\n",
    "\n",
    "One thing that I notice about the recommendation system is that it recommends movies regardless of ratings and popularity. It is true that Batman and Robin has a lot of similar characters as compared to The Dark Knight but it was a terrible movie that shouldn't be recommended to anyone.\n",
    "\n",
    "Therefore, I will add a mechanism to remove bad movies and return movies which are popular and have had a good critical response.\n",
    "\n",
    "I will take the top 25 movies based on similarity scores and calculate the vote of the 60th percentile movie. Then, using this as the value of  m , we will calculate the weighted rating of each movie using IMDB's formula like we did in the Simple Recommender section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "def improved_recommendations(title):\n",
    "    idx = indices[title]\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:26]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    \n",
    "    movies = smd.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'year']]\n",
    "    vote_counts = movies[movies['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "    vote_averages = movies[movies['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "    C = vote_averages.mean()\n",
    "    m = vote_counts.quantile(0.60)\n",
    "    qualified = movies[(movies['vote_count'] >= m) & (movies['vote_count'].notnull()) & (movies['vote_average'].notnull())]\n",
    "    qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "    qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "    qualified['wr'] = qualified.apply(weighted_rating, axis=1)\n",
    "    qualified = qualified.sort_values('wr', ascending=False).head(10)\n",
    "    return qualified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7648</th>\n",
       "      <td>Inception</td>\n",
       "      <td>14075</td>\n",
       "      <td>8</td>\n",
       "      <td>2010</td>\n",
       "      <td>7.917588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8613</th>\n",
       "      <td>Interstellar</td>\n",
       "      <td>11187</td>\n",
       "      <td>8</td>\n",
       "      <td>2014</td>\n",
       "      <td>7.897107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6623</th>\n",
       "      <td>The Prestige</td>\n",
       "      <td>4510</td>\n",
       "      <td>8</td>\n",
       "      <td>2006</td>\n",
       "      <td>7.758148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3381</th>\n",
       "      <td>Memento</td>\n",
       "      <td>4168</td>\n",
       "      <td>8</td>\n",
       "      <td>2000</td>\n",
       "      <td>7.740175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8031</th>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>9263</td>\n",
       "      <td>7</td>\n",
       "      <td>2012</td>\n",
       "      <td>6.921448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6218</th>\n",
       "      <td>Batman Begins</td>\n",
       "      <td>7511</td>\n",
       "      <td>7</td>\n",
       "      <td>2005</td>\n",
       "      <td>6.904127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134</th>\n",
       "      <td>Batman Returns</td>\n",
       "      <td>1706</td>\n",
       "      <td>6</td>\n",
       "      <td>1992</td>\n",
       "      <td>5.846862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>Batman Forever</td>\n",
       "      <td>1529</td>\n",
       "      <td>5</td>\n",
       "      <td>1995</td>\n",
       "      <td>5.054144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9024</th>\n",
       "      <td>Batman v Superman: Dawn of Justice</td>\n",
       "      <td>7189</td>\n",
       "      <td>5</td>\n",
       "      <td>2016</td>\n",
       "      <td>5.013943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1260</th>\n",
       "      <td>Batman &amp; Robin</td>\n",
       "      <td>1447</td>\n",
       "      <td>4</td>\n",
       "      <td>1997</td>\n",
       "      <td>4.287233</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   title  vote_count  vote_average  year  \\\n",
       "7648                           Inception       14075             8  2010   \n",
       "8613                        Interstellar       11187             8  2014   \n",
       "6623                        The Prestige        4510             8  2006   \n",
       "3381                             Memento        4168             8  2000   \n",
       "8031               The Dark Knight Rises        9263             7  2012   \n",
       "6218                       Batman Begins        7511             7  2005   \n",
       "1134                      Batman Returns        1706             6  1992   \n",
       "132                       Batman Forever        1529             5  1995   \n",
       "9024  Batman v Superman: Dawn of Justice        7189             5  2016   \n",
       "1260                      Batman & Robin        1447             4  1997   \n",
       "\n",
       "            wr  \n",
       "7648  7.917588  \n",
       "8613  7.897107  \n",
       "6623  7.758148  \n",
       "3381  7.740175  \n",
       "8031  6.921448  \n",
       "6218  6.904127  \n",
       "1134  5.846862  \n",
       "132   5.054144  \n",
       "9024  5.013943  \n",
       "1260  4.287233  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "improved_recommendations('The Dark Knight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, Batman and Robin does not disappear from our recommendation list. This is probably due to the fact that it is rated a 4, which is only slightly below average on TMDB. It certainly doesn't deserve a 4 when amazing movies like The Dark Knight Rises has only a 7. However, there is nothing much we can do about this. Therefore, we will conclude our Content Based Recommender section here and come back to it when we build a hybrid engine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collaborative Filtering\n",
    "\n",
    "Our content based engine suffers from some severe limitations. It is only capable of suggesting movies which are close to a certain movie. That is, it is not capable of capturing tastes and providing recommendations across genres.\n",
    "\n",
    "Also, the engine that we built is not really personal in that it doesn't capture the personal tastes and biases of a user. Anyone querying our engine for recommendations based on a movie will receive the same recommendations for that movie, regardless of who the user is.\n",
    "\n",
    "Therefore, in this section, we will use a technique called Collaborative Filtering to make recommendations to Movie Watchers. Collaborative Filtering is based on the idea that users similar to me can be used to predict how much I will like a particular product or service those users have used/experienced, but I have not.\n",
    "\n",
    "I will not be implementing Collaborative Filtering from scratch. Instead, I will use the Surprise library that used extremely powerful algorithms like Singular Value Decomposition (SVD) to minimise RMSE (Root Mean Square Error) and give great recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating   timestamp\n",
       "0       1       31     2.5  1260759144\n",
       "1       1     1029     3.0  1260759179\n",
       "2       1     1061     3.0  1260759182\n",
       "3       1     1129     2.0  1260759185\n",
       "4       1     1172     4.0  1260759205"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reader = Reader()\n",
    "ratings = pd.read_csv('./the-movies-dataset/ratings_small.csv')\n",
    "ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating RMSE, MAE of algorithm SVD on 5 split(s).\n",
      "\n",
      "                  Fold 1  Fold 2  Fold 3  Fold 4  Fold 5  Mean    Std     \n",
      "RMSE (testset)    0.8954  0.8992  0.8977  0.8857  0.9003  0.8957  0.0052  \n",
      "MAE (testset)     0.6908  0.6921  0.6912  0.6833  0.6948  0.6904  0.0038  \n",
      "Fit time          3.82    3.86    4.06    4.24    4.21    4.04    0.17    \n",
      "Test time         0.10    0.10    0.10    0.19    0.11    0.12    0.04    \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'test_rmse': array([0.89538268, 0.89915238, 0.89771983, 0.88574945, 0.90027159]),\n",
       " 'test_mae': array([0.69076987, 0.69208609, 0.69118191, 0.68331668, 0.69482486]),\n",
       " 'fit_time': (3.8175125122070312,\n",
       "  3.8626413345336914,\n",
       "  4.064144611358643,\n",
       "  4.2376179695129395,\n",
       "  4.212680816650391),\n",
       " 'test_time': (0.10203886032104492,\n",
       "  0.09977388381958008,\n",
       "  0.09974265098571777,\n",
       "  0.19452261924743652,\n",
       "  0.10870981216430664)}"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = Dataset.load_from_df(ratings[['userId', 'movieId', 'rating']], reader)\n",
    "algo = SVD()\n",
    "\n",
    "# Run 5-fold cross-validation and then print results\n",
    "cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=5, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I get a mean Root Mean Sqaure Error of 0.8957, which is more than enough for our case. Let us now train on our dataset and arrive at predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<surprise.prediction_algorithms.matrix_factorization.SVD at 0x1fd83283908>"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainset = data.build_full_trainset()\n",
    "algo.fit(trainset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>1263</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>1287</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>1293</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>1339</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1260759125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>1343</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>1371</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>1405</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1260759203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1</td>\n",
       "      <td>1953</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>2105</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1</td>\n",
       "      <td>2150</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1</td>\n",
       "      <td>2193</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1</td>\n",
       "      <td>2294</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1</td>\n",
       "      <td>2455</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>2968</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1260759200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1</td>\n",
       "      <td>3671</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759117</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    userId  movieId  rating   timestamp\n",
       "0        1       31     2.5  1260759144\n",
       "1        1     1029     3.0  1260759179\n",
       "2        1     1061     3.0  1260759182\n",
       "3        1     1129     2.0  1260759185\n",
       "4        1     1172     4.0  1260759205\n",
       "5        1     1263     2.0  1260759151\n",
       "6        1     1287     2.0  1260759187\n",
       "7        1     1293     2.0  1260759148\n",
       "8        1     1339     3.5  1260759125\n",
       "9        1     1343     2.0  1260759131\n",
       "10       1     1371     2.5  1260759135\n",
       "11       1     1405     1.0  1260759203\n",
       "12       1     1953     4.0  1260759191\n",
       "13       1     2105     4.0  1260759139\n",
       "14       1     2150     3.0  1260759194\n",
       "15       1     2193     2.0  1260759198\n",
       "16       1     2294     2.0  1260759108\n",
       "17       1     2455     2.5  1260759113\n",
       "18       1     2968     1.0  1260759200\n",
       "19       1     3671     3.0  1260759117"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings[ratings['userId'] == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(uid=1, iid=302, r_ui=3, est=2.797955380168156, details={'was_impossible': False})"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "algo.predict(1, 302, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For movie with ID 302, we get an estimated prediction of 2.686. One startling feature of this recommender system is that it doesn't care what the movie is (or what it contains). It works purely on the basis of an assigned movie ID and tries to predict ratings based on how the other users have rated the movie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hybrid Recommender\n",
    "\n",
    "In this section, I will try to build a simple hybrid recommender that brings together techniques we have implemented in the content based and collaborative filter based engines. This is how it will work:\n",
    "\n",
    "* Input: User ID and the Title of a Movie\n",
    "* Output: Similar movies sorted on the basis of expected ratings by that particular user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_int(x):\n",
    "    try:\n",
    "        return int(x)\n",
    "    except:\n",
    "        return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_map = pd.read_csv('./the-movies-dataset/links_small.csv')[['movieId', 'tmdbId']]\n",
    "id_map['tmdbId'] = id_map['tmdbId'].apply(convert_int)\n",
    "id_map.columns = ['movieId', 'id']\n",
    "id_map = id_map.merge(smd[['title', 'id']], on='id').set_index('title')\n",
    "#id_map = id_map.set_index('tmdbId')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_map = id_map.set_index('id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hybrid(userId, title):\n",
    "    idx = indices[title]\n",
    "    tmdbId = id_map.loc[title]['id']\n",
    "    #print(idx)\n",
    "    movie_id = id_map.loc[title]['movieId']\n",
    "    \n",
    "    sim_scores = list(enumerate(cosine_sim[int(idx)]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:26]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    \n",
    "    movies = smd.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'year', 'id']]\n",
    "    movies['est'] = movies['id'].apply(lambda x: algo.predict(userId, indices_map.loc[x]['movieId']).est)\n",
    "    movies = movies.sort_values('est', ascending=False)\n",
    "    return movies.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "      <th>est</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>522</th>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>4274.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1991</td>\n",
       "      <td>280</td>\n",
       "      <td>4.152059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>Aliens</td>\n",
       "      <td>3282.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1986</td>\n",
       "      <td>679</td>\n",
       "      <td>3.707908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3060</th>\n",
       "      <td>Sinbad and the Eye of the Tiger</td>\n",
       "      <td>39.0</td>\n",
       "      <td>6.3</td>\n",
       "      <td>1977</td>\n",
       "      <td>11940</td>\n",
       "      <td>3.298736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>The Terminator</td>\n",
       "      <td>4208.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>1984</td>\n",
       "      <td>218</td>\n",
       "      <td>3.251889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8658</th>\n",
       "      <td>X-Men: Days of Future Past</td>\n",
       "      <td>6155.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>2014</td>\n",
       "      <td>127585</td>\n",
       "      <td>3.214787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>344</th>\n",
       "      <td>True Lies</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>6.8</td>\n",
       "      <td>1994</td>\n",
       "      <td>36955</td>\n",
       "      <td>3.189446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>922</th>\n",
       "      <td>The Abyss</td>\n",
       "      <td>822.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>1989</td>\n",
       "      <td>2756</td>\n",
       "      <td>3.045144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1668</th>\n",
       "      <td>Return from Witch Mountain</td>\n",
       "      <td>38.0</td>\n",
       "      <td>5.6</td>\n",
       "      <td>1978</td>\n",
       "      <td>14822</td>\n",
       "      <td>2.957944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7088</th>\n",
       "      <td>Star Wars: The Clone Wars</td>\n",
       "      <td>434.0</td>\n",
       "      <td>5.8</td>\n",
       "      <td>2008</td>\n",
       "      <td>12180</td>\n",
       "      <td>2.532774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>831</th>\n",
       "      <td>Escape to Witch Mountain</td>\n",
       "      <td>60.0</td>\n",
       "      <td>6.5</td>\n",
       "      <td>1975</td>\n",
       "      <td>14821</td>\n",
       "      <td>2.516014</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                title  vote_count  vote_average  year      id  \\\n",
       "522        Terminator 2: Judgment Day      4274.0           7.7  1991     280   \n",
       "974                            Aliens      3282.0           7.7  1986     679   \n",
       "3060  Sinbad and the Eye of the Tiger        39.0           6.3  1977   11940   \n",
       "1011                   The Terminator      4208.0           7.4  1984     218   \n",
       "8658       X-Men: Days of Future Past      6155.0           7.5  2014  127585   \n",
       "344                         True Lies      1138.0           6.8  1994   36955   \n",
       "922                         The Abyss       822.0           7.1  1989    2756   \n",
       "1668       Return from Witch Mountain        38.0           5.6  1978   14822   \n",
       "7088        Star Wars: The Clone Wars       434.0           5.8  2008   12180   \n",
       "831          Escape to Witch Mountain        60.0           6.5  1975   14821   \n",
       "\n",
       "           est  \n",
       "522   4.152059  \n",
       "974   3.707908  \n",
       "3060  3.298736  \n",
       "1011  3.251889  \n",
       "8658  3.214787  \n",
       "344   3.189446  \n",
       "922   3.045144  \n",
       "1668  2.957944  \n",
       "7088  2.532774  \n",
       "831   2.516014  "
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hybrid(15, 'Avatar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "      <th>est</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>The Terminator</td>\n",
       "      <td>4208.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>1984</td>\n",
       "      <td>218</td>\n",
       "      <td>3.438652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>522</th>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>4274.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1991</td>\n",
       "      <td>280</td>\n",
       "      <td>3.368688</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8658</th>\n",
       "      <td>X-Men: Days of Future Past</td>\n",
       "      <td>6155.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>2014</td>\n",
       "      <td>127585</td>\n",
       "      <td>3.254805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1376</th>\n",
       "      <td>Titanic</td>\n",
       "      <td>7770.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>1997</td>\n",
       "      <td>597</td>\n",
       "      <td>3.153562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4347</th>\n",
       "      <td>Piranha Part Two: The Spawning</td>\n",
       "      <td>41.0</td>\n",
       "      <td>3.9</td>\n",
       "      <td>1981</td>\n",
       "      <td>31646</td>\n",
       "      <td>3.001482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8401</th>\n",
       "      <td>Star Trek Into Darkness</td>\n",
       "      <td>4479.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>2013</td>\n",
       "      <td>54138</td>\n",
       "      <td>2.978449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>Aliens</td>\n",
       "      <td>3282.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1986</td>\n",
       "      <td>679</td>\n",
       "      <td>2.975512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>831</th>\n",
       "      <td>Escape to Witch Mountain</td>\n",
       "      <td>60.0</td>\n",
       "      <td>6.5</td>\n",
       "      <td>1975</td>\n",
       "      <td>14821</td>\n",
       "      <td>2.956026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1621</th>\n",
       "      <td>Darby O'Gill and the Little People</td>\n",
       "      <td>35.0</td>\n",
       "      <td>6.7</td>\n",
       "      <td>1959</td>\n",
       "      <td>18887</td>\n",
       "      <td>2.941469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7265</th>\n",
       "      <td>Dragonball Evolution</td>\n",
       "      <td>475.0</td>\n",
       "      <td>2.9</td>\n",
       "      <td>2009</td>\n",
       "      <td>14164</td>\n",
       "      <td>2.938638</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   title  vote_count  vote_average  year  \\\n",
       "1011                      The Terminator      4208.0           7.4  1984   \n",
       "522           Terminator 2: Judgment Day      4274.0           7.7  1991   \n",
       "8658          X-Men: Days of Future Past      6155.0           7.5  2014   \n",
       "1376                             Titanic      7770.0           7.5  1997   \n",
       "4347      Piranha Part Two: The Spawning        41.0           3.9  1981   \n",
       "8401             Star Trek Into Darkness      4479.0           7.4  2013   \n",
       "974                               Aliens      3282.0           7.7  1986   \n",
       "831             Escape to Witch Mountain        60.0           6.5  1975   \n",
       "1621  Darby O'Gill and the Little People        35.0           6.7  1959   \n",
       "7265                Dragonball Evolution       475.0           2.9  2009   \n",
       "\n",
       "          id       est  \n",
       "1011     218  3.438652  \n",
       "522      280  3.368688  \n",
       "8658  127585  3.254805  \n",
       "1376     597  3.153562  \n",
       "4347   31646  3.001482  \n",
       "8401   54138  2.978449  \n",
       "974      679  2.975512  \n",
       "831    14821  2.956026  \n",
       "1621   18887  2.941469  \n",
       "7265   14164  2.938638  "
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hybrid(500, 'Avatar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that for hybrid recommender, we get different recommendations for different users although the movie is the same. Hence, our recommendations are more personalized and tailored towards particular users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "In this notebook, I have built 4 different recommendation engines based on different ideas and algorithms. \n",
    "They are as follows:\n",
    "\n",
    "* __Simple Recommender__: This system used overall TMDB Vote Count and Vote Averages to build Top Movies Charts, in general and   for   a specific genre. The IMDB Weighted Rating System was used to calculate ratings on which the sorting was finally         performed.\n",
    "\n",
    "* __Content Based Recommender__: We built two content based engines; one that took movie overview and taglines as input and the   other which took metadata such as cast, crew, genre and keywords to come up with predictions. We also deviced a simple filter   to give greater preference to movies with more votes and higher ratings.\n",
    "\n",
    "* __Collaborative Filtering__: We used the powerful Surprise Library to build a collaborative filter based on single value       decomposition. The RMSE obtained was less than 1 and the engine gave estimated ratings for a given user and movie.\n",
    "\n",
    "* __Hybrid Engine__: We brought together ideas from content and collaborative filterting to build an engine that gave movie       suggestions to a particular user based on the estimated ratings that it had internally calculated for that user."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}