Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zhangya/metrics formula #1488

Merged
merged 6 commits into from
Jul 27, 2021
Merged

Zhangya/metrics formula #1488

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1c640cb
remove special characters
Jul 26, 2021
1266cbb
fix formula
Jul 26, 2021
81cdff9
fix formula
YanZhangADS Jul 26, 2021
04ba88b
fix formula
YanZhangADS Jul 26, 2021
8266d12
fix formula
YanZhangADS Jul 26, 2021
9b6a137
fix package import
YanZhangADS Jul 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions examples/03_evaluate/als_movielens_diversity_metrics.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,10 @@
"In this notebook, we demonstrate how to evaluate a recommender using metrics other than commonly used rating/ranking metrics.\n",
"\n",
"Such metrics include:\n",
"- Coverage - The proportion of items that can be recommended. It includes two metrics:\n",
"- Coverage - We use following two metrics defined by \\[Shani and Gunawardana\\]:\n",
" \n",
" - (1) catalog_coverage, which measures the proportion of items that get recommended from the item catalog; \n",
" - (2) distributional_coverage, which measures how unequally different items are recommended in the recommendations to all users.\n",
" \n",
" We use definitions of these metrics from \\[Shani and Gunawardana\\].\n",
" - (2) distributional_coverage, which measures how equally different items are recommended in the recommendations to all users.\n",
"\n",
"- Novelty - A more novel item indicates it is less popular, i.e. it gets recommended less frequently.\n",
"We use the definition of novelty from \\[Castells et al.\\]\n",
Expand Down Expand Up @@ -86,7 +84,7 @@
"$$\n",
"where $M$ is the set of users and $N_r(u)$ the set of recommendations for user $u$. Finally, diversity is defined as\n",
"$$\n",
"\\textrm{diversity} = 1 - \\textrm{IL} ~.\n",
"\\textrm{diversity} = 1 - \\textrm{IL}\n",
"$$\n"
],
"cell_type": "markdown",
Expand All @@ -101,13 +99,15 @@
"$$\n",
"p(i) = \\frac{|M_t (i)|} {|\\textrm{train_df}|}\n",
"$$\n",
"where $M_t (i)$ is the set of users who have interacted with item $i$ in the historical data. The novelty of an item is then defined as\n",
"where $M_t (i)$ is the set of users who have interacted with item $i$ in the historical data. \n",
"\n",
"The novelty of an item is then defined as\n",
"$$\n",
"\\textrm{novelty}(i) = -\\log_2⁡ p(i) \n",
"\\textrm{novelty}(i) = -\\log_2 p(i)\n",
"$$\n",
"and the novelty of the recommendations across all users is defined as\n",
"$$\n",
"\\textrm{novelty} = \\sum_{i \\in N_r} \\frac{|M_r (i)|}{|\\textrm{reco_df}|} \\textrm{novelty}(i) ~.\n",
"\\textrm{novelty} = \\sum_{i \\in N_r} \\frac{|M_r (i)|}{|\\textrm{reco_df}|} \\textrm{novelty}(i)\n",
"$$\n"
],
"cell_type": "markdown",
Expand All @@ -126,7 +126,7 @@
"serendipity is defined as\n",
"$$\n",
"\\textrm{serendipity} = \\frac{1}{|M|} \\sum_{u \\in M_r}\n",
"\\frac{1}{|N_r (u)|} \\sum_{i \\in N_r (u)} \\big(1 - \\textrm{expectedness}(i|u) \\big) \\, \\textrm{relevance}(i) ~.\n",
"\\frac{1}{|N_r (u)|} \\sum_{i \\in N_r (u)} \\big(1 - \\textrm{expectedness}(i|u) \\big) \\, \\textrm{relevance}(i)\n",
"$$\n"
],
"cell_type": "markdown",
Expand Down