Benchmark partial

[miguelgfierro]

Description

Benchmark of some algos in Movielens, this was a request from @nikhilrj and @wutaomsft

As discussed, this will be temporal, in the future we will use directly the notebooks to execute the benchmarks

Related Issues

#464

Checklist:

• My code follows the code style of this project, as detailed in our contribution guidelines.
• I have added tests.
• I have updated the documentation accordingly.

[review-notebook-app]

Check out this pull request on ReviewNB: https://app.reviewnb.com/Microsoft/Recommenders/pull/715

Visit www.reviewnb.com to know how we simplify your Jupyter Notebook workflows.

[miguelgfierro]

Please note that this is a first iteration for you guys to review. I would need to add text in the notebook and docstrings in the utils

[anargyri]

Do we want to include some tuning for the benchmarking (either as part of this code or as a preprocessing step)?

[miguelgfierro]

@anargyri that's a good point, I guess in the benchmark we should have some "optimal" parameters precomputed via hyperparameter tuning, so we can claim that the results of the benchmark are reasonable enough.

[gramhagen]

I thought from #589 we were going to have separate notebooks / scripts for each algorithm? did i misinterpret what was meant there for scaling this out as we add new algorithms?

[loomlike]

This is super! Also very interesting that the more data to train, the less accuracy we get for both rating and ranking metrics, maybe because of the bigger number of users and items? Does this mean reco scenario is not easily generalizable (across users and items)?

Anyway see my comments. Can't wait to have this notebook in our staging and master branches.

[miguelgfierro]

I thought from #589 we were going to have separate notebooks / scripts for each algorithm? did i misinterpret what was meant there for scaling this out as we add new algorithms?

you are right, this was requested by Tao for having an initial benchmark until we start developing the benchmark through the notebooks

[anargyri]

Awesome! We now have some benchmarks; we can keep improving them as we get better hyperparameters.