7429/feature/add trending score to solr

[benbdeitch]

Closes #7429

This PR adds support for trending scores to Solr, allowing us to better track which works are achieving a statistically notable increase in popularity. It adds several new fields, and comes with two scripts to be run-- one daily, the other hourly, to keep this information constantly up to date.

Currently, it's still in draft mode, as there is currently no code to automatically run the scripts.

Technical

This implementation uses Solr's ability to update documents in place, which requires the new trending fields to not be stored or indexed, and instead treated as a docValue. Essentially, they are left out of Solr's inverted index, and instead treated as a more usual document-to-value mapping.

This is both A) more performant than atomic updates, and B) avoids the issues that atomic updates can have with copyfield values.

The relevant cron commands are located in an added file, docker/cron.local

1. Delete your solr container and all related volumes.
2. Run docker compose up.
3. Going to your local solr instance, run a search for a work on Solr (e.g. key:"/works/OL54120W"), and check to ensure that the new fields are present.
4. Save a work to your 'want-to-read' list.
5. Set up a docker/cron.local file to run the cron jobs in, along with a new container. Change the times on the cron tasks to run more frequently; (* * * * *) will make them run every minute.
6. Make sure the container has access to both dbnet and webnet networks, and has depends on: db.
7. After a minute or so, run the search on Solr again, and see if the appropriate trending fields have updated. You can also check the logs of the cron-jobs container in Docker, to see if they're running correctly.

Screenshot

Stakeholders

@cdrini

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 21.27660% with 37 lines in your changes missing coverage. Please review.

Project coverage is 17.12%. Comparing base (ce16a79) to head (a5545e4).
Report is 537 commits behind head on master.

Files with missing lines	Patch %	Lines
openlibrary/plugins/openlibrary/js/signup.js	40.00%	13 Missing and 2 partials ⚠️
...ary/plugins/openlibrary/js/editions-table/index.js	0.00%	9 Missing and 3 partials ⚠️
...y/plugins/openlibrary/js/bulk-tagger/BulkTagger.js	0.00%	9 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9878      +/-   ##
==========================================
+ Coverage   16.06%   17.12%   +1.06%     
==========================================
  Files          90       89       -1     
  Lines        4769     4752      -17     
  Branches      832      831       -1     
==========================================
+ Hits          766      814      +48     
+ Misses       3480     3428      -52     
+ Partials      523      510      -13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cdrini]

@benbdeitch notes this should be deleted; but maybe copy over the comments to the other file!

[cdrini]

Todo: investigate this one 😂

[cdrini]

I think this might not work, since _work is a database work, not a solr work.

I think the way we might have to solve this is to make a network requests to fetch solr work data (I don't think we currently use the solr work data anywhere). We do something sort of like this in the author updater; since that has to make a network request to fetch all the author's books in solr. See if we can copy that rough pattern about when/how to make the network request. This is slightly uncharted territory, so might require a unique approach.

The way to test this is:

1. Add the book to reading logs/etc so that it's z-score gets computed and saved in solr
2. Confirm the trending score is non-zero in solr
3. Edit the the work in question ; this will trigger a new reindex of the work record

Expected: The trending score should be copied over from the previous solr record
Actual: The trending score gets zero-d by this line.

[mekarpeles]

Can this one be closed in favor of #10057? And if so @benbdeitch do you mind doing that?