Optimize trending requests

[isaacsolo]

Description

I believe this is at least a 5x improvement on trending requests. The biggest bottleneck in trending is populating metadata. In some cases this can take 14 seconds alone.

{"level": "INFO", "msg": "got track play count completed 14.473582744598389", "timestamp": "2023-10-02 22:46:47,515", "service": "server", "otelSpanID": "8d775f5305530e6a", "otelTraceID": "18ac26b35ba35c53c24b3cea00435d6d", "otelServiceName": "discovery-provider", "path": "src.queries.query_helpers", "function": "populate_user_metadata"}
{"level": "INFO", "msg": "populate_track_metadata completed in 14.91640305519104", "timestamp": "2023-10-02 22:46:47,609", "service": "server", "otelSpanID": "8d775f5305530e6a", "otelTraceID": "18ac26b35ba35c53c24b3cea00435d6d", "otelServiceName": "discovery-provider", "path": "src.queries.query_helpers", "function": "populate_user_metadata", "duration": 14.91640305519104}

The main reason this is slow is because we're populating metadata for 100 tracks (the default limit of trending). Once we have those tracks fully populated, then we filter by the limit / offset. For the typical request of 10 trending tracks the client makes, this is 10x the work.

This would be optimized if we limit / offset before populating metadata.

Before:
It can take multiple seconds in bad cases but consistently .5 secs once postgres caches.

real	0m6.051s

ubuntu@isaac-sandbox:~/audius-docker-compose$ time curl 'localhost:5000/v1/full/tracks/trending?limit=4&offset=9&time=week&user_id=0MMZ6' > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20262  100 20262    0     0   6908      0  0:00:02  0:00:02 --:--:--  6905

real	0m2.942s

ubuntu@isaac-sandbox:~/audius-docker-compose$ time curl 'localhost:5000/v1/full/tracks/trending?limit=4&offset=9&time=week&user_id=0MMZ6' > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20262  100 20262    0     0  43952      0 --:--:-- --:--:-- --:--:-- 43952

real	0m0.468s
user	0m0.000s
sys	0m0.008s

After:
Always under .2 seconds.

ubuntu@isaac-sandbox:~/audius-docker-compose$ time curl 'localhost:5000/v1/full/tracks/trending?limit=4&offset=9&time=week&user_id=0MMZ6' > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20208  100 20208    0     0   177k      0 --:--:-- --:--:-- --:--:--  177k

real	0m0.119s
user	0m0.004s
sys	0m0.004s
ubuntu@isaac-sandbox:~/audius-docker-compose$ time curl 'localhost:5000/v1/full/tracks/trending?limit=4&offset=9&time=week&user_id=0MMZ6' > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20208  100 20208    0     0  19.2M      0 --:--:-- --:--:-- --:--:-- 19.2M

real	0m0.007s
user	0m0.007s
sys	0m0.000s
ubuntu@isaac-sandbox:~/audius-docker-compose$ time curl 'localhost:5000/v1/full/tracks/trending?limit=4&offset=9&time=week&user_id=0MMZ6' > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20208  100 20208    0     0   112k      0 --:--:-- --:--:-- --:--:--  113k

real	0m0.182s
user	0m0.000s
sys	0m0.008s

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.
Tested on sandbox, confirmed responses are consistent while adjusting offset / limit.

[stereosteve]

sick