Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix upload track w prom metrics #3568

Merged
merged 1 commit into from
Jul 28, 2022
Merged

Fix upload track w prom metrics #3568

merged 1 commit into from
Jul 28, 2022

Conversation

vicky-g
Copy link
Contributor

@vicky-g vicky-g commented Jul 28, 2022

Description

Uploads were failing after this PR was merged #3546
This was the error

Error: Missing process handler for job type trackContentUpload
    at Queue.processJob (/usr/src/app/node_modules/bull/lib/queue.js:1155:7)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

It's because we added a task name when we do not process tasks by job name. This is to revert named job processes but still maintain prom metrics for queues.

Tests

  • Created user + uploaded track locally successfully
  • Checked /prometheus_metrics route for metrics and appears
# HELP audius_cn_jobs_duration_milliseconds Time to complete jobs
# TYPE audius_cn_jobs_duration_milliseconds histogram
audius_cn_jobs_duration_milliseconds_bucket{le="1",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 0
audius_cn_jobs_duration_milliseconds_bucket{le="4",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 0
audius_cn_jobs_duration_milliseconds_bucket{le="19",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 0
audius_cn_jobs_duration_milliseconds_bucket{le="84",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 0
audius_cn_jobs_duration_milliseconds_bucket{le="370",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 0
audius_cn_jobs_duration_milliseconds_bucket{le="1622",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_duration_milliseconds_bucket{le="7114",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_duration_milliseconds_bucket{le="31197",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_duration_milliseconds_bucket{le="136815",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_duration_milliseconds_bucket{le="600000",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_duration_milliseconds_bucket{le="+Inf",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_duration_milliseconds_sum{status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2764
audius_cn_jobs_duration_milliseconds_count{status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2

# HELP audius_cn_jobs_waiting_duration_milliseconds Time spent waiting for jobs to run
# TYPE audius_cn_jobs_waiting_duration_milliseconds histogram
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="1",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 0
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="4",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 0
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="19",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="84",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="370",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="1622",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="7114",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="31197",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="136815",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="600000",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_waiting_duration_milliseconds_bucket{le="+Inf",status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2
audius_cn_jobs_waiting_duration_milliseconds_sum{status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 11
audius_cn_jobs_waiting_duration_milliseconds_count{status="completed",job_name="trackContentUpload",queue_name="asyncProcessing",queue_prefix="bull"} 2

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

We should be able to generate charts from these prometheus metrics \ grafana charts

Copy link
Contributor

@dmanjunath dmanjunath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fix and so quickly!

@vicky-g vicky-g merged commit 6725b54 into master Jul 28, 2022
@vicky-g vicky-g deleted the vg-fix-con-289-2 branch July 28, 2022 18:58
cheran-senthil pushed a commit that referenced this pull request Jul 28, 2022
audius-infra pushed a commit that referenced this pull request Jul 29, 2022
## Changelog

- 2022-07-29 [ba9602c] Revert "Sanity check setIsCreatorToTrue (#3531)" (#3578) [Raymond Jacobson]
- 2022-07-29 [e4e64d7] Dont reissue manual syncs + separate manual sync timeout (#3572) [Dheeraj Manjunath]
- 2022-07-29 [e6fdba5] [CON-289] Bug - cnodeUser.clock inconsistent with clockRecord data (#3560) [Johannes Naylor]
- 2022-07-28 [9da567c] [CON-299] Fix reading clock of undefined (#3571) [Theo Ilie]
- 2022-07-28 [0639488] Unwrap transaction signature from identity response (#3528) [Marcus Pasell]
- 2022-07-28 [0db89c5] [C-324] Migrate solana service to typescript (#3563) [Dylan Jeffers]
- 2022-07-28 [d70bdb9] Add tip email notifs (#3377) [Saliou Diallo]
- 2022-07-28 [42544c7] Fix hedgehog privateKey access (#3569) [Dylan Jeffers]
- 2022-07-28 [6725b54] Fix upload track w prom metrics (#3568) [vicky :)]
- 2022-07-28 [25f99d3] Fixes for Discovery Dockerfile following alpine update (#3566) [Cheran]
- 2022-07-28 [6876cb8] Fix reactions (#3564) [Michael Piazza]
- 2022-07-28 [8cb266e] [CON-295, CON-280] Typescriptify most of state machine & fix bugs (#3565) [Theo Ilie]
- 2022-07-27 [4c30053] [C-726] Consume typed @audius/hedgehog (#3557) [Dylan Jeffers]
sliptype pushed a commit that referenced this pull request Sep 10, 2023
[1957c34] Add remove from playlist overflow action (#3621) Kyle Shanks
[67e0012] [C-2712] Clear favorites on sign-out (#3620) Dylan Jeffers
[97b6b94] [PAY-1421] Fix android reaction offsets (#3611) Michael Piazza
[9867571] [PAY-1494] Change chats copy to beginning of message history (#3617) Reed
[1b7ca71] Add overflow menu options for track list items on mobile (#3608) Kyle Shanks
[810f0df] [PAY-1487] Increase hit slop of mobile dms send button (#3618) Reed
[06dae51]  [PAY-1470] Remove border radius from track/playlist tiles (#3613) Marcus Pasell
[dbdfb84] [PAY-1469] Clear the search after creating a chat (#3616) Marcus Pasell
[3f31554] [PAY-1489] Add more space to reacted to messages (#3615) Marcus Pasell
[ed4f321] [PAY-1473] Clear input on chat change (#3614) Marcus Pasell
[e9683d9] Use emojis instead of 'heart' etc as reactions (#3602) Marcus Pasell
[192fd44] [PAY-1486] DMs: Fix purple border around unfurls (#3612) Marcus Pasell
[b0c2e4a] Lowercase library restricted handle (#3590) Isaac Solo
[a8daa72] Change text to 'Beginning of Conversation' (#3605) Marcus Pasell
[332c0c0] Developer app modal - small fixes (#3609) nicoback2
[0e43db1] [PAY-1476] Inbox unavailable copy update (#3607) Reed
[d862fd5] [PAY-1484] Allow reactions while keyboard is open on mobile (#3606) Reed
[3bd5928] [PAY-944] DMs: Add Mobile Web drawer prompting app download (#3565) Marcus Pasell
[5bea105] Wait for libs to init before initting SDK (#3604) Marcus Pasell
[f4d28e0] Add option to cache bust node_modules (#3603) Dylan Jeffers
[9b165c4] [PAY-1455] Optimize reactions perf (maybe) (#3599) Michael Piazza
[0a1a68b] [PAY-1465] Auto-nav to chat only when permitted (#3598) Reed
[72e9f92] [C-2665 C-2714] Add developer apps settings modal (#3589) Dylan Jeffers
[b787344] [PAY-1456] Fix followee-gated create chat crash (#3593) Reed
[dbae5bd] Reduce MessageLockedButton size on profile screen (#3597) Reed
[097879b] [PAY-1437] Fix drawers sharing state causing crash (#3595) Reed
[818b215] [C-2770] Fix issue where playlist library can disappear (#3596) Dylan Jeffers
[f3ada88] [PAY-1461] Format follower count in chat search screen (#3594) Reed
[eb924e7] [C-2713] Add track metadata fields to new upload form (#3574) Andrew Mendelsohn
[be56d1f] Fix mobile textinput borders + labels (#3592) Reed
[fc7541b] Update collection list id flow to add the create tile (#3591) Kyle Shanks
[dc15518] [PAY-1432] Fix chat playlist artist sometimes undefined (#3585) Reed
[399d0b5] [C-2588] Update share modal and drawer to properly add user twitter handle (#3584) Kyle Shanks
[9f30855] [C-2761] Add mutations to audius-query (#3581) Dylan Jeffers
[597dfe5] [PAY-1448] End the race between SDK and Libs, once and for all (#3588) Marcus Pasell
[56914cf] [C-2768] Update InputV2, add TextField/TextAreaField (#3587) Dylan Jeffers
[c9358b6] [CON-742] Make users and track actions v2 for v2 users/tracks even when feature flags are disabled (#3582) Theo Ilie
[a94f035] audius-query debug flag usage in README (#3586) Reed
[bceb3fe] [INF-417] Generate sourcemaps when running web:prod (#3583) Sebastian Klingler
[f4ad466] [PAY-1454] Redline mobile header to latest figma design (#3576) Michael Piazza
[eec6af0] [PAY-1405] Add padding to bottom of chat user search screen (#3580) Reed
[a6c8ece] [PAY-1435] Navigate to chat after unblocking user on mobile (#3578) Reed
[ee1b6c3] [PLAT-659] Metamask sign in/up on ACDC (#3563) Raymond Jacobson
[fbd4053] [C-2472] Update mobile track list items for playlists to have the overflow menu (#3577) Kyle Shanks
[51f44ac] [PAY-1399] Fix Android chat send button color (#3575) Reed
[7d440de] [PAY-1442] Fix drawer touchables on android (#3572) Reed
[8c90ff6] Allow drag and dropping tracks into chat (#3573) Michael Piazza
[f7489e0] Upgrade dapp store version to v1.5.26 (#3504) Dylan Jeffers
[6be0c64] [PAY-1440][PAY-1441] Mobile inbox settings UI fixes (#3571) Reed
[001007e] [PAY-1434] Dismiss keyboard on create chat screen row press (#3569) Reed
[c70e9fe] [PAY-1446] Set MessageLocked icon in profile screen to 0.4 opacity (#3570) Reed
[2cc8ad0] [C-2716] Improve notifications on fresh account (#3568) Dylan Jeffers
[01f05ba] [C-2474] Generate playlist artwork from track artwork (#3518) Dylan Jeffers
[0355c10] [PAY-1325] Remove userbadges from chat unavailable message in chat screen (#3567) Reed
[da10bbc] [PAY-1389] CTA on create chat when not permitted (#3566) Reed
@AudiusProject AudiusProject deleted a comment from linear bot Sep 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants