Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Prometheus Metric Names for Bull Queue Durations #3649

Merged
merged 5 commits into from
Aug 5, 2022
Merged

Fix Prometheus Metric Names for Bull Queue Durations #3649

merged 5 commits into from
Aug 5, 2022

Conversation

jonaylor89
Copy link
Contributor

Description

This PR fixes a metric name mismatch and changes the name of the metric names from
JOBS_DURATION_MILLISECONDS_HISTOGRAM => JOBS_DURATION_SECONDS_HISTOGRAM
JOBS_WAITING_DURATION_MILLISECONDS_HISTOGRAM => JOBS_WAITING_DURATION_SECONDS_HISTOGRAM

the new prometheus metrics should look like this when deployed

# HELP audius_cn_jobs_duration_seconds Time to complete jobs
# TYPE audius_cn_jobs_duration_seconds histogram
audius_cn_jobs_duration_seconds_bucket{le="1",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="2",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="4",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="8",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="17",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="35",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="71",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="145",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="295",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="600",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 0
audius_cn_jobs_duration_seconds_bucket{le="+Inf",status="completed",job_name="__default__",queue_name="monitor-state-queue"} 4
audius_cn_jobs_duration_seconds_sum{status="completed",job_name="__default__",queue_name="monitor-state-queue"} 152000
audius_cn_jobs_duration_seconds_count{status="completed",job_name="__default__",queue_name="monitor-state-queue"} 4

Tests

A new test was added that Checks the duration of a bull queue job and added a new GenericBullQueue for testing purposes

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

These changes will be monitored with the prometheus metrics mentioned above

JOBS_DURATION_SECONDS_HISTOGRAM
JOBS_WAITING_DURATION_SECONDS_HISTOGRAM

@jonaylor89 jonaylor89 requested a review from vicky-g August 5, 2022 20:52
@jonaylor89 jonaylor89 marked this pull request as ready for review August 5, 2022 20:53
@vicky-g vicky-g assigned theoilie and unassigned SidSethi Aug 5, 2022
@jonaylor89 jonaylor89 assigned SidSethi and theoilie and unassigned theoilie Aug 5, 2022
@jonaylor89 jonaylor89 requested a review from vicky-g August 5, 2022 21:22
vicky-g
vicky-g suggested changes Aug 5, 2022
View reviewed changes
Copy link
Contributor

@vicky-g vicky-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one small thing

@jonaylor89 jonaylor89 requested a review from vicky-g August 5, 2022 22:03
@vicky-g
Copy link
Contributor

vicky-g commented Aug 5, 2022

thank you! lgtm lets go

@jonaylor89 jonaylor89 merged commit f25994c into master Aug 5, 2022
@jonaylor89 jonaylor89 deleted the jn-fix-prometheus-metric-name branch August 5, 2022 23:40
audius-infra pushed a commit that referenced this pull request Aug 11, 2022
Bump sdk to v0.0.33
6290701 
## Changelog

- 2022-08-11 [8c2deb9] Use correct configuration for full endpoints (#3673) [Sebastian Klingler]
- 2022-08-10 [5ab2d47] [CON-222] Fix SequelizeUniqueConstraintError for blockchain track+user IDs (#3660) [Theo Ilie]
- 2022-08-10 [1f55962] Bump to version 0.3.64 (#3669) [Cheran]
- 2022-08-10 [8e21b0e] Bump random replica select attempt count + filter out existing nodes from healthy replica set (#3661) [vicky :)]
- 2022-08-09 [19da757] Fix TypeError for userSecondarySyncMetrics (#3668) [Theo Ilie]
- 2022-08-09 [788d482] CON-297 Auth check the requester is the primary of the observed user (#3598) [vicky :)]
- 2022-08-09 [7436c17] Add unique constraint to notification (#3657) [Joseph Lee]
- 2022-08-09 [4d03415] Add slack-secrets to jobs that use Slack notifications (#3666) [Joaquin Casares]
- 2022-08-09 [feed5ab] Revert "Repair missing wallets (#3656)" (#3659) [Raymond Jacobson]
- 2022-08-09 [8196748] Change openresty location directive from exact match to regex (#3664) [Dheeraj Manjunath]
- 2022-08-08 [1c2f069] Repair missing wallets (#3656) [Raymond Jacobson]
- 2022-08-08 [f4d5481] Upgrade service-commands to latest sdk (#3655) [Dheeraj Manjunath]
- 2022-08-08 [90a1601] safer kill_running_queries_sql (#3653) [Steve Perkins]
- 2022-08-08 [6194260] Rename processImmediateSync to processManualImmediateSync for clarity (#3632) [Dheeraj Manjunath]
- 2022-08-05 [f25994c] Fix Prometheus Metric Names for Bull Queue Durations (#3649) [Johannes Naylor]
- 2022-08-05 [a2d9937] Add remix cosign (#3650) [Joseph Lee]
- 2022-08-05 [bbf5556] [C-749] Support req pagination and ordering in hist/fav/tracks (#3643) [Raymond Jacobson]
- 2022-08-05 [d6acc56] INF add alerts for failed gcp bake jobs (#3647) [Joaquin Casares]
- 2022-08-05 [a0ee00a] postgres triggers for discovery notifications (#3464) [Joseph Lee]
- 2022-08-05 [7a4d9ea] Bump sdk to v0.0.32 [audius-infra]
audius-infra pushed a commit that referenced this pull request Aug 11, 2022
Bump sdk to v0.0.33
bac7669 
## Changelog

- 2022-08-11 [8c2deb9] Use correct configuration for full endpoints (#3673) [Sebastian Klingler]
- 2022-08-10 [5ab2d47] [CON-222] Fix SequelizeUniqueConstraintError for blockchain track+user IDs (#3660) [Theo Ilie]
- 2022-08-10 [1f55962] Bump to version 0.3.64 (#3669) [Cheran]
- 2022-08-10 [8e21b0e] Bump random replica select attempt count + filter out existing nodes from healthy replica set (#3661) [vicky :)]
- 2022-08-09 [19da757] Fix TypeError for userSecondarySyncMetrics (#3668) [Theo Ilie]
- 2022-08-09 [788d482] CON-297 Auth check the requester is the primary of the observed user (#3598) [vicky :)]
- 2022-08-09 [7436c17] Add unique constraint to notification (#3657) [Joseph Lee]
- 2022-08-09 [4d03415] Add slack-secrets to jobs that use Slack notifications (#3666) [Joaquin Casares]
- 2022-08-09 [feed5ab] Revert "Repair missing wallets (#3656)" (#3659) [Raymond Jacobson]
- 2022-08-09 [8196748] Change openresty location directive from exact match to regex (#3664) [Dheeraj Manjunath]
- 2022-08-08 [1c2f069] Repair missing wallets (#3656) [Raymond Jacobson]
- 2022-08-08 [f4d5481] Upgrade service-commands to latest sdk (#3655) [Dheeraj Manjunath]
- 2022-08-08 [90a1601] safer kill_running_queries_sql (#3653) [Steve Perkins]
- 2022-08-08 [6194260] Rename processImmediateSync to processManualImmediateSync for clarity (#3632) [Dheeraj Manjunath]
- 2022-08-05 [f25994c] Fix Prometheus Metric Names for Bull Queue Durations (#3649) [Johannes Naylor]
- 2022-08-05 [a2d9937] Add remix cosign (#3650) [Joseph Lee]
- 2022-08-05 [bbf5556] [C-749] Support req pagination and ordering in hist/fav/tracks (#3643) [Raymond Jacobson]
- 2022-08-05 [d6acc56] INF add alerts for failed gcp bake jobs (#3647) [Joaquin Casares]
- 2022-08-05 [a0ee00a] postgres triggers for discovery notifications (#3464) [Joseph Lee]
- 2022-08-05 [7a4d9ea] Bump sdk to v0.0.32 [audius-infra]
sliptype pushed a commit that referenced this pull request Sep 10, 2023
@piazzatron
[PAY-1522][PAY-1451] Fix issues with DMs notification dot (#3649)
77b965e
sliptype pushed a commit that referenced this pull request Sep 10, 2023
@dylanjeffers
v1.5.31
bd21819 
[4a26077] [C-2678] Add Stems and Source Files Modal (#3671) Andrew Mendelsohn
[6a3342a] Bump android version to 1.1.391 to fix play-store build (#3678) Dylan Jeffers
[2064212] [C-2813] Catch collectibles runtime errors (#3675) Dylan Jeffers
[29233ea] [C-2808] Improve storageNodeSelector usage and perf (#3674) Dylan Jeffers
[2de2035] Fix mobile image uri (#3670) Dylan Jeffers
[55a6089] [C-2807] Add ModalField subforms with cancellation (#3664) Andrew Mendelsohn
[e26986f] [C-2812] Use storageNodeSelector in libs, fix mobile images (#3667) Dylan Jeffers
[5daea08] Enable playlist updates on prod (#3668) Dylan Jeffers
[d90c369] [PAY-1541] Autofocus textinput on mobile search users screen (#3666) Reed
[194f894] [PAY-1542][PAY-1539][PAY-1537] Mobile chats QA improvements (#3665) Reed
[58b9028] [C-2798] Fix playlist button overlap (#3662) Dylan Jeffers
[a935588] [PAY-1538] Fix autocorrect behavior (#3661) Michael Piazza
[1a8a6e7] [PAY-1518] DMs: Align the overflow menu bottom center (#3654) Marcus Pasell
[c880b2a] [plat-1085] disable fav and repost on hidden tracks in play bar (#3653) sabrina-kiam
[d80617e] [PAY-1529] Add initial types for usdc purchases (#3651) Randy Schott
[68e87ff] [C-2787] Reregister device-token on app startup (#3660) Dylan Jeffers
[ddfa510] [C-2802] Move embed to client monorepo! (#3659) Raymond Jacobson
[6a2f008] Add Sentry error logging to Audius query C-2800 (#3656) nicoback2
[79213de] Add Amplitude events to new Write OAuth flows C-2801 (#3657) nicoback2
[6576ae4] Fix android chat message cut off at top (#3658) Reed
[160b325] [C-2677] Remix Settings Modal layout complete (#3647) Andrew Mendelsohn
[c01c022] [C-2790] Add private DogEar to desktop playlist cards (#3655) Dylan Jeffers
[50e8f4e] [PAY-1522][PAY-1451] Fix issues with DMs notification dot (#3649) Michael Piazza
[1658810] [PAY-1530] adds usdc feature flag and hooks for fetching it (#3645) Randy Schott
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vicky-g vicky-g vicky-g approved these changes

Assignees

@SidSethi SidSethi

@theoilie theoilie

Labels
size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jonaylor89 @vicky-g @SidSethi @theoilie