refactor(metrics): support streaming metrics level

[fuyufjh]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Closes #11855

Reduce the streaming metrics by

• disabling all tokio metrics
• folding streaming Histogram metrics in fragment-level instead of actor-level

under steaming_metrics_level = INFO. Switch to DEBUG to get the original behavior.

This PR is based on #11895 and moved some structs to common crate in order to reuse them.

Excluding the state_store metrics (wich is significantly reduced by #11895), the metrics size will be cut off more than half (8.5MB -> 4.1MB per collecting request). (Sampled from a customer with 500+ materialized views)

-rw-r--r--  1 eric  staff    46M Aug 31 14:10 prom_cn.txt
-rw-r--r--  1 eric  staff   8.5M Aug 31 14:20 prom_cn_without_state_store.txt
-rw-r--r--  1 eric  staff   4.1M Aug 31 14:21 prom_cn_without_state_store_with_this_pr.txt

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[yuhao-su]

LGTM if you have checked the grafana figures locally

[fuyufjh]

LGTM if you have checked the grafana figures locally

Preview:

[zwang28]

LGTM.
Note that there has already been a metrics_level defined in src/common/src/config.rs, which determines whether starts Prometheus for compute node/frontend/compactor. Shall we remove storage_metric_level and streaming_metric_level, extend and use the existent metrics_level instead? The downside is we can no longer set metric level for storage and streaming separately.

[yuhao-su]

Shall we remove storage_metric_level and streaming_metric_level, extend and use the existent metrics_level instead?

Looks like the pr already did this?
https://github.com/risingwavelabs/risingwave/pull/11994/files#diff-c81fa6ff8ebe6cc6822331e0f5ae282664862104a8b9189edce5e9336685690fR570

[zwang28]

Shall we remove storage_metric_level and streaming_metric_level, extend and use the existent metrics_level instead?

Looks like the pr already did this? https://github.com/risingwavelabs/risingwave/pull/11994/files#diff-c81fa6ff8ebe6cc6822331e0f5ae282664862104a8b9189edce5e9336685690fR570

No, I mean this one

risingwave/src/common/src/config.rs

Line 366 in 0132954

pub metrics_level: u32,

[fuyufjh]

Shall we remove storage_metric_level and streaming_metric_level, extend and use the existent metrics_level instead?

Looks like the pr already did this? https://github.com/risingwavelabs/risingwave/pull/11994/files#diff-c81fa6ff8ebe6cc6822331e0f5ae282664862104a8b9189edce5e9336685690fR570

No, I mean this one

risingwave/src/common/src/config.rs

Line 366 in 0132954

pub metrics_level: u32,

+1. Let me merge this PR first 🤣 Didn't notice it was blocked by CI...

[fuyufjh]

cc. @zwang28 This is a side effect of changing GLOBAL_STREAMING_METRICS from LazyLock to OnceLock. It's difficult to tell whether GLOBAL_STREAMING_METRICS is guaranteed to be initialized, so we might have to put a if let here to check its availability, which seems a little bit ugly.

[codecov]

Codecov Report

Merging #11994 (dbb7716) into main (fdaf30e) will increase coverage by 0.00%.
Report is 1 commits behind head on main.
The diff coverage is 62.57%.

@@           Coverage Diff           @@
##             main   #11994   +/-   ##
=======================================
  Coverage   69.76%   69.76%           
=======================================
  Files        1404     1404           
  Lines      235070   235137   +67     
=======================================
+ Hits       163992   164048   +56     
- Misses      71078    71089   +11     

Flag	Coverage Δ
rust	69.76% <62.57%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
src/common/src/metrics.rs	13.15% <ø> (ø)
src/compute/src/server.rs	0.00% <0.00%> (ø)
src/storage/src/monitor/mod.rs	0.00% <ø> (ø)
src/stream/src/executor/dynamic_filter.rs	93.42% <ø> (ø)
src/stream/src/executor/hash_join.rs	96.58% <ø> (ø)
src/stream/src/task/stream_manager.rs	1.74% <0.00%> (-0.01%)	⬇️
...c/stream/src/task/barrier_manager/managed_state.rs	83.24% <66.66%> (-0.19%)	⬇️
src/common/src/metrics/relabeled_metric.rs	79.16% <88.09%> (ø)
src/common/src/config.rs	85.30% <100.00%> (+0.08%)	⬆️
...storage/src/monitor/hummock_state_store_metrics.rs	77.04% <100.00%> (ø)
... and 3 more

... and 7 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more