Das client using chunks metric

[Tristan-Wilson]

This PR adds a metric on the batch poster side to indicate which DAS backends are using the chunked store method and which are using the legacy method.

The metrics that are added are:

arb_das_rpcclient_<sanitized backend hostname>_legacystore_total
arb_das_rpcclient_<sanitized backend hostname>_chunkedstore_total

[Tristan-Wilson]

Fixes NIT-2742

[gligneul]

LGTM! Does this PR replace #2549?

[ganeshvanahalli]

LGTM

[Tristan-Wilson]

LGTM! Does this PR replace #2549?

Oops, I had meant for this PR to be based on that PR, changed the base now. Thanks.

The base branch was changed.

[tsahee]

Didn't review yet. Marking as approved because it has previous approvals that were dismissed when it's parent changed. That way it should come up in my searches for approved PRs

[Tristan-Wilson]

In promql you can make a query to discover and show the rates for the metrics with different names like this:

rate(
  label_replace(
    {__name__=~"arb_das_rpcclient_.*_legacystore_total", chain="nova"},
    "metric_name", "$0", "__name__", "(.*)"
  )[${__rate_interval}:15s]
)

• label_replace(...): Adds a metric_name label to each time series, which is required by rate which needs the metrics to have differing label sets (__name__ does not count as a label in this case).
• [${__rate_interval}:15s]: The subquery notation that creates a range vector from the result of label_replace(). rate(...) requires a range vector as its input, which label_replace does not provide by default.

[PlasmaPower]

One comment but generally LGTM

[PlasmaPower]

It looks like CanonicalizeMetricName doesn't filter out periods, which don't seem to be an allowed Prometheus metric name but are of course common in URLs.

[Tristan-Wilson]

I think it does replace them. The regex is picking characters that don't match the range, which doesn't include ., and replacing them with _.

// Prometheus metric names must contain only chars [a-zA-Z0-9:_]
func CanonicalizeMetricName(metric string) string {
	invalidPromCharRegex := regexp.MustCompile(`[^a-zA-Z0-9:_]+`)
	return invalidPromCharRegex.ReplaceAllString(metric, "_")

}

If you look at the Nova dashboard, first panel, you can see the dots in domains have been replaced with _.

[joshuacolvin0]

Prometheus metrics are really for things that should be tracked long term, but legacy/chunked is just a transition between new and old, so better to just emit an info log when using legacy (no log when doing chunked)

[Tristan-Wilson]

Will redo as a new PR to just log when we are using the legacy method.