Support prometheus client 1.x

[pjfanning]

Please describe the feature request.

Prometheus Client 1.x releases differ from the 0.16 release.

Rationale

Use the v1 release instead of the v0 releases.

Additional context

• https://prometheus.github.io/client_java/
• the jars seem to have been renamed - eg https://github.com/prometheus/client_java/blob/main/examples/example-prometheus-properties/pom.xml
• io.prometheus:prometheus-metrics-core:1.1.0 seems to have the same package names as the io.prometheus:simpleclient jars have

[jonatan-ivanov]

This is related: #3987
Unfortunately this is not very easy since Prometheus Client 1.0 was released with a lot of breaking changes.

[cachescrubber]

Any chance the v1 client might improve the situation with #877 ?

[shakuzen]

Any chance the v1 client might improve the situation with #877 ?

It's possible. We'll revisit that when we've figured out the details of the upgrade.

[jonatan-ivanov]

Fyi: we added initial support for this (see the PR description, TL;DR: exemplars and native histograms do not work yet): #4846.
It was released yesterday in 1.13.0-M2 please feel free to try it out and give us feedback.

[cachescrubber]

Here is my initial feedback. Since the current spring-boot auto-configuration is based on the old simpleclient, it's not straight forward to test the new implementation. I used the following configuration which registers the PrometheusMetricsServlet instead of the actuator endpoint.

@Configuration
public class MicrometerPrometheusConfiguration {

  @Bean
  public PrometheusMeterRegistry PrometheusMeterRegistry() {
    return new PrometheusMeterRegistry(PrometheusConfig.DEFAULT, new PrometheusRegistry(), Clock.SYSTEM);
  }
  @Bean
  public ServletRegistrationBean<PrometheusMetricsServlet> createPrometheusMetricsEndpoint(PrometheusMeterRegistry prometheusMeterRegistry) {
    return new ServletRegistrationBean<>(new PrometheusMetricsServlet(prometheusMeterRegistry.getPrometheusRegistry()), "/metrics/*");
  }
}

I spotted another illegal metric name:

2024-03-20T09:50:46.394+01:00 ERROR 89680 --- [vf-av-check-app] [omcat-handler-0] .[.[.[.[createPrometheusMetricsEndpoint] : Servlet.service() for servlet [createPrometheusMetricsEndpoint] in context with path [] threw exception

java.lang.IllegalArgumentException: 'jvm_info': Illegal metric name. The metric name must not include the '_info' suffix. Call PrometheusNaming.sanitizeMetricName(name) to avoid this error.
	at io.prometheus.metrics.model.snapshots.MetricMetadata.validate(MetricMetadata.java:106)
	at io.prometheus.metrics.model.snapshots.MetricMetadata.<init>(MetricMetadata.java:63)
	at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.getMetadata(PrometheusMeterRegistry.java:529)
	at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.getMetadata(PrometheusMeterRegistry.java:522)
	at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.lambda$newGauge$10(PrometheusMeterRegistry.java:315)
	at io.micrometer.prometheusmetrics.MicrometerCollector.collect(MicrometerCollector.java:77)
	at io.prometheus.metrics.model.registry.MultiCollector.collect(MultiCollector.java:26)
	at io.prometheus.metrics.model.registry.PrometheusRegistry.scrape(PrometheusRegistry.java:72)
	at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.scrape(PrometheusScrapeHandler.java:112)
	at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.handleRequest(PrometheusScrapeHandler.java:53)
	at io.prometheus.metrics.exporter.servlet.jakarta.PrometheusMetricsServlet.doGet(PrometheusMetricsServlet.java:39)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)

After I disabled the JVM metrics (using management.metrics.enable.jvm=false) I could successfully scrape the application.

I haven't currently analyzed the scrape output for other incompatibilities or feed it into prometheus.

[cachescrubber]

I haven't currently analyzed the scrape output for other incompatibilities or feed it into prometheus.

Most notably difference in the scrape output is a change in the names of TYPE summary metrics. Is this expected with the new client? It seems in line with https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md

Old (Simpleclient)

# HELP spring_security_authorizations_active_seconds
# TYPE spring_security_authorizations_active_seconds summary
spring_security_authorizations_active_seconds_active_count{application="my-app",spring_security_authentication_type="n/a",spring_security_authorization_decision="unknown",spring_security_object="request",} 0.0
spring_security_authorizations_active_seconds_duration_sum{application="my-app",spring_security_authentication_type="n/a",spring_security_authorization_decision="unknown",spring_security_object="request",} 0.0

New Client

# HELP spring_security_authorizations_active_seconds
# TYPE spring_security_authorizations_active_seconds summary
spring_security_authorizations_active_seconds_count{application="my-app",spring_security_authentication_type="n/a",spring_security_authorization_decision="unknown",spring_security_object="request"} 0
spring_security_authorizations_active_seconds_sum{application="my-app",spring_security_authentication_type="n/a",spring_security_authorization_decision="unknown",spring_security_object="request"} 0.0

[shakuzen]

@cachescrubber thank you for trying things out and giving feedback. It's very appreciated. We are aware of the issues you pointed out, and we should capture them in an upgrade guide. Thanks for writing them out; that will make it easier to write the guide for other users to follow. We were hoping to get auto-config support into Boot in the milestone that was just released, but unfortunately we weren't able to get it in. The current plan is for exemplar support and auto-configuration to be worked out in time for RC1. The jvm_info issue should be fixed in snapshots when #4866 is merged. The change to the LongTaskTimer timeseries' name in Prometheus output is a breaking change that aligns it with what Prometheus convention expects (but we were never doing). We may take another look at what we could/should be doing there. Conceptually LTTs are GaugeHistograms, but Prometheus format doesn't have this concept - only OpenMetrics does.

[jonatan-ivanov]

@cachescrubber Thank you very much!

Since the current spring-boot auto-configuration is based on the old simpleclient, it's not straight forward to test the new implementation.

We already have a draft PR for Boot: spring-projects/spring-boot#40023 as Tommy mentioned above, we hope we can have this in Boot 3.3.0-RC1.

I used the following configuration which registers the PrometheusMetricsServlet instead of the actuator endpoint.

I think that might be the simplest way to try this out with Boot.

Fyi: I merged #4866 in so if you use the latest -SNAPSHOT jvm.info should be ok.

On top of what Tommy said about the breaking changes around LongTaskTimer: in the 0.x version of the Prometheus client we had more control over over the output. With 1.x, we lost that control and things that did not follow Prometheus conventions now need to.

The next thing is adding Exemplars support and updating the Boot PR with the new bits: #4867

[jonatan-ivanov]

I think this should be closed since these were merged in superseding this issue:

• Add support for Prometheus 1.x #4846
• Sanitize metric names for the Prometheus client #4866
• Add Exemplars support to Prometheus 1.x #4867
• Allow customizing PrometheusProperties via PrometheusConfig #4875

Future improvements:

• Support parsing PrometheusConfig#prometheusProperties from "get" #4922
• Support native histogram with the Prometheus 1.x client #4923
• Document support for Prometheus Client 1.x #4957