When httpServerEnabled=true, Prometheus reports doesn't set content-type

[addisonj]

BUG REPORT

Describe the bug

When running with httpServerEnabled set to true, /metrics endpoint stops sending a content-type header (this uses the MetricsService, see code here https://github.com/apache/bookkeeper/blob/28dee8464764b0edceeb31ce24424f0947b5789b/bookkeeper-server/src/main/java/org/apache/bookkeeper/server/http/service/MetricsService.java).
With the httpServerEnabled=false the PrometheusServlet is used, and it does set a content-type header (

bookkeeper/bookkeeper-stats-providers/prometheus-metrics-provider/src/main/java/org/apache/bookkeeper/stats/prometheus/PrometheusServlet.java

Line 44 in 28dee84

resp.setContentType(TextFormat.CONTENT_TYPE_004);

)

The content-type header is expected by some prometheus tools to differentiate between plaintext and protobuf formatted metrics and no content-type is considered an error.

To Reproduce

Steps to reproduce the behavior:

1. Set httpServerEnabled=false and enable prometheus reporter
2. run curl localhost:8000/metrics and notice that a content-type header is set
3. Set httpServerEnabled=true and enable prometheus reporter
4. run curl localhost:8000/metrics -v notice that there is no content-type header set

Expected behavior

I expect a content-type header

It seems like the statsProvider interface should have a method for allowing to set headers on the response. Default to a given content type might cause problems with other providers.

[eolivelli]

Good catch.
Do you want to work on a fix?
Maybe the implementation of the provider may have some callback to return one or more headers, with Content-Type among them

[addisonj]

yeah, I was just capturing this, don't have time at the moment but hope I get a minute to circle back soon

[zhongxp1219]

hi, @shoothzj @fu-turer when i set httpServerEnabled=true, prometheus will not provide metrics, the code show as follows.

And, if i modified the code that let prometheus provides metric endpoint while set httpServerEnabled=true, it will return content-Type with text/plain:

is there any mistakes i make ? thanks. $.$

[huanghuazhang]

For your reference，I have to uncomment the line below in conf/bk_server.conf to make the /metrics api available.
Bookkeeper Version: 4.16.2
statsProviderClass=org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider