pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters

[ahothan]

Describe the bug
pulsar_out_bytes_total and pulsar_out_messages_total metrics should normally be counters (always increasing) in the same way as pulsar_in_bytes_total and pulsar_in_messages_total metrics.
However they behave incorrectly as they go up and down.
Furthermore, when producer is idle, these 2 metrics are no longer reporting when pulsar is scraped.

To Reproduce
Steps to reproduce the behavior:
Create a topic, produce and consume from that topic for a long period of time.
Then check how these 2 counters evolve over time

Expected behavior
These 2 counters should continuously increase.

Screenshots
See attached grafana dashboard

Desktop (please complete the following information):

• OS: Linux/Kubernetes
• deployed with sn-platform and pulsar operator
  streamnative/sn-platform:2.9.2.17

Additional context
See

• how the out metric goes up and down while the in metric behaves properly
• how the out metric stops reporting values when producer is idle while the in metric keeps reporting the last value as expected

[ahothan]

related PR: #6918

[ahothan]

corresponding stats from the cli shows correct values for msgOutCounter and msgInCounter,

root@pulsar-sn-platform-toolset-0:/pulsar# pulsar-admin topics stats benchmark/ns-cluster/pperf-test0-partition-0
{
  "msgRateIn" : 5000.481725033992,
  "msgThroughputIn" : 5206017.593981283,
  "msgRateOut" : 5000.8455656363685,
  "msgThroughputOut" : 5208596.899526573,
  "bytesInCounter" : 6463689301,
  "msgInCounter" : 6208551,
  "bytesOutCounter" : 6466197060,
  "msgOutCounter" : 6208417,

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[pgier]

I'm not sure if this is exactly the same issue, but what I'm seeing is when I disconnect from a subscription, the message/byte count for that subscription goes back down to zero.

[michaeljmarshall]

@pgier - a similar issue was fixed here #10644. Can you clarify what you mean by "disconnect from a subscription"? That'll help me with looking into the issue.

[pgier]

@michaeljmarshall I mean when the consumer disconnects. So, for example ctrl-c using pulsar-client. Pulsar topic stats show the correct totals, so maybe #10644 fixed the issue for topic stats, but the Prometheus metrics still have the old problem?

[michaeljmarshall]

That's a very helpful observation @pgier. @dlg99 - any chance you're able to look into this again?

[pgier]

The issue seems to be that pulsar_out_bytes_total and pulsar_out_messages_total are calculated by adding up the values from the currently connected consumers, instead of just using the subscription stats directly. I'm not sure why it was done this way, but I created #18451 to hopefully address this.

[ahothan]

@pgier thanks for the RCA and for fixing this!

The issue seems to be that pulsar_out_bytes_total and pulsar_out_messages_total are calculated by adding up the values from the currently connected consumers, instead of just using the subscription stats directly. I'm not sure why it was done this way, but I created #18451 to hopefully address this.