output/cloudv2: Optimized metric sinks #3085
Conversation
Codecov Report
@@ Coverage Diff @@
## master #3085 +/- ##
==========================================
+ Coverage 73.69% 73.74% +0.04%
==========================================
Files 241 241
Lines 18435 18451 +16
==========================================
+ Hits 13585 13606 +21
+ Misses 3977 3973 -4
+ Partials 873 872 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
|
codebien
force-pushed
the
cloud-v2-optimized-sinks
branch
from
1ebd2c3
to
9152f40
Compare
codebien
force-pushed
the
ingestion/histogram
branch
2 times, most recently
from
5c61d6f
to
ede28b0
Compare
codebien
force-pushed
the
cloud-v2-optimized-sinks
branch
2 times, most recently
from
3d4683b
to
eb0c005
Compare
| type gauge struct { | ||
| Last float64 | ||
| Sum float64 | ||
| Min, Max float64 | ||
| Avg float64 | ||
| Count uint32 | ||
|
|
||
| minSet bool | ||
| } |
There was a problem hiding this comment.
Moving here the discussion from #3083 (comment)
Shouldn't this be just Last? Which is what Gauge basically is?
There was a problem hiding this comment.
Hey @esquonk, apparently other protocols (OTel, Prometheus remote write) don't support gauge in the same way. We may have issues if we want to migrate to them in the future. Considering this are not used at the moment, can we just drop them?
There was a problem hiding this comment.
We have those in the backend historically, and they are used. For example, if we have a "max" query for gauge metric, the code will look at "max" column.
We could just fill all the columns with "last" value at the cost losing a bit of data, but if we can collect some more statistical values, it make sense to do so. Unless it is too expensive to maintain.
As for compatibility with other protocols, I don't see how having more data internally can cause issues with that. If you need to produce an output in some protocol that only has last, you can just put last in there and ignore other values in your output.
codebien
force-pushed
the
ingestion/histogram
branch
2 times, most recently
from
d2add9c
to
128e55a
Compare
codebien
force-pushed
the
cloud-v2-optimized-sinks
branch
from
eb0c005
to
b50020a
Compare
There was a problem hiding this comment.
LGTM! in general with the understandign that Gauge is not really a Gauge because the k6 cloud backend wants it like this.
I have left some small comments, and one big one which is mostly an optimizaiton idea for after we have merged everything and this output works.
| type metricValue interface { | ||
| Add(v float64) | ||
| } |
There was a problem hiding this comment.
for future: I kind of wonder if we should not just have a
type metricSinks struct {
counters map[metrics.TimeSeries]counter
gauges map[metrics.TimeSeries]gauge
trends map[metrics.TimeSeries]histogram
rates map[metrics.TimeSeries]rate
}as a replacement for what map[metrics.TimeSeries]metricValue is basically used now.
We will drop the interface here.
We will drop a bunch of the switches including in the final write path where you will just write all the countes, all the gauges and so on. Instead of checking which is which all the time.
IMO this will be faster but we definitely should benchmark it and this code is good enough.
| } | ||
|
|
||
| type rate struct { | ||
| NonZeroCount uint32 |
There was a problem hiding this comment.
tiny nit: I would suggest rename this to something (I would find) more explicit: StrictlyPositiveCount instead.
There was a problem hiding this comment.
They are not only positive, negative are accepted, only zero is excluded. The condition:
Lines 75 to 77 in 986db71
It address this request change #3071 (comment)