ui: latency graphs reporting 0 latency incorrectly

[petermattis]

The above is from a cluster that is performing ~2k ops/sec. It is impossible for the latency to be 0.

[mrtracy]

What cluster was this?

[mrtracy]

There's only one line there, but the 99th percentile graph shows a line for every node (unless you are looking at a single node). Is it possible that this node has no requests?

Edit: I took a look, yeah something might not be getting recorded correctly on the backend.

[petermattis]

This was a 1-node cluster so it was definitely receiving requests. I've seen this happen multiple times.

[mrtracy]

After reading the code, i'm surprised it ever has non-zero values; the sliding histogram is only advanced when the histogram values are read. This means that, immediately before returning values, the histogram might rotate all previously recorded data out of the histogram, resulting in a zero value.

Two possible fixes:

• "tick" histograms when values are recorded, rather than when values are read. This would involve taking a timestamp whenever a value is recorded; probably not too expensive, but it's something to consider.
• Set the histogram duration to a longer interval than the recording window. This is not as accurate as the solution above (some values will be recorded in the wrong window), but is probably accurate enough for our purposes and involves taking fewer timestamps. That would also align with the suggestion in ui: latency metrics P99/P50 on cluster page fluctuate a lot #12835, although too much smoothing might make the data unintuitive.

[rjnn]

What's the current status on this issue?

[mrtracy]

While a "fix" was prepared to create non-zero values, we are looking into prioritizing a much bigger change to the time series system that will allow us to handle histograms correctly (see #7896). This will allow for data which is far more intuitive (no odd "window" sizes), as well as the correct computation of quantiles across the cluster.