Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dbnode] Fix m3tsz encoding precision issue #3872

Merged
merged 3 commits into from
Oct 26, 2021

Conversation

linasm
Copy link
Collaborator

@linasm linasm commented Oct 26, 2021

What this PR does / why we need it:
There was avoidable precision loss in M3TSZ encoding. Accumulated multiplication was being done in a loop:

val = val * 10.0

Replaced this accumulation with multiplication by an precomputed lookup table (1, 10, 100, ...):

val := v * multipliers[int(mult)] * sign

This improves the precision of encoded datapoint values, as demonstrated by the new unit test.

Special notes for your reviewer:

Does this PR introduce a user-facing and/or backwards incompatible change?:
NONE

Does this PR require updating code package or user-facing documentation?:
NONE

@codecov
Copy link

codecov bot commented Oct 26, 2021

Codecov Report

Merging #3872 (40bc7ff) into master (40bc7ff) will not change coverage.
The diff coverage is n/a.

❗ Current head 40bc7ff differs from pull request most recent head e3b288d. Consider uploading reports for the commit e3b288d to get more accurate results

Impacted file tree graph

@@          Coverage Diff           @@
##           master   #3872   +/-   ##
======================================
  Coverage    57.0%   57.0%           
======================================
  Files         552     552           
  Lines       63272   63272           
======================================
  Hits        36124   36124           
  Misses      23940   23940           
  Partials     3208    3208           
Flag Coverage Δ
aggregator 64.3% <0.0%> (ø)
cluster ∅ <0.0%> (∅)
collector 58.4% <0.0%> (ø)
dbnode 60.6% <0.0%> (ø)
m3em 46.4% <0.0%> (ø)
metrics 19.7% <0.0%> (ø)
msg 74.2% <0.0%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 40bc7ff...e3b288d. Read the comment docs.

Copy link
Collaborator

@robskillington robskillington left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@linasm linasm merged commit fe1a38a into master Oct 26, 2021
@linasm linasm deleted the linasm/dbnode-fix-m3tsz-precision-issue branch October 26, 2021 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants