statistics: fix get wrong topn with data -0 and 0 #57464

hawkingrei · 2024-11-18T10:20:16Z

What problem does this PR solve?

Issue Number: close #57448

Problem Summary:

What changed and how does it work?

first, you can add the following code to debug this code, then create the table and insert the data as describe by the issue.

Subject: [PATCH] statistics: fix get wrong topn with data -0 and 0

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
---
 pkg/statistics/builder.go | 36 ++++++++++++++++++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)

diff --git a/pkg/statistics/builder.go b/pkg/statistics/builder.go
index f93afe6c73..761ca90402 100644
--- a/pkg/statistics/builder.go
+++ b/pkg/statistics/builder.go
@@ -16,6 +16,7 @@ package statistics
 
 import (
        "bytes"
+       "fmt"
        "math"
 
        "github.com/pingcap/errors"
@@ -336,7 +337,20 @@ func BuildHistAndTopN(
        }
        curCnt := float64(0)
        var corrXYSum float64
-
+       fmt.Println("check data")
+       for i := int64(0); i < sampleNum; i++ {
+               strstr, err := samples[i].Value.ToString()
+               if err != nil {
+                       continue
+               }
+               fmt.Printf("str: %s\n", strstr)
+               sampleBytes, err := getComparedBytes(samples[i].Value)
+               if err != nil {
+                       continue
+               }
+               fmt.Printf("% X\n", sampleBytes)
+       }
+       fmt.Println("start debug")
        // Iterate through the samples
        for i := int64(0); i < sampleNum; i++ {
                if isColumn {
@@ -362,6 +376,9 @@ func BuildHistAndTopN(
                }
                // case 2-2, now topn is empty: append the "current" count directly
                if len(topNList) == 0 {
+                       fmt.Println("===== =======")
+                       fmt.Printf("first: % X %d\n", cur)
+                       fmt.Println("===== =======")
                        topNList = append(topNList, TopNMeta{Encoded: cur, Count: uint64(curCnt)})
                        cur, curCnt = sampleBytes, 1
                        continue
@@ -380,13 +397,28 @@ func BuildHistAndTopN(
                }
                topNList = append(topNList, TopNMeta{})
                copy(topNList[j+1:], topNList[j:])
+               fmt.Println("===== =======")
+               fmt.Printf("insert: % X %d %d\n", cur, curCnt, j)
                topNList[j] = TopNMeta{Encoded: cur, Count: uint64(curCnt)}
                if len(topNList) > numTopN {
                        topNList = topNList[:numTopN]
                }
+
+               for _, item := range topNList {
+                       fmt.Printf("% X %d %d\n", item.Encoded, item.Count, j)
+               }
+               fmt.Println("===== =======")
                cur, curCnt = sampleBytes, 1
        }
-
+       checkMap := make(map[string]struct{}, len(samples))
+       for _, sample := range topNList {
+               _, ok := checkMap[string(sample.Encoded)]
+               if !ok {
+                       checkMap[string(sample.Encoded)] = struct{}{}
+               } else {
+                       fmt.Println("WTF")
+               }
+       }
        // Calc the correlation of the column between the handle column.
        if isColumn {
                hg.Correlation = calcCorrelation(sampleNum, corrXYSum)
-- 
2.47.0

so you will get the print log as following.

check data
str: -8
05 3F DF FF FF FF FF FF FF
str: -8
05 3F DF FF FF FF FF FF FF
str: -8
05 3F DF FF FF FF FF FF FF
str: -8
05 3F DF FF FF FF FF FF FF
str: -8
05 3F DF FF FF FF FF FF FF
str: -8
05 3F DF FF FF FF FF FF FF
str: -8
05 3F DF FF FF FF FF FF FF
str: -8
05 3F DF FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -6
05 3F E7 FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -4
05 3F EF FF FF FF FF FF FF
str: -2
05 3F FF FF FF FF FF FF FF
str: -2
05 3F FF FF FF FF FF FF FF
str: -2
05 3F FF FF FF FF FF FF FF
str: -2
05 3F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: 0
05 80 00 00 00 00 00 00 00
str: -0
05 7F FF FF FF FF FF FF FF
str: 0
05 80 00 00 00 00 00 00 00
str: -0
05 7F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: 0
05 80 00 00 00 00 00 00 00
str: 0
05 80 00 00 00 00 00 00 00
str: -0
05 7F FF FF FF FF FF FF FF
str: -0
05 7F FF FF FF FF FF FF FF
str: 2
05 C0 00 00 00 00 00 00 00
str: 2
05 C0 00 00 00 00 00 00 00
str: 2
05 C0 00 00 00 00 00 00 00
str: 2
05 C0 00 00 00 00 00 00 00
str: 2
05 C0 00 00 00 00 00 00 00
str: 2
05 C0 00 00 00 00 00 00 00
str: 2
05 C0 00 00 00 00 00 00 00
str: 2
05 C0 00 00 00 00 00 00 00
str: 2
05 C0 00 00 00 00 00 00 00
str: 2
05 C0 00 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 4
05 C0 10 00 00 00 00 00 00
str: 6
05 C0 18 00 00 00 00 00 00
str: 6
05 C0 18 00 00 00 00 00 00
str: 6
05 C0 18 00 00 00 00 00 00
str: 6
05 C0 18 00 00 00 00 00 00
str: 6
05 C0 18 00 00 00 00 00 00
str: 6
05 C0 18 00 00 00 00 00 00
str: 6
05 C0 18 00 00 00 00 00 00
str: 6
05 C0 18 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
str: 8
05 C0 20 00 00 00 00 00 00
start debug
===== =======
first: 05 3F DF FF FF FF FF FF FF %!d(MISSING)
===== =======
===== =======
insert: 05 3F E7 FF FF FF FF FF FF %!d(float64=15) 0
05 3F E7 FF FF FF FF FF FF 15 0
05 3F DF FF FF FF FF FF FF 8 0
===== =======
===== =======
insert: 05 3F EF FF FF FF FF FF FF %!d(float64=11) 1
05 3F E7 FF FF FF FF FF FF 15 1
05 3F EF FF FF FF FF FF FF 11 1
05 3F DF FF FF FF FF FF FF 8 1
===== =======
===== =======
insert: 05 3F FF FF FF FF FF FF FF %!d(float64=4) 3
05 3F E7 FF FF FF FF FF FF 15 3
05 3F EF FF FF FF FF FF FF 11 3
05 3F DF FF FF FF FF FF FF 8 3
05 3F FF FF FF FF FF FF FF 4 3
===== =======
===== =======
insert: 05 7F FF FF FF FF FF FF FF %!d(float64=7) 3
05 3F E7 FF FF FF FF FF FF 15 3
05 3F EF FF FF FF FF FF FF 11 3
05 3F DF FF FF FF FF FF FF 8 3
05 7F FF FF FF FF FF FF FF 7 3
05 3F FF FF FF FF FF FF FF 4 3
===== =======
===== =======
insert: 05 80 00 00 00 00 00 00 00 %!d(float64=1) 5
05 3F E7 FF FF FF FF FF FF 15 5
05 3F EF FF FF FF FF FF FF 11 5
05 3F DF FF FF FF FF FF FF 8 5
05 7F FF FF FF FF FF FF FF 7 5
05 3F FF FF FF FF FF FF FF 4 5
05 80 00 00 00 00 00 00 00 1 5
===== =======
===== =======
insert: 05 7F FF FF FF FF FF FF FF %!d(float64=1) 5
05 3F E7 FF FF FF FF FF FF 15 5
05 3F EF FF FF FF FF FF FF 11 5
05 3F DF FF FF FF FF FF FF 8 5
05 7F FF FF FF FF FF FF FF 7 5
05 3F FF FF FF FF FF FF FF 4 5
05 7F FF FF FF FF FF FF FF 1 5
05 80 00 00 00 00 00 00 00 1 5
===== =======
===== =======
insert: 05 80 00 00 00 00 00 00 00 %!d(float64=1) 5
05 3F E7 FF FF FF FF FF FF 15 5
05 3F EF FF FF FF FF FF FF 11 5
05 3F DF FF FF FF FF FF FF 8 5
05 7F FF FF FF FF FF FF FF 7 5
05 3F FF FF FF FF FF FF FF 4 5
05 80 00 00 00 00 00 00 00 1 5
05 7F FF FF FF FF FF FF FF 1 5
05 80 00 00 00 00 00 00 00 1 5
===== =======
===== =======
insert: 05 7F FF FF FF FF FF FF FF %!d(float64=3) 5
05 3F E7 FF FF FF FF FF FF 15 5
05 3F EF FF FF FF FF FF FF 11 5
05 3F DF FF FF FF FF FF FF 8 5
05 7F FF FF FF FF FF FF FF 7 5
05 3F FF FF FF FF FF FF FF 4 5
05 7F FF FF FF FF FF FF FF 3 5
05 80 00 00 00 00 00 00 00 1 5
05 7F FF FF FF FF FF FF FF 1 5
05 80 00 00 00 00 00 00 00 1 5
===== =======
===== =======
insert: 05 80 00 00 00 00 00 00 00 %!d(float64=2) 6
05 3F E7 FF FF FF FF FF FF 15 6
05 3F EF FF FF FF FF FF FF 11 6
05 3F DF FF FF FF FF FF FF 8 6
05 7F FF FF FF FF FF FF FF 7 6
05 3F FF FF FF FF FF FF FF 4 6
05 7F FF FF FF FF FF FF FF 3 6
05 80 00 00 00 00 00 00 00 2 6
05 80 00 00 00 00 00 00 00 1 6
05 7F FF FF FF FF FF FF FF 1 6
05 80 00 00 00 00 00 00 00 1 6
===== =======
===== =======
insert: 05 7F FF FF FF FF FF FF FF %!d(float64=2) 6
05 3F E7 FF FF FF FF FF FF 15 6
05 3F EF FF FF FF FF FF FF 11 6
05 3F DF FF FF FF FF FF FF 8 6
05 7F FF FF FF FF FF FF FF 7 6
05 3F FF FF FF FF FF FF FF 4 6
05 7F FF FF FF FF FF FF FF 3 6
05 7F FF FF FF FF FF FF FF 2 6
05 80 00 00 00 00 00 00 00 2 6
05 80 00 00 00 00 00 00 00 1 6
05 7F FF FF FF FF FF FF FF 1 6
05 80 00 00 00 00 00 00 00 1 6
===== =======
===== =======
insert: 05 C0 00 00 00 00 00 00 00 %!d(float64=10) 2
05 3F E7 FF FF FF FF FF FF 15 2
05 3F EF FF FF FF FF FF FF 11 2
05 C0 00 00 00 00 00 00 00 10 2
05 3F DF FF FF FF FF FF FF 8 2
05 7F FF FF FF FF FF FF FF 7 2
05 3F FF FF FF FF FF FF FF 4 2
05 7F FF FF FF FF FF FF FF 3 2
05 7F FF FF FF FF FF FF FF 2 2
05 80 00 00 00 00 00 00 00 2 2
05 80 00 00 00 00 00 00 00 1 2
05 7F FF FF FF FF FF FF FF 1 2
05 80 00 00 00 00 00 00 00 1 2
===== =======
===== =======
insert: 05 C0 10 00 00 00 00 00 00 %!d(float64=13) 1
05 3F E7 FF FF FF FF FF FF 15 1
05 C0 10 00 00 00 00 00 00 13 1
05 3F EF FF FF FF FF FF FF 11 1
05 C0 00 00 00 00 00 00 00 10 1
05 3F DF FF FF FF FF FF FF 8 1
05 7F FF FF FF FF FF FF FF 7 1
05 3F FF FF FF FF FF FF FF 4 1
05 7F FF FF FF FF FF FF FF 3 1
05 7F FF FF FF FF FF FF FF 2 1
05 80 00 00 00 00 00 00 00 2 1
05 80 00 00 00 00 00 00 00 1 1
05 7F FF FF FF FF FF FF FF 1 1
05 80 00 00 00 00 00 00 00 1 1
===== =======
===== =======
insert: 05 C0 18 00 00 00 00 00 00 %!d(float64=8) 4
05 3F E7 FF FF FF FF FF FF 15 4
05 C0 10 00 00 00 00 00 00 13 4
05 3F EF FF FF FF FF FF FF 11 4
05 C0 00 00 00 00 00 00 00 10 4
05 C0 18 00 00 00 00 00 00 8 4
05 3F DF FF FF FF FF FF FF 8 4
05 7F FF FF FF FF FF FF FF 7 4
05 3F FF FF FF FF FF FF FF 4 4
05 7F FF FF FF FF FF FF FF 3 4
05 7F FF FF FF FF FF FF FF 2 4
05 80 00 00 00 00 00 00 00 2 4
05 80 00 00 00 00 00 00 00 1 4
05 7F FF FF FF FF FF FF FF 1 4
05 80 00 00 00 00 00 00 00 1 4
===== =======

From the logs below, it can be observed that the final topn contains multiple identical values. Additionally, the inserted data also has duplicates. Finally, in the sorted results, we found that the order of 0 and -0 is incorrect.

the root cause is that sortSampleItems is sorted by the semantics, but getComparedBytes is compared by the binary order.

The two semantics are different, which led to the final issue.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

fix get wrong topn with data -0 and 0

修复因为 0 和 -0 ，导致构建统计信息 topn 构建错误

codecov · 2024-11-18T10:37:55Z

Codecov Report

Attention: Patch coverage is 77.50000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 73.6385%. Comparing base (1b6d01b) to head (85f2113).
Report is 8 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57464        +/-   ##
================================================
+ Coverage   72.8394%   73.6385%   +0.7991%     
================================================
  Files          1672       1702        +30     
  Lines        462873     470888      +8015     
================================================
+ Hits         337154     346755      +9601     
+ Misses       104938     102721      -2217     
- Partials      20781      21412       +631

Flag	Coverage Δ
integration	`46.1452% <57.5000%> (?)`
unit	`72.2282% <77.5000%> (+0.0095%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7673% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`45.0999% <ø> (-0.0092%)`	⬇️

---- 🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

hawkingrei · 2024-11-19T09:08:32Z

/retest

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

hawkingrei · 2024-11-19T10:01:52Z

/retest

hawkingrei · 2024-11-19T10:59:57Z

/retest

tiprow · 2024-11-19T11:11:38Z

@hawkingrei: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`85f2113`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

AilinKid

rest lgtm

AilinKid · 2024-11-19T13:42:24Z

pkg/statistics/sample.go

@@ -61,15 +62,36 @@ func CopySampleItems(items []*SampleItem) []*SampleItem {
 	return n
 }

-func sortSampleItems(sc *stmtctx.StatementContext, items []*SampleItem) error {
+func sortDatumByBinary(items []types.Datum, getComparedBytes func(datum types.Datum) ([]byte, error)) error {


could you comment on why binary format matters rather than simple item value comparison? for later reviewer's quick grabbing

ti-chi-bot · 2024-11-19T14:02:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/planner/OWNERS~~ [winoros]
~~pkg/statistics/OWNERS~~ [winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-11-19T14:02:24Z

[LGTM Timeline notifier]

Timeline:

2024-11-19 14:02:24.014586044 +0000 UTC m=+969706.205455041: ☑️ agreed by winoros.

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Nov 18, 2024

hawkingrei removed the do-not-merge/needs-triage-completed label Nov 18, 2024

ti-chi-bot bot added do-not-merge/needs-triage-completed and removed do-not-merge/needs-triage-completed labels Nov 18, 2024

hawkingrei marked this pull request as draft November 19, 2024 03:10

ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 19, 2024

hawkingrei marked this pull request as ready for review November 19, 2024 07:55

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 19, 2024

hawkingrei added 9 commits November 19, 2024 17:35

statistics: fix get wrong topn with data -0 and 0

d0cbe9d

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

fix

11eda53

fix

2dbca9b

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

fix

75a39f5

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

fix

1cb8885

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

fix

20522ab

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

fix

cb705dc

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

fix

68975de

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

fix

03a00ec

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

fix

85f2113

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

hawkingrei force-pushed the 57448 branch from 9dcde54 to 85f2113 Compare November 19, 2024 09:35

AilinKid reviewed Nov 19, 2024

View reviewed changes

winoros approved these changes Nov 19, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: fix get wrong topn with data -0 and 0 #57464

statistics: fix get wrong topn with data -0 and 0 #57464

hawkingrei commented Nov 18, 2024 •

edited

Loading

codecov bot commented Nov 18, 2024 •

edited

Loading

hawkingrei commented Nov 19, 2024

hawkingrei commented Nov 19, 2024

hawkingrei commented Nov 19, 2024

tiprow bot commented Nov 19, 2024

AilinKid left a comment

AilinKid Nov 19, 2024

ti-chi-bot bot commented Nov 19, 2024

ti-chi-bot bot commented Nov 19, 2024

statistics: fix get wrong topn with data -0 and 0 #57464

Are you sure you want to change the base?

statistics: fix get wrong topn with data -0 and 0 #57464

Conversation

hawkingrei commented Nov 18, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

codecov bot commented Nov 18, 2024 • edited Loading

Codecov Report

hawkingrei commented Nov 19, 2024

hawkingrei commented Nov 19, 2024

hawkingrei commented Nov 19, 2024

tiprow bot commented Nov 19, 2024

AilinKid left a comment

Choose a reason for hiding this comment

AilinKid Nov 19, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Nov 19, 2024

ti-chi-bot bot commented Nov 19, 2024

[LGTM Timeline notifier]

hawkingrei commented Nov 18, 2024 •

edited

Loading

codecov bot commented Nov 18, 2024 •

edited

Loading