Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize TypedSet and map_concat, array_union #15362

Merged
merged 5 commits into from
Nov 30, 2020
Merged

Optimize TypedSet and map_concat, array_union #15362

merged 5 commits into from
Nov 30, 2020

Commits on Nov 20, 2020

  1. Introducing OptimizedTypedSet

    Ying Su committed Nov 20, 2020
    Configuration menu
    Copy the full SHA
    bb9f3f2 View commit details
    Browse the repository at this point in the history

  2. Optimize map_concat using OptimizedTypedSet 

    JMH benchmark shows 1.82x improvement for non_empty case when keyCount=100
and POSITIONS=1000:

Baseline
Benchmark                     Mode  Cnt      Score      Error  Units
BenchmarkMapConcat.mapConcat  avgt   20  26710.925 ± 2005.756  ns/op
Retained Size: 1,402,374 bytes

After
Benchmark                     Mode  Cnt      Score      Error  Units
BenchmarkMapConcat.mapConcat  avgt   20  14605.437 ± 1209.786  ns/op
Retained Size: 1,373,273 bytes

When keyCount=1000 and POSITIONS=1000, the baseline just OOMed. The
optimized version succeeded.

Add different sizes to BenchmarkMapConcat
    Ying Su committed Nov 20, 2020
    Configuration menu
    Copy the full SHA
    9fa35e6 View commit details
    Browse the repository at this point in the history

  3. Optimize array_union using OptimizedTypedSet 

    JMH benchmark shows up to 40% improvement:

Type       | Baseline  | Specialized Baseline | OptimizedTypedSet | Gain%
-----------|-----------|----------------------|-------------------|----------
BIGINT     |      5511 |                 3742 |            3320   |      40%
VARCHAR    |     20414 |                 N/A  |            14155  |      31%
    Ying Su committed Nov 20, 2020
    Configuration menu
    Copy the full SHA
    5efff89 View commit details
    Browse the repository at this point in the history

  4. Optimize array_except by using OptimizedTypedSet 

    JMH benchmark shows 40% improvement:

Before
Benchmark                               Mode  Cnt       Score        Error  Units
BenchmarkArrayIntersect.arrayIntersect  avgt   10  618074.452 ± 119912.203  ns/op

After
Benchmark                               Mode  Cnt       Score       Error  Units
BenchmarkArrayIntersect.arrayIntersect  avgt   10  376854.064 ± 21616.063  ns/op
    Ying Su committed Nov 20, 2020
    Configuration menu
    Copy the full SHA
    766de45 View commit details
    Browse the repository at this point in the history

  5. Optimize array_intersect by using OptimizedTypedSet 

    JMH benchmark shows 35% improvement:

Before
Benchmark                               Mode  Cnt       Score       Error  Units
BenchmarkArrayIntersect.arrayIntersect  avgt   10  540349.423 ± 66298.751  ns/op

After
Benchmark                               Mode  Cnt       Score       Error  Units
BenchmarkArrayIntersect.arrayIntersect  avgt   10  350934.564 ± 34092.598  ns/op
    Ying Su committed Nov 20, 2020
    Configuration menu
    Copy the full SHA
    87d7f18 View commit details
    Browse the repository at this point in the history