kv: replace memdb with a more memory efficient version

[bobotu]

What problem does this PR solve?

The existing memdb from goleveldb use a contiguous memory to store data. It will put pressure on system memory when enlarging the underlying slice. After supporting large transaction, this problem will be more obvious.

What is changed and how it works?

Implementing a new skiplist with a block-based memory allocator, when enlarging the memdb simply allocate a new block instead of allocating a large amount of memory and copy all of the existing data into the new memory location.

The block size starts at an initial block size then doubled when each new block allocated until it reaches 128M.

Because there are some internal data structure optimization, the performance of Get and Put is slightly improved.

Here is the result of benchmarks in kv package:

opCnt = 100000

master

BenchmarkIsPoint-16                     300000000                4.84 ns/op            0 B/op          0 allocs/op
BenchmarkMemDbBufferSequential-16             20          61710345 ns/op        11979968 B/op          3 allocs/op
BenchmarkMemDbBufferRandom-16                 10         116388720 ns/op        13374355 B/op          7 allocs/op
BenchmarkMemDbIter-16                        300           5030566 ns/op             208 B/op          3 allocs/op
BenchmarkMemDbCreation-16                3000000               426 ns/op            4096 B/op          1 allocs/op

new

BenchmarkIsPoint-16                     300000000                4.81 ns/op            0 B/op          0 allocs/op
BenchmarkMemDbBufferSequential-16             30          60371850 ns/op         8947658 B/op          1 allocs/op
BenchmarkMemDbBufferRandom-16                 20         101129510 ns/op         6710522 B/op          1 allocs/op
BenchmarkMemDbIter-16                       1000           1220734 ns/op             128 B/op          1 allocs/op      
BenchmarkMemDbCreation-16                5000000               394 ns/op            4096 B/op          1 allocs/op  

opCnt = 10000000

master

BenchmarkIsPoint-16                     300000000                4.82 ns/op            0 B/op          0 allocs/op
BenchmarkMemDbBufferSequential-16              1        7888385900 ns/op        3561011328 B/op       96 allocs/op
BenchmarkMemDbBufferRandom-16                  1        46474700000 ns/op       3561046784 B/op       98 allocs/op
BenchmarkMemDbIter-16                          2         504151600 ns/op             208 B/op          3 allocs/op
BenchmarkMemDbCreation-16                3000000               417 ns/op            4096 B/op          1 allocs/op

new

BenchmarkIsPoint-16                     300000000                4.82 ns/op            0 B/op          0 allocs/op
BenchmarkMemDbBufferSequential-16              1        7038679100 ns/op        536864032 B/op        39 allocs/op
BenchmarkMemDbBufferRandom-16                  1        36407376900 ns/op       536864032 B/op        39 allocs/op
BenchmarkMemDbIter-16                         10         122093460 ns/op             128 B/op          1 allocs/op
BenchmarkMemDbCreation-16                5000000               400 ns/op            4096 B/op          1 allocs/op

Check List

Tests

• Unit test

Side effects

• Increased code complexity

[codecov]

Codecov Report

Merging #11807 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #11807   +/-   ##
===========================================
  Coverage   81.2942%   81.2942%           
===========================================
  Files           441        441           
  Lines         94805      94805           
===========================================
  Hits          77071      77071           
  Misses        12257      12257           
  Partials       5477       5477

[jackysp]

/run-all-tests

[coocood]

can we not +1 here?

[bobotu]

blockIdx == 0 && blockOffset == 0 means null.
Maybe we can place a nil pointer at blocks[0] to make blockIdx starts at 1?

[coocood]

LGTM

[ngaut]

/run-all-tests

[tiancaiamao]

LGTM

[ngaut]

/run-all-tests

[sre-bot]

/run-all-tests