optimization: Unified Sorter (#972)

[ti-srebot]

cherry-pick #972 to release-4.0

---

What problem does this PR solve?

#945

• Improves serialization performance.
• Supports concurrent sorting.
• Enables automatic fallback to disk when memory is not enough for sorting.

What is changed and how it works?

• Implemented UnifiedSorter.
• Made UnifiedSorter the default sorter.
• Moved mounting after sorting, to ease serialization to temp files.

Check List

Tests

• Unit test
• Integration test

Code changes

• Has exported function/method change

Side effects

• Possible performance regression
• Increased code complexity

Benchmarks

Method

We generate mock KV-entries with 1KB data in each entry. We use 256 goroutines to generate data in parallel. There is no resolved event until all data has been fed to the sorter.

Results

Data	Time
50GB	7 min
100GB	16 min
200GB	32 min

Machine specifications

CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Memory: 16GB DDR4 2133MHz * 8
Disk: Intel Corporation NVMe Datacenter SSD

Sorter configuration

{
	NumConcurrentWorker:  16,
	ChunkSizeLimit:       1 * 1024 * 1024 * 1024,   // 1GB
	MaxMemoryPressure:    60,
	MaxMemoryConsumption: 16 * 1024 * 1024 * 1024,  // 16GB
}

Release note

• No release note

[ti-srebot]

/run-all-tests

[liuzix]

/run-all-tests

[liuzix]

/run-all-tests