Optimization performance of file sort

[amyangfei]

Feature Request

Is your feature request related to a problem? Please describe:

In current file sort implement, there exits too much cpu cost in PolymorphicEvent encode and decode, besides this codec is single thread, so the performance is poor.

Following is a benchmark when running sysbench in upstream.

Besides as the slow of file flush, during incremental scan stage, the puller could accumulate too much data and leads to OOM, so the file sort doesn't support large incremental scan data size now.

Describe the feature you'd like:

Optimize the performance of file sort, by enabling multiple thread codec or change the architecture, such as cache key in memory and store data in a KV storage, which can take advantage of the sorting mechanism provided by KV storage. The optimized file sort should

• Support per table incremental scan data size up to 20G (we should discuss about the upper size)
• Support a high throughput real time replication

[liuzix]

I have assigned myself and I have the following proposals

• Explore ways to unify memory sorter and the file sorter. We need one implementation for two kind of sorts.
• Decouple the serializer and the comparator from the sorter itself, so that we can easily test various solutions and optimize for performance. It will also make refactoring of the processor easier if the sorter(s) are more configurable.

[amyangfei]

https://github.com/pingcap/ticdc/pull/972