Dual channel replication

[naglera]

In this PR we introduce the main benefit of dual channel replication by continuously steaming the COB (client output buffers) in parallel to the RDB and thus keeping the primary's side COB small AND accelerating the overall sync process. By streaming the replication data to the replica during the full sync, we reduce

1. Memory load from the primary's node.
2. CPU load from the primary's main process. Latest performance tests

Motivation

• Reduce primary memory load. We do that by moving the COB tracking to the replica side. This also decrease the chance for COB overruns. Note that primary's input buffer limits at the replica side are less restricted then primary's COB as the replica plays less critical part in the replication group. While increasing the primary’s COB may end up with primary reaching swap and clients suffering, at replica side we’re more at ease with it. Larger COB means better chance to sync successfully.
• Reduce primary main process CPU load. By opening a new, dedicated connection for the RDB transfer, child processes can have direct access to the new connection. Due to TLS connection restrictions, this was not possible using one main connection. We eliminate the need for the child process to use the primary's child-proc -> main-proc pipeline, thus freeing up the main process to process clients queries.

Dual Channel Replication high level interface design

• Dual channel replication begins when the replica sends a REPLCONF CAPA DUALCHANNEL to the primary during initial
  handshake. This is used to state that the replica is capable of dual channel sync and that this is the replica's main channel, which is not used for snapshot transfer.
• When replica lacks sufficient data for PSYNC, the primary will send -FULLSYNCNEEDED response instead
  of RDB data. As a next step, the replica creates a new connection (rdb-channel) and configures it against
  the primary with the appropriate capabilities and requirements. The replica then requests a sync
  using the RDB channel.
• Prior to forking, the primary sends the replica the snapshot's end repl-offset, and attaches the replica
  to the replication backlog to keep repl data until the replica requests psync. The replica uses the main
  channel to request a PSYNC starting at the snapshot end offset.
• The primary main threads sends incremental changes via the main channel, while the bgsave process
  sends the RDB directly to the replica via the rdb-channel. As for the replica, the incremental
  changes are stored on a local buffer, while the RDB is loaded into memory.
• Once the replica completes loading the rdb, it drops the rdb-connection and streams the accumulated incremental
  changes into memory. Repl steady state continues normally.

New replica state machine

Data

Explanation

These graphs demonstrate performance improvements during full sync sessions using rdb-channel + streaming rdb directly from the background process to the replica.

First graph- with at most 50 clients and light weight commands, we saw 5%-7.5% improvement in write latency during sync session.
Two graphs below- full sync was tested during heavy read commands from the primary (such as sdiff, sunion on large sets). In that case, the child process writes to the replica without sharing CPU with the loaded main process. As a result, this not only improves client response time, but may also shorten sync time by about 50%. The shorter sync time results in less memory being used to store replication diffs (>60% in some of the tested cases).

Test setup

Both primary and replica in the performance tests ran on the same machine. RDB size in all tests is 3.7gb. I generated write load using valkey-benchmark ./valkey-benchmark -r 100000 -n 6000000 lpush my_list __rand_int__.

[zuiderkwast]

I like the HLD. I didn't review in dept yet.

I don't like that we use "This is the main connection" to implicitly mean "I'm capable of receiving the RDB dump on another connection".

Here's my suggestion:

• The replica can use REPLCONF CAPA to tell the primary.
• Check the reply +FULLRESYNC vs -FULLSYNCNEEDED to find out if the primary supports this or not.

[zuiderkwast]

Great! I'll take a closer look another day. You'll need to fix some merge conflicts (replace all master/slave with primary/replica in source code and comments) and fix the DCO (the CI job log says use git rebase HEAD~16 --signoff, then force-push) and probably run clang-format.

[PingXie]

Thanks @naglera! I am directionally aligned with the idea and the high level implementation but I have only scratched the surface of this PR. The core team has marked this for Valkey 8.0. If you are committed as well, can you please 1) resolve the merge conflicts; 2) fix the primary/replica terminology; 3) clang-format the changes, when you get a chance? I will continue my review afterwards. Thanks again for the contribution!

[PingXie]

So this along with the previous change on L4017 is essentially saying CLIENT_PROTECTED_RDB_CHANNEL can't protect a client from being evicted?

[naglera]

TLDR: The CLIENT_PROTECTED_RDB_CHANNEL flag's primary role is to preserve the shared replication buffer data blocks, not to protect the client from eviction.

Expanded explanation:

The CLIENT_PROTECTED_RDB_CHANNEL flag is used in a specific replication process between a primary and a replica valkey instance. This process involves:

1. The replica requests a full synchronization using the RDB channel.

2. The primary approves this request and sends:

   • The RDB_END_OFFSET (indicating where the RDB transfer ends)
   • The actual RDB data

3. The replica completes loading the RDB data via the RDB connection. In the meanwhile, the replica's main connection hasn't yet established a PSYNC with the primary.

Typically, after this step, the primary would free its shared replication buffer at the RDB_END_OFFSET. However, the CLIENT_PROTECTED_RDB_CHANNEL flag prevents this from happening. It's used to stop the primary from releasing the replication data blocks that the replica's RDB client was using, even though that RDB client is no longer active at this stage. This protection ensures that necessary replication data remains available for the subsequent synchronization steps.

[enjoy-binbin]

thanks for the PR. I roughly read the code locally, high Level LGTM.

The code mixes connection and channel at some points. I suggest we stick with just one name.
In addition, some comments are outdated. I didn't read them carefully and marked them.
I think there's a lot of new replconf / status interaction, is there any way to reduce it? (I haven't reviewed all of them, they may be necessary)

thanks again, please resolve the conflict (i know it's painful in a big PR...)

[zuiderkwast]

LGTM, though I haven't done a complete review.

[naglera]

Fixed test failure logs:

*** [err]: Psync established after RDB load - beyond grace period in tests/integration/dual-channel-replication.tcl
log message of '"*Replica main channel failed to establish PSYNC within the grace period*"' not found in ./tests/tmp/server.7063.182/stdout after line: 0 till line: 196

PR: #804