Remove dual-channel-replication Feature Flag's Protection

[naglera]

Currently, the dual-channel-replication feature flag is immutable if enable-protected-configs is enabled, which is the default behavior. This PR proposes to make the dual-channel-replication flag mutable, allowing it to be changed dynamically without restarting the cluster.

Motivation:
The ability to change the dual-channel-replication flag dynamically is essential for testing and validating the feature on real clusters running in production environments. By making the flag mutable, we can enable or disable the feature without disrupting the cluster's operations, facilitating easier testing and experimentation. Additionally, this change would provide more flexibility for users to enable or disable the feature based on their specific requirements or operational needs without requiring a cluster restart.

[madolson]

This title isn't right, it's removing that it's a protected config, which has wider implications that just making it a mutable config. Making it protected doesn't make sense, was there a reason @PingXie?

[PingXie]

This title isn't right, it's removing that it's a protected config, which has wider implications that just making it a mutable config. Making it protected doesn't make sense, was there a reason @PingXie?

I don't remember the exact context now but I guess "protected" was introduced as a guard rail to prevent "fat-fingering" while keeping the config mutable. In a sense, this setting is kind of "semi-immutable". BTW, I have always been in favor of "immutable" but "mutable" was not a blocker to me either.

I think two more things need to be done as part of this PR:

1. A description of the expected behavior. For example, what would happen if a user flips this config while a dual-channel full sync is in progress?
2. a test suit that validates such a contract is upheld by the server.

[madolson]

I don't remember the exact context now but I guess "protected" was introduced as a guard rail to prevent "fat-fingering" while keeping the config mutable. In a sense, this setting is kind of "semi-immutable". BTW, I have always been in favor of "immutable" but "mutable" was not a blocker to me either.

Ok, protected feels wrong to me, protected was intended for security configurations to make sure nobody can maliciously escalate their permissions. This feels like we aren't sure if something can be changed.

A description of the expected behavior. For example, what would happen if a user flips this config while a dual-channel full sync is in progress?

I don't know what Amit had in mind, but it the sync should continue with full-sync. I don't think flipping these configs should abort the ongoing synchronization.

a test suit that validates such a contract is upheld by the server.

What contract are you referring to?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.44%. Comparing base (131857e) to head (cdebc55).
Report is 63 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable     #908      +/-   ##
============================================
+ Coverage     70.37%   70.44%   +0.07%     
============================================
  Files           112      113       +1     
  Lines         61492    61728     +236     
============================================
+ Hits          43273    43486     +213     
- Misses        18219    18242      +23     

Files with missing lines	Coverage Δ
src/config.c	78.69% <ø> (ø)

... and 31 files with indirect coverage changes

[madolson]

From core discussion, let's document that the sync finishes the replication with dual channel even when the flag is flipped part way through. (Ideally it should finish with the initial settings, but for now the scope is just dual channel).

[naglera]

Added tests to verify that sync continues normally in case of dual-channel config changed.

[zuiderkwast]

Good. I didn't review the test.

[PingXie]

LGTM. Nits only. Thanks @naglera