Support Feature Gates

[ahrtr]

What would you like to be added?

Currently we add any new feature as an experimental feature with a configuration flag prefixed with "experimental", e.g. --experimental-feature-name. When we decide to graduate the experimental feature, we remove the prefix experimental. Refer to contributor-guide/features.md. Obviously it's a breaking change when we rename any configuration flag, so it isn't good.

So propose to add feature gates in the similar way as Kubernetes does. We can turn on or off any feature using the --feature-gates flag.

Reference:

• https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

I expect any new etcd features should follow the feature gates process once the feature gates feature is ready,

• added as an alpha feature
• disabled by default in the beginning

cc @fuweid @jmhbnz @serathius @siyuanfoundation @neolit123 @logicalhan @ArkaSaha30

Why is this needed?

Support a better & non-breaking way to manage new features

[serathius]

+1 from me

[neolit123]

Obviously it's a breaking change when we rename any configuration flag, so it isn't good.

i'm a proponent to not prefix experimental features (flags, k8s labels, etc) with "experimental" because it's a breaking change, yes.
a flag can be just --foo instead of --experimental-foo and when used it can throw a warning and also be documented in its description as "experimental". if the user does not read the description, i would say it's their fault.

in terms of using feature gates in etcd, i don't see an issue. it's a smoother op-in for users when a beta FG enables a feature by default and has the opt-out.

[logicalhan]

+100 from me as well! Really happy to see this initiative being driven forward!

(GitHub is doing weird things in their mobile UI)

[tjungblu]

Sounds good, what's the plan for the current experimental features?

[serathius]

Sounds good, what's the plan for the current experimental features?

Expect we alias them to prevent breaking changes at least for 1 release, have both --experimental-feature and --feature flip the same state +/- some integration to work the same way after introduction of --feature-gates.

[ahrtr]

Expect we alias them to prevent breaking changes at least for 1 release, have both --experimental-feature and --feature flip the same state

Agreed with this.

If we decide to graduate or deprecate an experimental feature, we should keep the experimental flag --experimental-feature (and co-exist with the flag --feature for the graduation case) for at least one major/minor release , and print warning if users still use it.

some integration to work the same way after introduction of --feature-gates.

No strong opinion on this.

I expect the --feature-gates only applies to new features being added after introducing --feature-gates, and we still follow the existing contributor-guide/features.md to handle all legacy experimental flags. Forcing users to use --feature-gates for legacy experimental flags is also another breaking change?

[siyuanfoundation]

+1 for the feature gate.
For existing experimental features, we can translate the flag to feature gate under the hood, and deprecate them after the next minor release

[fuweid]

+1 for this.

Is it good timing to migrate and deprecate all the existing experimental features in v3.6? 👀

[ahrtr]

Is it good timing to migrate and deprecate all the existing experimental features in v3.6?

Let's discuss it under #16292.

[siyuanfoundation]

I can take on this issue, and start a KEP

cc @stackbaek