design-proposal: Adjust virtiofs feature gates

This design document explains how and why virtiofs feature gates should
be adjusted. The main purpose is to avoid mixing feature control and
configuration with feature enablement.

Signed-off-by: Javier Cano Cano <jcanocan@redhat.com>

design-proposals/virtiofs-support/virtiofs-feature-gates.md

@@ -0,0 +1,60 @@
+# Overview
+
+Virtiofs allows sharing Secrets, ConfigMaps, ServiceAccounts and DownwardAPI (a.k.a., ConfigVolumes) in a dynamic way,
+i.e., any change in those objects are reflected in the VirtualMachine (VM) without restarting the VM. Moreover, PVCs may
+also be shared using virtiofs allowing multiple readers and writers in the shared PVCs.
+
+## Motivation
+
+Currently, the usage of feature gates are mixed with the feature control. The feature gate `ExperimentalVirtiofsSupport`
+controls whether virtiofsd runs as root or not. Moreover, there is no way to disable it completely and this feature is
+not in GA.
+
+In order to GA virtiofs, VMs should be able to live migrate while sharing data with virtiofs. The virtiofsd’s live
+migration support will distinguish between different use cases:  sharing ConfigMaps, Secrets, DownwardAPI and
+ServiceAccounts and sharing PVCs. This distinction will be useful for a staggered and granular support.
+
+## Goals
+- Drop the usage of `ExperimentalVirtiofsSupport` feature gate, marking it as deprecated.
+- Add feature gate `EnableVirtioFsConfigVolumes`, which will enable the sharing of ConfigMaps, Secrets, DownwardAPI and ServiceAccount.
+- Add feature gate `EnableVirtioFsPVC`, which will enable the sharing of PVCs.
+
+## Non Goals
+- Support live migration while sharing data with virtiofs.
+## Definition of Users
+A VM owner with permissions to read ConfigVolumes and/or read and/or write PVCs.
+
+## User Stories
+As a VM owner, I want to share Secrets, ConfigMaps, ServiceAccounts and/or DownwardAPI in a dynamic way.
+
+As a VM owner, I want to share PVCs with one or multiple VMs and be able to read and/or write them.
+
+## Repos
+Kubevirt/kubevirt
+
+# Design
+## KubeVirt Feature Gates
+Two feature gates will be introduced as a means to granular support virtiofs functionalities:
+- `EnableVirtioFsConfigVolumes` to allow sharing ConfigMaps, Secrets, DownwardAPI and ServiceAccount.
+- `EnableVirtioFsPVC` to allow sharing PVCs.
+The feature gates may start its path to GA once a VM will be able to live migrate while sharing data with virtiofs.
+
+## VM API
+From the VM API point of view, everything should keep as it is.
+
+## Update/Rollback Compatibility
+This is a breaking change, users using virtiofs in any way won’t be able to further use virtiofs without enabling the proper feature gate/s.
+
+## Functional Testing Approach
+Current functional tests will be adjusted to those new feature gates. 
+
+# Implementation Phases
+- Drop usage of `ExperimentalVirtiofsSupport` 
+- Add the feature gate `EnableVirtioFsConfigVolumes`.
+- Add the feature gate and `EnableVirtioFsPVC`.
+
+# Open Questions
+
+- Currently, it is possible to run rootful virtiofs containers.
+   - Should we support this use case at all?, since the possibility of running a rootful container will be dropped.
+   - Should we drop this functionality entirely?