diff --git a/keps/sig-storage/20191125-bucket-provisioning.md b/keps/sig-storage/20191125-bucket-provisioning.md index 9655bb17c92..ac7a55c7694 100644 --- a/keps/sig-storage/20191125-bucket-provisioning.md +++ b/keps/sig-storage/20191125-bucket-provisioning.md @@ -34,20 +34,24 @@ status: provisional - [User](#user) - [System Configuration](#system-configuration) - [Workflows](#workflows) - - [Create Bucket](#create-bucket) - - [Delete Bucket](#delete-bucket) - - [Grant Bucket Access for Static Brownfield](#grant-bucket-access-for-static-brownfield) - - [Grant Bucket Access for Dynamic Brownfield](#grant-bucket-access-for-dynamic-brownfield) - - [Revoke Bucket Access (needs to handle stratic and dynamic brownfield!)](#revoke-bucket-access-needs-to-handle-stratic-and-dynamic-brownfield) + - [Greenfield Buckets](#greenfield-buckets) + - [Create Bucket](#create-bucket) + - [Delete Bucket](#delete-bucket) + - [Brownfield Buckets](#brownfield-buckets) + - [Grant Access](#grant-access) + - [Revoke Access](#revoke-access) + - [Static Buckets](#static-buckets) + - [Grant Access](#grant-access-1) + - [Revoke Access](#revoke-access-1) - [Custom Resource Definitions](#custom-resource-definitions) - - [ObjectBucketClaim (OBC)](#objectbucketclaim-obc) - - [ObjectBucket (OB)](#objectbucket-ob) + - [Bucket](#bucket) + - [BucketContent](#bucketcontent) - [BucketClass](#bucketclass) # Summary -This proposal introduces the Container Object Storage Interface (COSI), whose purpose is to provide a familiar and standardized method of provisioning object storage buckets in an manner agnostic to the storage vendor. The COSI environment is comprised of Kubernetes CRDs, operators to manage these CRDs, and a gRPC interface by which these operators communicate with vendor drivers. This design is of course heavily inspired by the Kubernetes’ implementation of the Container Storage Interface (CSI). However, bucket management lacks some of the notable requirements of block and file provisioning and allows for an architecture with lower overall complexity. This proposal does not include a standardized protocol or abstraction of storage vendor APIs. +This proposal introduces the Container Object Storage Interface (COSI), whose purpose is to provide a familiar and standardized method of provisioning object storage buckets in an manner agnostic to the storage vendor. The COSI environment is comprised of Kubernetes CRDs, operators to manage these CRDs, and a gRPC interface by which these operators communicate with vendor drivers. This design is heavily inspired by the Kubernetes’ implementation of the Container Storage Interface (CSI). However, bucket management lacks some of the notable requirements of block and file provisioning and allows for an architecture with lower overall complexity. This proposal does not include a standardized protocol or abstraction of storage vendor APIs. ## Motivation @@ -64,21 +68,23 @@ File and block are first class citizens within the Kubernetes ecosystem. Object ## Non-Goals + Define a native _data-plane_ object store API which would greatly improve object store app portability. - -## Vocabulary -+ _Brownfield_ - also called "dynamic brownfield" - buckets are created outside the COSI system but granted access via COSI. The OB, driver secret, app secret and binding are all performed by COSI. -+ _Bucket_ - A namespace where objects reside, similar to a flat POSIX directory. -+ _BucketClass_ (BC) - A cluster-wide (non-namespaced) custom resource containing fields defining the provisioner and an immutable parameter set. Referenced by ObjectBucketClaims and populated into the OB. Only used for provisioning new buckets. -+ _Container Object Storage Interface (COSI)_ - The specification of gRPC data and methods making up the communication protocol between the driver and the sidecar. -+ _COSI Controller_ - A single, central controller which manages OBCs, OBs, and Secrets cluster-wide. Often referred to below as the "central controller". -+ _"cosi-system"_ - The namespace name for all COSI drivers and sidecars. Even if a cluster supports different, concurrent object stores, all drivers live in this namespace. -+ _Driver - A containerized gRPC server which implements a storage vendor’s business logic through the COSI interface. It can be written in any language supported by gRPC and is independent of Kubernetes. There is historical precedence in Kubernetes to call this type of container a _plugin_, but going forward _driver_ is the preferred name. -+ _Greenfield_ - new buckets dynamically created by the driver. -+ _OB_ (Object Bucket) - A cluster-wide (non-namespaced) custom resource representing the provisioned bucket and relevant metadata. -+ _OBC_ (Object Bucket Claim) - A user-namespaced custom resource representing a user’s bucket request. -+ _Object_ - An atomic, immutable unit of data stored in buckets. -+ _Sidecar_ - A controller deployed in the same pod as the driver, responsible for managing OBs and communicating with the Driver via gRPC. Needs write access to its OB. Note: the sidecar controller can use [predicates](https://godoc.org/sigs.k8s.io/controller-runtime/pkg/predicate) to filter OBs by driver name (assuming a unique label is added to OBs). -+ _Static_ - also called "static brownfield" - buckets are created outside the COSI system but granted access via COSI without the need for the driver/sidecar. The OB and driver secret are created manually in the expected locations. The app secret and binding are performed by COSI. ++ Mirror the static workflow of PersistentVolumes wherein users are given the first available Volume. Pre-provisioned buckets are expected to be non-empty and thus unique. + +## Vocabulary + ++ _Brownfield Bucket_ - externally created, represented by a BuckteClass and managed by a provisioner ++ _Bucket_ - A user-namespaced custom resource representing an object store bucket. ++ _BucketClass_ - A cluster-scoped custom resource containing fields defining the provisioner and an immutable parameter set. + + _In Greenfield_: an abstraction of new bucket provisioning. + + _In Brownfield_: an abstration of an existing objet store bucket. ++ _BucketContent_ - A cluster-scoped custom resource, bound to a Bucket and containing relevant metadata. ++ _Container Object Storage Interface (COSI)_ - A specification of gRPC data and methods making up the communication protocol between the driver and the sidecar. ++ _COSI Controller_ - A central controller responsible for mananing Buckets, BucketContents, and Secrets cluster-wide. ++ _Driver_ - A containerized gRPC server which implements a storage vendor’s business logic through the COSI interface. It can be written in any language supported by gRPC and is independent of Kubernetes. ++ _Greenfield Bucket_ - created and managed by the COSI system. ++ _Object_ - An atomic, immutable unit of data stored in buckets. ++ _Sidecar_ - A BucketContent controller that communicates to the driver via a gRPC client. ++ _Static Bucket_ - externally created and manually integrated but **lacking** a provisioner # Proposal @@ -99,182 +105,181 @@ File and block are first class citizens within the Kubernetes ecosystem. Object ## System Configuration -+ The central controller runs in a protected namespace with RBAC privileges for managing OBCs, OBs, and Secrets cluster wide. - + Responsible for the binding relationship of OBs and OBCs. -+ The "cosi-system" namespace must be created. - + This namespace must be granted write access to cluster-wide scoped OBs. -+ The Driver and Sidecar containers run together in a single Pod in the "cosi-system" namespace. - + The gRPC connection is made through the Pod’s localhost. -+ BucketClasses must exist (cluster-wide scope). -+ No node affinity or other requirements exist. ++ The COSI controller runs in the `cosi-system` namespace where it manages Buckets, BucketContents, and Secrets cluster-wide. ++ The Driver and Sidecar containers run together in a Pod and are deployed in the `cosi-system` namespace, communicating via the Pod's internal network. + Operations must be idempotent in order to handle failure recovery. - ## Workflows -### Create Bucket -1. The user creates an OBC in the app’s namespace. -1. The COSI central controller detects the OBC and creates an OB (cluster-wide), containing driver-relevant data collected from the OBC and BC. -1. The Sidecar detects the OB and issues a CreateBucket() rpc, passing to the driver config data. -1. The Driver creates the bucket and returns pertinent endpoint, credential, and metadata. -1. The Sidecar creates a secret in its namespace ("cosi-system") containing essential connection information. -1. The central controller copies the secret to the OBC’s namespace and the app is ready to run. - -### Delete Bucket -1. The user deletes the OBC, which blocks until the finalizer is removed. -1. The central controller deletes the OB, which also blocks until its finalizer is removed. -1. The Sidecar detects this and issues a DeleteBucket() rpc to the Driver. It passes data stored in the OB. -1. The Driver deletes the bucket and responds with no errors. -1. The Sidecar deletes the OB’s secret. -1. The central controller deletes the secret’s copy in the app’s namespace. -1. The central controller removes the finalizers from the OBC, OB, and Secret allowing them to be deleted. - -### Grant Bucket Access for Static Brownfield -Reminder: BucketClass is ignored since it is used only for dynamic provisioning. - -1. An OB is manually created with enough information to identify an existing bucket in the object store. -1. A secret granting bucket-create privilege is manually created in the Driver’s namespace. -1. The user creates an OBC in the app’s namespace which references the pre-existing OB and driver secret. -1. The central controller detects the OBC and notices its `objectBucketRef` and `secretRef` are filled in. -1. The central controller clones the secret from the driver's namespace to the app's namespace. -1. The central controller binds the OBC to the OB and the app is ready to run. - -### Grant Bucket Access for Dynamic Brownfield -Reminder: BucketClass is ignored since it is used only for dynamic provisioning. - -1. An OB is manually created with enough information to identify an existing bucket in the object store. -1. The user creates an OBC in the app’s namespace which references the pre-existing OB. -1. The central controller detects the OBC and notices its `objectBucketRef` is filled in but its `secretRef` is empty. -1. The central controller updates OB Phase and OBC reference. -1. The Sidecar detects the OB update and calls the `Grant()` gRPC method. -1. The Driver grants access to the bucket and returns pertinent endpoint, credential, and metadata. -1. The Sidecar creates a secret in its namespace containing essential connection information. -1. The central controller clones the secret to the OBC’s namespace, binds the OB, and the app is ready to run. - -### Revoke Bucket Access (needs to handle stratic and dynamic brownfield!) - -Reminder: BucketClass is ignored in browfield operations. - -1. The user deletes the OBC, which blocks until the finalizer is removed. -1. The central controller detects the OBC change and updates OB’s Phase. -1. The Sidecar detects the OB update and calls the `Revoke()` gRPC method. -1. The Driver revokes access to the bucket and responds with no errors. -1. The Sidecar deletes the OB’s secret. -1. The central controller deletes the secret’s copy in the app’s namespace. -1. The central controller removes the finalizers from the OBC, OB, and Secret. -1. The OBC and its secret are garbage collected and the OB remains. +### Greenfield Buckets + +#### Create Bucket + +1. The user creates Bucket in the app’s namespace. +1. The COSI Controller detects the Bucket and creates a BucketContent object containing driver-relevant data. +1. The Sidecar detects the BucketContent and issues a CreateBucket() rpc with any parameters defined in the BucketClass. +1. The driver provisions the bucket and returns pertinent endpoint, credential, and metadata information. +1. The sidecar creates a Secret in it's namespace (`cosi-system`) containing endpoint and credential information and an owener reference to the BucketContent. +1. The COSI controller generates a Secret in the Bucket's namespace containing the data of it's parent Secret, with an owner reference to the Bucket. +1. The workflow ingests the Secret to begin operation. + +#### Delete Bucket + +1. The user deletes their Bucket, which blocks until backend deletion operations complete. +1. The COSI controller detects the update and deletes the BucketContent, which also blocks until backend deletion operations complete. +1. The sidecar detects this and issues a DeleteBucket() rpc to the Driver. It passes pertitent data stored in the BucketContent. +1. The driver deletes the bucket from the object store and responds with no error. +1. The sidecar removes the BucketContent's finalizer. The BucketContent and parent Secret are garbage collected. +1. The COSI controller removes the finalizer on the Bucket. The Bucket and the child Secret are garbage collected. + +### Brownfield Buckets + +#### Grant Access + +1. A BucketClass is defined specifically referencing an object store bucket. +1. A user creates a Bucket in the app namespace, specifying the BucketClass. +1. The COSI controller detects the Bucket and creates a BucketContent object in the `cosi-system` namespace. +1. The sidecar detects the BucketContent object and calls the GrantAccess() rpc to the driver, returing a set of credentials for accessing the bucket. +1. The sidecar writes the credentials and endpoint information to a Secret in the `cosi-system` namespace, with an owner reference to the BucketContent. +1. The COSI controller generates a Secret in the Bucket's namespace containing the parent Secret's data, with an owner reference to the Bucket. +1. The workflow ingests the Secret to begin operation. + +#### Revoke Access + +1. The user deletes the Bucket, which blocks until revoke operations complete. +2. The COSI controller detects the update and deletes the BucketContent, which also blocks until backend revoke operations complete. +3. The sidecar detects the BucketContent update and invokes the RevokeAccess() rpc method. +4. The driver terminates the access for the associated credentials. +5. The sidecar removes the finalizer from the BucketContent, allowing it and the parent Secret to be garbage collected. +6. The COSI controller removes the finalizer from the Bucket, allowing it and the child Secret to be garbage collected. + +### Static Buckets + +> Note: No driver is present to manage provisioning. The COSI controller only automates Bucket/BucketContent binding operations. + +#### Grant Access + +1. A BucketClass is defined specifically referencing an object store bucket and a Secret in the `cosi-system` namespace containing access credentials. +1. A user creates a Bucket in the app namespace, specifying the BucketClass. +1. The COSI controller detects the object store bucket and Secret in the BucketClass and creates a BucketContent in the `cosi-system` namespace. +1. The COSI controller generates a Secret in the Bucket's namespace containing the parent Secret's data, with an owner reference to the Bucket. +1. The workflow ingests the Secret to begin operation. + +#### Revoke Access + +1. The user deletes the Bucket, which blocks until revoke operations complete. +1. The COSI controller deletes the Bucket's bound BucketContent object. +1. The COSI controller removes the finalizer from the Bucket, allowing it and the child Secret to be garbage collected. ## Custom Resource Definitions -#### ObjectBucketClaim (OBC) +#### Bucket -A user facing API object representing a request for a bucket, or access to an existing bucket. OBCs are created by users in their namespaces. Once the request is fulfilled, the OBC is “bound” to an Object Bucket (OB). The binding is used to mark the request as fulfilled and prevent further binds to the OB. +A user facing API object representing an object store bucket. Created by a user in their app's namespace. Once provisiong is complete, the Bucket is "bound" to the corresponding BucketContent. The is used to prevent further binds to the BucketContent. ```yaml apiVersion: cosi.io/v1alpha1 -kind: ObjectBucketClaim +kind: Bucket metadata: - name: [1] + name: namespace: labels: - “cosi.io/driver”: [2] + cosi.io/provisioner: [1] finalizers: - - cosi.io/finalizer [3] + - cosi.io/finalizer [2] spec: - bucketName: [4] - generateBucketName: [5] - bucketClassName: [6] - objectBucketName: [7] - secretName: [8] - driverSecretName: [9] + bucketName: [3] + generateBucketName: [4] + bucketClassName: [5] + secretName: [6] + protocol: [7] + accessMode: {"ro", "rw"} [8] status: + bucketContentName: [9] phase: - conditions: []ObjectBucketClaimCondition + conditions: ``` -1. `name`: the metadata name of the OBC; however this name can be generated and thus cannot be relied on to predictably name other related resources, eg. secrets. -1. `labels`: central controller adds the label to its managed resources for easy GET ops. Value is the driver name returned by GetDriverInfo() rpc*. -1. `finalizers`: central controller adds the finalizer to defer OBC deletion until backend deletion ops succeed. +1. `labels`: COSI controller adds the label to its managed resources to easy CLI GET ops. Value is the driver name returned by GetDriverInfo() rpc*. +1. `finalizers`: COSI controller adds the finalizer to defer Bucket deletion until backend deletion ops succeed. 1. `bucketName`: Desired name of the bucket to be created**. -1. `generateBucketName`: Prefix to a randomly generated name. Mutually exclusive with `bucketName`**. -1. `bucketClassName`: Name of the target BucketClass**. -1. `objectBucketName`: Name of a bound OB. - - Injected by the central controller during greenfield ops. - - Defined by the OBC author for static and dynamic brownfield ops. -1. `secretName`: Desired name of the app's secret for greenfield. Fails if secret exists. Defined here so that app deployment (where the secret name is required) is independent of OBC creation. -1. `driverSecretName`: Name of the driver's secret with the namespace assumed to be "cosi-system". - - Injected by the central controller during greenfield and dynamic brownfield ops. - - Defined by the OBC author for static brownfield ops. +1. `generateBucketName`: Desired prefix to a randomly generated bucket name. Mutually exclusive with `bucketName`**. +1. `bucketClassName`: Name of the target BucketClass. +1. `secretName`: Desired name for user's credential Secret. Fails on name collisions. Deterministic names allow for a single manifest workflow. +1. `protocol`: String array of protocols (e.g. s3, gcs, swift, etc.) requested by the user. Used in matching Buckets to BucketClasses and ensuring compatibility with backing object stores. +1. `accessMode`: The requested level of access provided to the returned access credentials. +1. `bucketContentName`: Name of a bound BucketContent -\* Characters that do not adhere to [Kubernetes label conventions](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) will be converted to ‘-’. +> \* Characters that do not adhere to [Kubernetes label conventions](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) will be converted to ‘-’. -** Ignored for brownfield usage. +> ** Ignored in Brownfield and Static operations -#### ObjectBucket (OB) -A cluster-wide scoped resource representing the bucket. The OB is expected to store stateful data relevant to bucket deprovisioning. The OB is bound to the OBC in a 1-1 mapping. +#### BucketContent + +A cluster-scoped resource representing an object store bucket. The BucketContent is expected to store stateful data relevant to bucket deprovisioning. The BucketContent is bound to the Bucket in a 1:1 mapping. ```yaml apiVersion: cosi.io/v1alpha1 -kind: ObjectBucket +kind: BucketContent Metadata: name: [1] - namespace: [2] labels: - “cosi.io/driver”: [3] + cosi.io/provisioner: [2] finalizers: - - "cosi.io/finalizer" [4] + - cosi.io/finalizer [3] spec: - bucketClassName: [5] + bucketClassName: [4] + supportedProtocols: [5] releasePolicy: {"Delete", "Retain"} [6] - bucketConfig: map[string]string [7] - objectBucketClaimRef: [8] + bucketRef: [7] name: namespace: - secretName: [9] + secretRef: [8] + accessMode: {"rw", "ro"} [9] status: - bucketMetaData: [10] - phase: {"Bound", "Released", "Failed", “Errored”} [11] - conditions: []ObjectBucketCondition + bucketAttributes: [10] + phase: {"Bound", "Released", "Failed", "Errored"} [11] + conditions: ``` -1. `name`: Generated in the pattern of “obc-”\"-"\ -1. `namespace`: The namespace of the Driver. -1. `labels`: central controller adds the label to its managed resources for easy GET ops. Value is the driver name returned by GetDriverInfo() rpc*. -1. `finalizers`: Set and cleared by the COSI Controller and prevents accidental deletion of an OB. -1. `bucketClassName`: Name of the bucket class -1. `releasePolicy`: release policy from the Bucket Class referenced in the OBC. See `BucketClass` spec for values. -1. `bucketConfig`: a string:string map of driver defined key-value pairs -1. `objectBucketClaimRef`: the name & namespace of the bound OBC. -1. `secretName`: the name of the sidecar-generated secret. Its namespace is assumed "cosi-system". -1. `bucketMetaData`: stateful data relevant to the managing of the bucket but potentially inappropriate user knowledge (e.g. the user’s in-store user name) -1. `phase`: is the current state of the ObjectBucket: - - `Bound`: the operator finished processing the request and linked the OBC and OB - - `Released`: the OBC has been deleted, leaving the OB unclaimed. +1. `name`: Generated in the pattern of `“bucket-”"-"` +1. `labels`: central controller adds the label to its managed resources for easy CLI GET ops. Value is the driver name returned by GetDriverInfo() rpc*. +1. `finalizers`: COSI controller adds the finalizer to defer Bucket deletion until backend deletion ops succeed. +1. `bucketClassName`: Name of the associated BucketClass +1. `supportedProtocols`: String array of protocols (e.g. s3, gcs, swift, etc.) supported by the associated object store. +1. `releasePolicy`: the release policy defined in the associated BucketClass. (see [BucketClass](#BucketClass) for more information) +1. `bucketRef`: the name & namespace of the bound Bucket. +1. `secretRef`: the name of the sidecar-generated secret. It's namespace is assumed `cosi-system`. +1. `accessMode`: The level of access granted to the credentials stored in `secretRef`, one of "read only" or "read/write". +1. `bucketAttributes`: stateful data relevant to the managing of the bucket but potentially inappropriate user knowledge (e.g. user's IAM role name) +1. `phase`: is the current state of the BucketContent: + - `Bound`: the operator finished processing the request and bound the Bucket and BucketContent + - `Released`: the Bucket has been deleted, leaving the BucketContent unclaimed. - `Failed`: error and all retries have been exhausted. - `Retrying`: set when a recoverable driver or kubernetes error is encountered during bucket creation or access granting. Will be retried. - -\* Characters that do not adhere to [Kubernetes label conventions](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) will be converted to ‘-’. +> \* Characters that do not adhere to [Kubernetes label conventions](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) will be converted to ‘-’. #### BucketClass -A cluster-wide scoped custom resource. -During greenfield workflows an OBC references a Bucket Class (BC). The bucket class defines a release policy, and specifies driver specific parameters, such as region, bucket lifecycle policies, etc., as well as the name of the driver as returned by the `GetDriverInfo()` rpc. The driver name is used to filter OBs meant to be handled by the given driver. +A cluster-scoped custom resource. The BucketClass defines a release policy, and specifies driver specific parameters, such as region, bucket lifecycle policies, etc., as well as the name of the driver. The driver name is used to filter BuckeContenst meant to be handled by a given driver. In static bucket workflows, the driver name may be empty if the object bucket is defined. ```yaml apiVersion: cosi.io/v1alpha1 kind: BucketClass metadata: name: - namespace: [1] -driver: [2] # should this be named "provisioner:" ?? -config: string:string [3] +provisioner: [1] +supportedProtocols: [2] +accessMode: {"ro", "rw"} [3] releasePolicy: {"Delete", "Retain"} [4] +bucketContentName: [5] +parameters: string:string [6] ``` -1. `namespace`: BucketClasses are co-namespaced with their associated driver -1. `driver`: Name of the driver, provided via the GetDriverInfo() rpc. Used to filter OBs. -1. `config`: object store specific key-value pairs passed to the driver. -1. `releasePolicy`: Prescribes outcome of an OBC/OB deletion. +1. `provisioner`: Used by sidecars to filter BucketContent objects +1. `supportedProtocols`: A strings array of protocols the associated object store supports (e.g. swift, s3, gcs, etc.) +1. `accessModes`: Declares the level of access given to credentials provisioned through this class. +1. `releasePolicy`: Prescribes outcome of a Deletion and Revoke events. - `Delete`: the bucket and its contents are destroyed - `Retain`: the bucket and its contents are preserved, only the user’s access privileges are terminated - - +- `bucketContentName`: (Optional). An admin defined BucketContent representing a Brownfield or Static bucket. A non-nil value in this field prevents the BucketClass from being used for Greenfield. +- `parameters`: object store specific key-value pairs passed to the driver. \ No newline at end of file