Update documentation for Volume Group Snapshot

book/src/SUMMARY.md

@@ -33,6 +33,7 @@
         - [Data Sources](volume-datasources.md)
             - [Cloning](volume-cloning.md)
             - [Volume Snapshot & Restore](snapshot-restore-feature.md)
+            - [Volume Group Snapshot & Restore](group-snapshot.md)
         - [Ephemeral Local Volumes](ephemeral-local-volumes.md)
         - [Volume Limits](volume-limits.md)
         - [Storage Capacity Tracking](storage-capacity-tracking.md)
@@ -42,7 +43,6 @@
         - [CSI Windows](csi-windows.md)
         - [Volume Mode Conversion](prevent-volume-mode-conversion.md)
         - [Cross-Namespace Data Sources](cross-namespace-data-sources.md)
-        - [Volume Group Snapshot](group-snapshot.md)
 - [Deploying a CSI Driver on Kubernetes](deploying.md)
     - [Example](example.md)
 - [Driver Testing](testing-drivers.md)

book/src/external-snapshotter.md

@@ -34,13 +34,17 @@ To use the snapshot beta and GA feature, a snapshot controller is also required.
 
 ### Description
 
-Starting with the Beta version, the snapshot controller will be watching the Kubernetes API server for `VolumeSnapshot` and `VolumeSnapshotContent` CRD objects. The CSI `external-snapshotter` sidecar only watches the Kubernetes API server for `VolumeSnapshotContent` CRD objects. The CSI `external-snapshotter` sidecar is also responsible for calling the CSI RPCs CreateSnapshot, DeleteSnapshot, and ListSnapshots.
+The CSI `external-snapshotter` sidecar watches the Kubernetes API server for `VolumeSnapshotContent` CRD objects. The CSI `external-snapshotter` sidecar is also responsible for calling the CSI RPCs `CreateSnapshot`, `DeleteSnapshot`, and `ListSnapshots`.
 
-#### VolumeSnapshotClass Parameters
+Volume Group Snapshot support can be enabled with the `--enable-volume-group-snapshots` option. When enabled, the CSI `external-snapshotter` sidecar watches the API server for `VolumeGroupSnapshotContent` CRD object, and will be responsible for calling the CSI RPCs `CreateVolumeGroupSnapshot`, `DeleteVolumeGroupSnapshot` and `GetVolumeGroupSnapshot`.
+
+#### VolumeSnapshotClass and VolumeGroupSnapshotClass Parameters
 
 When provisioning a new volume snapshot, the CSI `external-snapshotter` sets the `map<string, string> parameters` field in the CSI `CreateSnapshotRequest` call to the key/values specified in the `VolumeSnapshotClass` it is handling.
 
-The CSI `external-snapshotter` also reserves the parameter keys prefixed with `csi.storage.k8s.io/`. Any `VolumeSnapshotClass` keys prefixed with `csi.storage.k8s.io/` are not passed to the CSI driver as an opaque `parameter`.
+When volume group snapshot support is enabled, the `map<string, string> parameters` field is set in the CSI `CreateVolumeGroupSnapshotRequest` call to the key/values specified in the `VolumeGroupSnapshotClass` it is handling.
+
+The CSI `external-snapshotter` also reserves the parameter keys prefixed with `csi.storage.k8s.io/`. Any `VolumeSnapshotClass` or `VolumeGroupSnapshotClass` keys prefixed with `csi.storage.k8s.io/` are not passed to the CSI driver as an opaque `parameter`.
 
 The following reserved `VolumeSnapshotClass` parameter keys trigger behavior in the CSI `external-snapshotter`:
 
@@ -51,9 +55,9 @@ The following reserved `VolumeSnapshotClass` parameter keys trigger behavior in
 
 For more information on how secrets are handled see [Secrets & Credentials](secrets-and-credentials.md).
 
-#### VolumeSnapshot and VolumeSnapshotContent Parameters
+#### VolumeSnapshot, VolumeSnapshotContent, VolumeGroupSnapshot and VolumeGroupSnapshotContent Parameters
 
-The CSI `external-snapshotter` (v4.0.0+) introduces the `--extra-create-metadata` flag, which automatically sets the following `map<string, string> parameters` in the CSI `CreateSnapshotRequest`:
+The CSI `external-snapshotter` (v4.0.0+) introduces the `--extra-create-metadata` flag, which automatically sets the following `map<string, string> parameters` in the CSI `CreateSnapshotRequest` and `CreateVolumeGroupSnapshotRequest`:
 
 * `csi.storage.k8s.io/volumesnapshot/name`
 * `csi.storage.k8s.io/volumesnapshot/namespace`
@@ -69,6 +73,8 @@ For detailed information about volume snapshot and restore functionality, see [V
 
 CSI drivers that support provisioning volume snapshots and the ability to provision new volumes using those snapshots should use this sidecar container, and advertise the CSI `CREATE_DELETE_SNAPSHOT` controller capability.
 
+CSI drivers that support provisioning volume group snapshots should use this side container too, and advertise the CSI `CREATE_DELETE_GET_VOLUME_GROUP_SNAPSHOT` controller capability.
+
 For detailed information (binary parameters, RBAC rules, etc.), see [https://github.com/kubernetes-csi/external-snapshotter/blob/release-6.2/README.md](https://github.com/kubernetes-csi/external-snapshotter/blob/release-6.2/README.md).
 
 ### Deployment

book/src/group-snapshot.md

@@ -4,7 +4,9 @@
 
 Status | Min K8s Version | Max K8s Version | snapshot-controller Version | snapshot-validation-webhook Version | CSI external-snapshotter sidecar Version | external-provisioner Version
 --|--|--|--|--|--|--
-Alpha  | 1.27 | - | 7.0+ | 7.0+ | 7.0+ | 3.5+
+Alpha  | 1.27 | - | 7.0+ | 7.0+ | 7.0+ | 4.0+
+
+**IMPORTANT**: The validation logic for VolumeGroupSnapshots and VolumeGroupSnapshotContents has been replaced by CEL validation rules. The validating webhook is now only being used for VolumeGroupSnapshotClasses to ensure that there's at most one class per CSI Driver. The validation webhook is deprecated and will be removed in the next release
 
 ## Overview
 
@@ -14,12 +16,16 @@ are taken at the same point-in-time. A group snapshot can be used either to rehy
 new volumes (pre-populated with the snapshot data) or to restore existing volumes to
 a previous state (represented by the snapshots).
 
-Kubernetes CSI currently enables CSI Drivers to expose the following functionality via the Kubernetes API:
+Kubernetes CSI currently enables CSI Drivers to expose the following
+functionality via the Kubernetes API:
 
-1. Creation and deletion of volume group snapshots via [Kubernetes native API](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/apis/volumegroupsnapshot/v1alpha1). 
-2. Creation of new volumes pre-populated with the data from a snapshot that is part of the volume group snapshot via Kubernetes [dynamic volume provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/).
+1. Creation and deletion of volume group snapshots via [Kubernetes native
+   API](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/apis/volumegroupsnapshot/v1alpha1).
+2. Creation of new volumes pre-populated with the data from a snapshot that is
+   part of the volume group snapshot via Kubernetes [dynamic volume
+   provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/).
 
-## Implementing Volume Group Snapshot Functionality in Your CSI Driver
+## Implementing Volume Group Snapshot Functionality in your CSI Driver
 
 To implement the volume group snapshot feature, a CSI driver MUST:
 
@@ -31,13 +37,21 @@ For details,  see the [CSI spec](https://github.com/container-storage-interface/
 
 ## Sidecar Deployment
 
-The Kubernetes CSI development team maintains the [external-snapshotter](external-snapshotter.md) Kubernetes CSI [Sidecar Containers](sidecar-containers.md). This sidecar container implements the logic for watching the Kubernetes API objects and issuing the appropriate CSI volume group snapshot calls against a CSI endpoint. For more details, see [external-snapshotter documentation](external-snapshotter.md).
+The Kubernetes CSI development team maintains the
+[external-snapshotter](external-snapshotter.md) Kubernetes CSI [Sidecar
+Containers](sidecar-containers.md). This sidecar container implements the logic
+for watching the Kubernetes API objects and issuing the appropriate CSI volume
+group snapshot calls against a CSI endpoint. For more details, see
+[external-snapshotter documentation](external-snapshotter.md).
 
 ## Volume Group Snapshot APIs
 
-With the introduction of Volume Group Snapshot, the user can create and delete a group snapshot using Kubernetes APIs. 
+With the introduction of Volume Group Snapshot, the user can create and delete a
+group snapshot using Kubernetes APIs.
 
-The schema definition for the custom resources (CRs) can be found [here](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd). The CRDs should be installed by the Kubernetes distributions.
+The schema definition for the custom resources (CRs) can be found
+[here](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd).
+The CRDs should be installed by the Kubernetes distributions.
 
 There are 3 APIs:
 
@@ -63,21 +77,34 @@ created. including the driver information, the deletion policy, etc.
 
 ## Controller
 
-* The controller logic for volume group snapshot is added to the snapshot controller and the CSI external-snapshotter sidecar.
+* The controller logic for volume group snapshot is added to the snapshot
+  controller and the CSI external-snapshotter sidecar.
 
-The snapshot controller is deployed by the Kubernetes distributions and is responsible for watching the VolumeGroupSnapshot CRD objects and manges the creation and deletion lifecycle of volume group snapshots.
+The snapshot controller is deployed by the Kubernetes distributions and is
+responsible for watching the VolumeGroupSnapshot CRD objects and manges the
+creation and deletion lifecycle of volume group snapshots.
 
-The CSI external-snapshotter sidecar watches Kubernetes VolumeGroupSnapshotContent CRD objects and triggers CreateVolumeGroupSnapshot/DeleteVolumeGroupSnapshot against a CSI endpoint.
+The CSI external-snapshotter sidecar watches Kubernetes
+VolumeGroupSnapshotContent CRD objects and triggers
+CreateVolumeGroupSnapshot/DeleteVolumeGroupSnapshot against a CSI endpoint.
 
 ## Snapshot Validation Webhook
 
-The validating webhook server is updated to validate volume group snapshot objects. This SHOULD be installed by the Kubernetes distros along with the snapshot-controller, not end users. It SHOULD be installed in all Kubernetes clusters that has the volume group snapshot feature enabled. See [Snapshot Validation Webhook](snapshot-validation-webhook.md) for more details on how to use the webhook.
+The validating webhook server is updated to validate volume group snapshot
+objects. This SHOULD be installed by the Kubernetes distros along with the
+snapshot-controller, not end users. It SHOULD be installed in all Kubernetes
+clusters that has the volume group snapshot feature enabled. See [Snapshot
+Validation Webhook](snapshot-validation-webhook.md) for more details on how to
+use the webhook.
 
 ## Kubernetes Cluster Setup
 
-See the Deployment section of [Snapshot Controller](snapshot-controller.md) on how to set up the snapshot controller and CRDs.
+See the Deployment section of [Snapshot Controller](snapshot-controller.md) on
+how to set up the snapshot controller and CRDs.
 
-See the Deployment section of [Snapshot Validation Webhook](snapshot-validation-webhook.md) for more details on how to use the webhook.
+See the Deployment section of [Snapshot Validation
+Webhook](snapshot-validation-webhook.md) for more details on how to use the
+webhook.
 
 ## Test Volume Group Snapshot Feature
 
@@ -88,32 +115,53 @@ Create a _StorageClass_:
 kubectl create -f storageclass.yaml
 ```
 
-Create _PVC_:
+Create _PVCs_:
 ```
+# This will create a PVC named hpvc
 kubectl create -f pvc.yaml
-kubectl create -f pvc2.yaml
+
+# This will create a PVC named hpvc-2
+sed "s/hpvc/hpvc-2/" pvc.yaml | kubectl create -f -
 ```
 
 Add a label to _PVC_:
 ```
-kubectl label pvcs <your-pvc-name> group:my-apps
+kubectl label pvc hpvc hpvc-2 app.kubernetes.io/name=postgresql
 ```
 
 Create a _VolumeGroupSnapshotClass_:
 ```
-kubectl create -f group-snapshot-class.yaml
+kubectl create -f groupsnapshotclass-v1alpha1.yaml
 ```
 
 Create a _VolumeGroupSnapshot_:
 ```
-kubectl create -f group-snapshot.yaml
+kubectl create -f groupsnapshot-v1alpha1.yaml
+```
+
+Once the _VolumeGroupSnapshot_ is ready, the `pvcVolumeSnapshotRefList` status field will contain the names of the generated _VolumeSnapshot_ objects:
+```
+kubectl get volumegroupsnapshot new-groupsnapshot-demo  -o yaml | sed -n '/pvcVolumeSnapshotRefList/,$p'
+
+  pvcVolumeSnapshotRefList:
+  - persistentVolumeClaimRef:
+      name: hpvc
+    volumeSnapshotRef:
+      name: snapshot-4bcc4a322a473abf32babe3df5779d14349542b1f0eb6f9dab0466a85c59cd42-2024-06-19-12.35.17
+  - persistentVolumeClaimRef:
+      name: hpvc-2
+    volumeSnapshotRef:
+      name: snapshot-62bd0be591e1e10c22d51748cd4a53c0ae8bf52fabb482bee7bc51f8ff9d9589-2024-06-19-12.35.17
+  readyToUse: true
 ```
 
 Create a _PVC_ from a _VolumeSnapshot_ that is part of the group snapshot:
 ```
-kubectl create -f restore-from-snapshot-in-group.yaml
+# In the command below, the volume snapshot name should be chosen from
+# the ones listed in the output of the previous command
+sed 's/new-snapshot-demo-v1/snapshot-4bcc4a322a473abf32babe3df5779d14349542b1f0eb6f9dab0466a85c59cd42-2024-06-19-12.35.17/' restore.yaml | kubectl create -f -
 ```
 
 ## Examples
 
-See the [Drivers](drivers.md) for a list of CSI drivers that implement the snapshot feature.
+See the [Drivers](drivers.md) for a list of CSI drivers that implement the group snapshot feature.

book/src/sidecar-containers.md

@@ -1,8 +1,11 @@
 # Kubernetes CSI Sidecar Containers
 
-Kubernetes CSI Sidecar Containers are a set of standard containers that aim to simplify the development and deployment of CSI Drivers on Kubernetes.
+Kubernetes CSI Sidecar Containers are a set of standard containers that aim to
+simplify the development and deployment of CSI Drivers on Kubernetes.
 
-These containers contain common logic to watch the Kubernetes API, trigger appropriate operations against the “CSI volume driver” container, and update the Kubernetes API as appropriate.
+These containers contain common logic to watch the Kubernetes API, trigger
+appropriate operations against the “CSI volume driver” container, and update the
+Kubernetes API as appropriate.
 
 The containers are intended to be bundled with third-party CSI driver containers and deployed together as pods.
 

book/src/snapshot-controller.md

@@ -34,10 +34,14 @@ For more information on the CSI external-snapshotter sidecar, see [this external
 
 The snapshot controller will be watching the Kubernetes API server for `VolumeSnapshot` and `VolumeSnapshotContent` CRD objects. The CSI `external-snapshotter` sidecar only watches the Kubernetes API server for `VolumeSnapshotContent` CRD objects. The snapshot controller will be creating the `VolumeSnapshotContent` CRD object which triggers the CSI `external-snapshotter` sidecar to create a snapshot on the storage system.
 
+The snapshot controller will be watching for `VolumeGroupSnapshot` and `VolumeGroupSnapshotContent` CRD objects when Volume Group Snapshot support is enabled via the `--enable-volume-group-snapshots` option.
+
 For detailed snapshot beta design changes, see the design doc [here](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/README.md).
 
 For detailed information about volume snapshot and restore functionality, see [Volume Snapshot & Restore](snapshot-restore-feature.md).
 
+For detailed information about volume group snapshot and restore functionality, see [Volume Snapshot & Restore](group-snapshot.md).
+
 For detailed information (binary parameters, RBAC rules, etc.), see [https://github.com/kubernetes-csi/external-snapshotter/blob/release-6.2/README.md](https://github.com/kubernetes-csi/external-snapshotter/blob/release-6.2/README.md).
 
 ## Deployment

book/src/snapshot-restore-feature.md

@@ -8,6 +8,8 @@ Alpha | 1.12 | 1.12 | | | 0.4.0 <= version < 1.0 | 0.4.1 <= version < 1.0
 Alpha | 1.13 | 1.16 | | | 1.0.1 <= version < 2.0 | 1.0.1 <= version < 1.5
 Beta  | 1.17 | - | 2.0+ | 3.0+ | 2.0+ | 1.5+
 
+**IMPORTANT**: The validation logic for VolumeSnapshots and VolumeSnapshotContents has been replaced by CEL validation rules. The validating webhook is now only being used for VolumeSnapshotClasses to ensure that there's at most one class per CSI Driver. The validation webhook is deprecated and will be removed in the next release.
+
 ## Overview
 
 Many storage systems provide the ability to create a "snapshot" of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to provision a new volume (pre-populated with the snapshot data) or to restore the existing volume to a previous state (represented by the snapshot).
@@ -83,12 +85,12 @@ kubectl create -f pvc.yaml
 
 Create a _VolumeSnapshotClass_:
 ```
-kubectl create -f snapshotclass.yaml
+kubectl create -f snapshotclass-v1.yaml
 ```
 
 Create a _VolumeSnapshot_:
 ```
-kubectl create -f snapshot.yaml
+kubectl create -f snapshot-v1.yaml
 ```
 
 Create a _PVC_ from a _VolumeSnapshot_: