rbd: VolumeGroupSnapshot support #4502
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nixpanic
force-pushed
the
rbd/group-snapshot
branch
from
25e7f97
to
65c17e7
Compare
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions. |
nixpanic
force-pushed
the
rbd/group-snapshot
branch
3 times, most recently
from
57e136d
to
6d85b34
Compare
|
This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏 |
nixpanic
force-pushed
the
rbd/group-snapshot
branch
from
6d85b34
to
d60f341
Compare
nixpanic
force-pushed
the
rbd/group-snapshot
branch
from
d60f341
to
3722692
Compare
nixpanic
force-pushed
the
rbd/group-snapshot
branch
from
3722692
to
0ca9915
Compare
|
/test ci/centos/mini-e2e-helm/k8s-1.30 |
nixpanic
force-pushed
the
rbd/group-snapshot
branch
2 times, most recently
from
3e5713f
to
5c9ae9f
Compare
This was referenced Oct 3, 2024
|
@Mergifyio refresh |
✅ Pull request refreshed |
nixpanic
force-pushed
the
rbd/group-snapshot
branch
from
5c9ae9f
to
b55ddb2
Compare
This was referenced Oct 10, 2024
nixpanic
force-pushed
the
rbd/group-snapshot
branch
from
b55ddb2
to
9c2fdec
Compare
This was referenced Oct 11, 2024
nixpanic
force-pushed
the
rbd/group-snapshot
branch
3 times, most recently
from
0c356f7
to
5d3b859
Compare
|
/test ci/centos/mini-e2e-helm/k8s-1.29 |
|
/test ci/centos/mini-e2e/k8s-1.30 |
|
/test ci/centos/k8s-e2e-external-storage/1.31 |
|
/test ci/centos/mini-e2e/k8s-1.29 |
|
/test ci/centos/mini-e2e-helm/k8s-1.31 |
|
/test ci/centos/mini-e2e/k8s-1.31 |
Madhu-1
reviewed
Nov 4, 2024
internal/rbd/rbd_util.go
Outdated
| @@ -578,6 +578,10 @@ func (ri *rbdImage) isInUse() (bool, error) { | |||
| defaultWatchers += count | |||
| } | |||
|
|
|||
| if len(watchers) > defaultWatchers { | |||
| log.ErrorLogMsg("too many watchers (%d > %d): %v", len(watchers), defaultWatchers, watchers) | |||
There was a problem hiding this comment.
@nixpanic This wont be useful unless we log this with the context or with details about the rbd image
There was a problem hiding this comment.
That's a debugging change I made while tracking a left-over opened image. I'll drop this with the next update.
There was a problem hiding this comment.
Lets drop it, I will place LGTM
There was a problem hiding this comment.
Done, dropped the commit and rebased. Please have a look again.
Madhu-1
reviewed
Nov 4, 2024
When the image is not closed, it keeps a watch open. This prevents the CSI Controller to delete the Volume, as there is still a user of it. Fixes: f9ab14e "rbd: check if an image is part of a group before adding it" Signed-off-by: Niels de Vos <ndevos@ibm.com>
Commit 95733b3 introduced the `StoreGroupID()` function, but that unfortunately set an empty key in the journal. Passing the `csiGroupIDKey` key (with value `csi.groupid`) caused setting `csi.csi.groupid` as a key. Reading the value back with the right `csi.groupid` key always returned an empty value. Fixes: 95733b3 "journal: add option to store the groupID" Signed-off-by: Niels de Vos <ndevos@ibm.com>
…ot found Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
Each object is responsible for maintaining a connection to the journal. By sharing a single journal, cleanup of objects becomes more complex as the journal is used in deferred functions and only the last should destroy the journal connection resources. Signed-off-by: Niels de Vos <ndevos@ibm.com>
The NewSnapshotByID() function makes it possible to clone a new Snapshot from an existing RBD-image and the ID of an RBD-snapshot on that image. This will be used by the VolumeGroupSnapshot feature, where the ID of an RBD-snapshot is obtained for the RBD-snapshot on the RBD-images. Signed-off-by: Niels de Vos <ndevos@ibm.com>
When the rbd.Manager creates a VolumeGroupSnapshot, each RBD-snapshot that is created as part of the RBD-group needs to be cloned into its own RBD-image that will be used as a CSI Snapshot. The VolumeGroup.CreateSnapshots() creates the RBD-group snapshot and returns a list of the Snapshot structs. Signed-off-by: Niels de Vos <ndevos@ibm.com>
The VolumeGroupSnapshot type will be used by the rbd.Manager to create, inspect and delete VolumeGroupSnapshos. Signed-off-by: Niels de Vos <ndevos@ibm.com>
A (CSI) VolumeGroupSnapshot object contains references to Snapshot IDs (or CSI Snapshot handles). In order to work with a VolumeGroupSnapshot struct, the Snapshot IDs need to be resolved into rbdSnapshot structs. Signed-off-by: Niels de Vos <ndevos@ibm.com>
Implement the CreateVolumeGroupSnapshot for the rbd.Manager. A Group Controller Server can use the rbd.Manager to create VolumeGroupSnapshots in an easy an idempotent way. Signed-off-by: Niels de Vos <ndevos@ibm.com>
The GetVolumeGroupSnapshotByID function makes it possible to get a VolumeGroupSnapshot object from the Manager by passing a request-id. This makes it simple for the Group Controller Server to check if a VolumeGroupSnapshot already exists, so it is not needed to try and re-create an existing one. Signed-off-by: Niels de Vos <ndevos@ibm.com>
The Group Controller Server may need to fetch a VolumeGroupSnapshot that was statically provisioned. In that case, only the name of the VolumeGroupSnapshot is known and should be resolved to an object. Signed-off-by: Niels de Vos <ndevos@ibm.com>
When creating a Snapshot with the new NewSnapshotByID() function, the name of the RBD-image that is created is the same as the name of the Snapshot. The `RbdImageName` points to the name of parent image, which causes deleting the Snapshot to delete the parent image instead. Correcting the `RbdImageName` and setting it to the `RbdSnapName` makes sure that upon deletion, the Snapshot RBD-image is removed, and not the parent image. Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
When the GroupSnapGetInfo go-ceph function is supported by librbd, the Group Controller Servive and VolumeGroupSnapshot capabilities can be exposed to the Container Orchestrator. Signed-off-by: Niels de Vos <ndevos@ibm.com>
Without the SnapshotGroupID in the Snapshot object, Kubernetes CSI does not know that the Snapshot belongs to a group. In that case, it allows the deletion of the Snapshot, which should be denied. Signed-off-by: Niels de Vos <ndevos@ibm.com>
nixpanic
force-pushed
the
rbd/group-snapshot
branch
from
0c48b27
to
4d68022
Compare
Madhu-1
approved these changes
Nov 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add support for VolumeGroupSnapshots in RBD. The last two commits enable the feature in the Group Controller Server and expose the capability. All other commits provide the functionality through the
rbd.Managerinterface.Currently there is no Ceph container-image release that provides the required librbd features. Building from this PR will not provide support for VolumeGroupSnapshot yet, the base container-image needs to be set to Ceph CI main branch (
quay.ceph.io/ceph-ci/ceph:main) for that. A test-image based on Ceph main can be found atquay.io/nixpanic/cephcsi:pr_4739.Notable changes:
internal/rbd_typespackage with interfaces so that objects can be passed around cleanerinternal/rbd/volume.goimplementing the newVolumeinterface forrbdImageinternal/rbd_grouppackage for all RBD-group functionalitiesinternal/rbd/group_controller.gofor all CSI VolumeGroup service proceduresDepends-on: #4794
Depends-on: #4870
Depends-on: #4871
Depends-on: #4884
Depends-on: #4885
Depends-on: #4898
Depends-on: #4902
Depends-on: #4904
Depends-on: #4915
Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:
/retest ci/centos/<job-name>: retest the<job-name>after unrelatedfailure (please report the failure too!)