Btrfs snapshotter implementation #1957

davidcassany · 2024-02-14T10:46:52Z

This PR includes the btrfs snapshotter implementation and required changes to specially in the boot process area (elemental-sysroot, elemental-setup, grub, etc.) to also support btrfs snapshots.

Fixes #1923
Fixes #1925

davidcassany · 2024-02-14T10:50:27Z

For the time being this is missing the unit tests and few adaptations to fix/skip integration tests. Note that a btrfs deployment can't be downgraded to an image that does not apply the new btrfs related setup for booting.

Included now

codecov-commenter · 2024-02-14T10:51:03Z

Codecov Report

Attention: 281 lines in your changes are missing coverage. Please review.

Comparison is base (3917e95) 72.04% compared to head (39992cb) 73.10%.

Files	Patch %	Lines
pkg/snapshotter/btrfs.go	73.24%	117 Missing and 28 partials ⚠️
pkg/snapshotter/loopdevice.go	66.25%	20 Missing and 7 partials ⚠️
pkg/action/upgrade.go	33.33%	21 Missing and 5 partials ⚠️
pkg/types/v1/snapshotter.go	24.00%	16 Missing and 3 partials ⚠️
pkg/action/build-disk.go	61.11%	11 Missing and 3 partials ⚠️
pkg/utils/common.go	52.00%	9 Missing and 3 partials ⚠️
pkg/action/mount.go	60.00%	6 Missing and 4 partials ⚠️
pkg/elemental/elemental.go	63.63%	5 Missing and 3 partials ⚠️
pkg/action/install.go	28.57%	4 Missing and 1 partial ⚠️
pkg/action/reset.go	16.66%	4 Missing and 1 partial ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1957      +/-   ##
==========================================
+ Coverage   72.04%   73.10%   +1.05%     
==========================================
  Files          72       74       +2     
  Lines        8041     8666     +625     
==========================================
+ Hits         5793     6335     +542     
- Misses       1784     1819      +35     
- Partials      464      512      +48

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kkaempf · 2024-02-14T10:55:19Z

.obs/specfile/elemental-toolkit.spec

-Requires:       xorriso >= 1.5.6
+Requires:       btrfsprogs
+Requires:       snapper
+Requires:       xorriso >= 1.5


That's effectively a xorriso downgrade !?

No, this is just making the constraint more relaxed. We do not care about the patch level version. Our real constraint is on v1.5.x. Leap 15.5 and SP5 ship 1.4.x.

kkaempf · 2024-02-14T10:56:46Z

cmd/config/config.go

@@ -301,7 +301,7 @@ func applyKernelCmdline(r *v1.RunConfig, mount *v1.MountSpec) error {
 		}

 		switch split[0] {
-		case "elemental.image", "cos-img/filename":
+		case "elemental.mode":


All these (breaking?) changes need good documentation

Yes, boot process needs to be documented

Not really a breaking chance since these kernel parameters are set in /etc/cos/bootargs.cfg which is always stored in the image which also contains the initrd that has to read them. In fact thats on of the reasons to not include any specific kernel argument in grub.cfg (which is stored in EFI partition and common to all installed images). Changes in grub.cfg are real breaking changes.

kkaempf · 2024-02-14T10:59:54Z

examples/green/snapshotter.yaml

@@ -0,0 +1,3 @@
+snapshotter:
+  type: btrfs


isn't "type" superfluous ? (are there any other types 🤔 )

Yes, it can use a loopdevice based approach which is the default to facilitate upgrades from 5.5

For the loop device approach I would assume snapshotter: to be absent. 😕

But

snapshotter: type: loopdevice

is strange since the old way didn't really do snapshots ?!

If snapshotter is undefined, not included within the config yaml defaults to loopdevice. My idea is including a specific config for SLE Micro images through the elemental package as in rancher/elemental#1213

Keeping type: btrfs needed IMHO makes sense as there is no reason to assume there can't be a additional snapshotter interface implementations in the future. I can easily dream on an overlayfs based approach 🤔

davidcassany · 2024-02-14T11:48:34Z

pkg/utils/common.go

@@ -175,7 +175,7 @@ func SyncData(log v1.Logger, runner v1.Runner, fs v1.FS, source string, target s

 	log.Infof("Starting rsync...")

-	args := []string{"--progress", "--partial", "--human-readable", "--archive", "--xattrs", "--acls"}
+	args := []string{"--progress", "--partial", "--human-readable", "--archive", "--xattrs", "--acls", "--delete"}


This is to ensure we can track removed files across upgrades (image A contains a file which image B does not). Since in btrfs snapshots we upgrade over an already existing root-tree we need to make sure it perfectly mirrors the upgraded image. The library we use to extract an image does not perform a perfect sync, just overwrite existing files where needed.

davidcassany · 2024-02-14T11:59:07Z

pkg/snapshotter/btrfs.go

+	currentSnapshotID int
+	activeSnapshotID  int
+	bootloader        v1.Bootloader
+	installing        bool


We need to track first snapshot deployment in most of the methods of the public snapshotter interface. This is caused by snapper, as it always requires a previous subvolume to operate from. Hence, for the first deployment we are not using snapper to create the first snapshot and relay on btrfs commands to create a subvolume that looks like a btrfs snapper snapshot. This causes having two different execution paths and having to wrap different set of tools for operations that are pretty similar from an Elemental PoV. This is also an inconvenience in SLE Micro, KIWI runs similar tricks to make first root as a snapshot of disk images.

snapper is mostly though to operate on a running system with certain pre-installed btrfs layoyt, but it is not really useful to run a clean installation and deploy the first root as a snapper snapshot.

Would it make sense to separate this into a Btrfs snapshotter and a snapper snapshotter?

davidcassany · 2024-02-14T12:13:01Z

cmd/config/config.go

@@ -301,7 +301,7 @@ func applyKernelCmdline(r *v1.RunConfig, mount *v1.MountSpec) error {
 		}

 		switch split[0] {
-		case "elemental.image", "cos-img/filename":
+		case "elemental.mode":


Yes, boot process needs to be documented

davidcassany · 2024-02-14T12:15:23Z

pkg/features/embedded/elemental-rootfs/usr/lib/systemd/system/elemental-rootfs.service

@@ -2,7 +2,7 @@
 Description=Elemental system rootfs overlay mounts
 DefaultDependencies=no
 After=initrd-root-fs.target
-Requires=initrd-root-fs.target
+Wants=initrd-root-fs.target


Relaxing this requirement so we could consider removing the RemainAfterExit if needed, which could be useful to track start and stop of the service.

davidcassany · 2024-02-14T12:17:56Z

pkg/features/embedded/grub-config/etc/elemental/grub.cfg

+  set_volume
+  source (${volume})/etc/cos/bootargs.cfg
+  linux (${volume})${kernel} ${kernelcmd} ${extra_cmdline} ${extra_active_cmdline}
+  initrd (${volume})${initramfs}
 }

 for passive_snap in ${passive_snaps}; do


passive_snaps includes a space separated list of passive snapshots IDs. Then the path of the image is computed in set_volume function above.

davidcassany · 2024-02-14T12:22:39Z

cmd/config/config.go

@@ -301,7 +301,7 @@ func applyKernelCmdline(r *v1.RunConfig, mount *v1.MountSpec) error {
 		}

 		switch split[0] {
-		case "elemental.image", "cos-img/filename":
+		case "elemental.mode":


Not really a breaking chance since these kernel parameters are set in /etc/cos/bootargs.cfg which is always stored in the image which also contains the initrd that has to read them. In fact thats on of the reasons to not include any specific kernel argument in grub.cfg (which is stored in EFI partition and common to all installed images). Changes in grub.cfg are real breaking changes.

davidcassany · 2024-02-14T12:26:33Z

pkg/action/build-disk.go

+
+	if b.cfg.Snapshotter.Type == constants.BtrfsSnapshotterType {
+		if !b.spec.Expandable {
+			cfg.Logger.Errorf("Non expandable disk images are not supported for btrfs snapshotter")


Shouldn't be hard to implement support for non expandable (full disk including all partitions) disk images in way that is compatible for btrfs snapshotter too, but requires some build-disk command refactor which is beyond the scope of this PR and also I don't think is really a relevant use case for Elemental.

davidcassany · 2024-02-14T12:27:44Z

pkg/action/build-disk.go


 	rootMap := map[string]string{}

 	if b.spec.Expandable {
-		exclude = b.spec.Partitions.Persistent
+		excludes = append(excludes, b.spec.Partitions.Persistent, b.spec.Partitions.State)


We also exclude State partition from the build since this is no longer required as the grub configuration is stored in EFI partition.

davidcassany · 2024-02-14T12:29:33Z

pkg/features/embedded/elemental-setup/usr/lib/systemd/system/elemental-setup-initramfs.service

@@ -8,7 +8,6 @@ Before=initrd.target
 [Service]
 RootDirectory=/sysroot
 BindPaths=/proc /sys /dev /run /tmp
-BindPaths=-/oem


Verified that the mount command already mounts that on earlier stages. Former initrd /oem mountpoint is unmounted by stopping initrd.target before switching root (PartOf relationship).

Yep, verified that initramfs stage is still executed for yaml files stored in /oem 👍

davidcassany · 2024-02-14T12:32:33Z

.obs/specfile/elemental-toolkit.spec

-Requires:       xorriso >= 1.5.6
+Requires:       btrfsprogs
+Requires:       snapper
+Requires:       xorriso >= 1.5


No, this is just making the constraint more relaxed. We do not care about the patch level version. Our real constraint is on v1.5.x. Leap 15.5 and SP5 ship 1.4.x.

davidcassany · 2024-02-16T16:01:06Z

Just got a green CI run here https://github.com/rancher/elemental/actions/runs/7932002562/job/21658757154 🎉
Gonna rebase, lauch CI again and move it to ready to review

davidcassany · 2024-02-16T16:15:41Z

Requires rancher/elemental#1213 to be merged after/together

pkg/action/mount.go

Signed-off-by: David Cassany <dcassany@suse.com>

…ition on expandable images Signed-off-by: David Cassany <dcassany@suse.com>

Signed-off-by: David Cassany <dcassany@suse.com>

…nfig constructor Signed-off-by: David Cassany <dcassany@suse.com>

Signed-off-by: David Cassany <dcassany@suse.com>

Current loopdevice snapshotter does not require anymore having symlinks to list passive snapshots. Current grub2 configuration configures the passive snapshot path based on the snapshot ID. Signed-off-by: David Cassany <dcassany@suse.com>

Signed-off-by: David Cassany <dcassany@suse.com>

davidcassany · 2024-02-21T12:27:04Z

UI CI run passes at: https://github.com/rancher/elemental/actions/runs/7988344296
CLI CI run passes for k3s at: https://github.com/rancher/elemental/actions/runs/7987631264

test-downgrade and test-recovery are not going to pass in this PR, incompatibility to downgrade from elemental v2 to v1.x. Workarounds for tests are to be included in separate and follow up PRs.

davidcassany requested a review from a team as a code owner February 14, 2024 10:46

davidcassany marked this pull request as draft February 14, 2024 10:47

kkaempf reviewed Feb 14, 2024

View reviewed changes

davidcassany commented Feb 14, 2024

View reviewed changes

davidcassany force-pushed the btrfs_implementation branch from 4eb4eb0 to eeaaf88 Compare February 14, 2024 12:56

davidcassany force-pushed the btrfs_implementation branch from b8e7a47 to db4f41d Compare February 16, 2024 16:01

davidcassany marked this pull request as ready for review February 16, 2024 16:10

davidcassany mentioned this pull request Feb 16, 2024

Adapt bootargs file to support btrfs snapshotter rancher/elemental#1213

Merged

davidcassany force-pushed the btrfs_implementation branch from db4f41d to 2d01d9a Compare February 19, 2024 13:23

frelon reviewed Feb 19, 2024

View reviewed changes

pkg/action/mount.go Show resolved Hide resolved

davidcassany added 15 commits February 19, 2024 16:50

Implementation of Btrfs snapshotter

718cd1b

Signed-off-by: David Cassany <dcassany@suse.com>

Btrfs based examples

92dbee5

Signed-off-by: David Cassany <dcassany@suse.com>

Refined and adapt features

a70b9de

Signed-off-by: David Cassany <dcassany@suse.com>

Update build-disk to new snapshotter and prevent including State part…

75e6e9b

…ition on expandable images Signed-off-by: David Cassany <dcassany@suse.com>

Remove /oem bind mount in initramfs, already mounted by mount command

d89b079

Signed-off-by: David Cassany <dcassany@suse.com>

Adapt unit tests

9ccce16

Signed-off-by: David Cassany <dcassany@suse.com>

Add mount command unit tests

bb44f06

Signed-off-by: David Cassany <dcassany@suse.com>

Make grubfallback test more generic

e496594

Signed-off-by: David Cassany <dcassany@suse.com>

Adding btrfs snapshotter unit tests and fixing default snapshotter co…

81e10a3

…nfig constructor Signed-off-by: David Cassany <dcassany@suse.com>

Add utils test

f9900a7

Signed-off-by: David Cassany <dcassany@suse.com>

Fix upgrade ENV variables mapping

e74a6ef

Signed-off-by: David Cassany <dcassany@suse.com>

Include transactional-update package in example

6220842

Signed-off-by: David Cassany <dcassany@suse.com>

Fix persistent bind mounts

ac900a9

Signed-off-by: David Cassany <dcassany@suse.com>

Make sure state is RW mounted upgrading from legacy

b1e09a7

Signed-off-by: David Cassany <dcassany@suse.com>

Remove unused passive symlinks for loopdevice

5b67978

Current loopdevice snapshotter does not require anymore having symlinks to list passive snapshots. Current grub2 configuration configures the passive snapshot path based on the snapshot ID. Signed-off-by: David Cassany <dcassany@suse.com>

davidcassany force-pushed the btrfs_implementation branch from 1733a87 to 5b67978 Compare February 19, 2024 15:51

davidcassany added 2 commits February 19, 2024 18:04

Fix upgrade from older version

41dfb07

Signed-off-by: David Cassany <dcassany@suse.com>

Update hack message

39992cb

Signed-off-by: David Cassany <dcassany@suse.com>

frelon approved these changes Feb 21, 2024

View reviewed changes

davidcassany merged commit 8f802fa into rancher:main Feb 21, 2024
17 of 19 checks passed

davidcassany deleted the btrfs_implementation branch July 11, 2024 14:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Btrfs snapshotter implementation #1957

Btrfs snapshotter implementation #1957

davidcassany commented Feb 14, 2024 •

edited

Loading

davidcassany commented Feb 14, 2024 •

edited

Loading

codecov-commenter commented Feb 14, 2024 •

edited

Loading

kkaempf Feb 14, 2024

davidcassany Feb 14, 2024

kkaempf Feb 14, 2024

davidcassany Feb 14, 2024

davidcassany Feb 14, 2024

kkaempf Feb 14, 2024

davidcassany Feb 14, 2024

kkaempf Feb 14, 2024

davidcassany Feb 16, 2024

davidcassany Feb 14, 2024

davidcassany Feb 14, 2024

frelon Feb 20, 2024

davidcassany Feb 14, 2024

davidcassany Feb 14, 2024

davidcassany Feb 14, 2024

davidcassany Feb 14, 2024

davidcassany Feb 14, 2024

davidcassany Feb 14, 2024

davidcassany Feb 14, 2024

frelon Feb 19, 2024

davidcassany Feb 14, 2024

davidcassany commented Feb 16, 2024

davidcassany commented Feb 16, 2024

davidcassany commented Feb 21, 2024

Btrfs snapshotter implementation #1957

Btrfs snapshotter implementation #1957

Conversation

davidcassany commented Feb 14, 2024 • edited Loading

davidcassany commented Feb 14, 2024 • edited Loading

codecov-commenter commented Feb 14, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidcassany commented Feb 16, 2024

davidcassany commented Feb 16, 2024

davidcassany commented Feb 21, 2024

davidcassany commented Feb 14, 2024 •

edited

Loading

davidcassany commented Feb 14, 2024 •

edited

Loading

codecov-commenter commented Feb 14, 2024 •

edited

Loading