implement `takeover` for nydus fscache/fusedev mode #1487

kevinXYin · 2023-11-18T06:45:46Z

Relevant Issue (if applicable)

#1421

Details

This is based on #1434 , make the takeover feature can support both fusedev and fscache mode.

Types of changes

What types of changes does your PullRequest introduce? Put an x in all the boxes that apply:

Bug fix (non-breaking change which fixes an issue)
[x ] New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation Update (if none of the other choices apply)

Checklist

Go over all the following points, and put an x in all the boxes that apply.

I have updated the documentation accordingly.
I have added tests to cover my changes.

Cargo.toml

ccx1024cc

Smoke test is required.

codecov · 2023-11-26T12:16:05Z

Codecov Report

Merging #1487 (a038f89) into master (50b8988) will decrease coverage by 0.11%.
Report is 2 commits behind head on master.
The diff coverage is 51.28%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1487      +/-   ##
==========================================
- Coverage   62.82%   62.72%   -0.11%     
==========================================
  Files         125      129       +4     
  Lines       43599    44158     +559     
  Branches    43599    44158     +559     
==========================================
+ Hits        27391    27698     +307     
- Misses      14854    15092     +238     
- Partials     1354     1368      +14

Files	Coverage Δ
api/src/http.rs	`0.00% <ø> (ø)`
upgrade/src/lib.rs	`100.00% <100.00%> (ø)`
api/src/config.rs	`87.61% <0.00%> (ø)`
upgrade/src/backend/mod.rs	`97.56% <97.56%> (ø)`
service/src/lib.rs	`78.66% <0.00%> (ø)`
service/src/daemon.rs	`11.35% <0.00%> (-0.13%)`	⬇️
src/bin/nydusd/main.rs	`0.19% <0.00%> (-0.01%)`	⬇️
rafs/src/fs.rs	`34.53% <0.00%> (-0.16%)`	⬇️
src/bin/nydusd/api_server_glue.rs	`0.00% <0.00%> (ø)`
service/src/fusedev.rs	`0.00% <0.00%> (ø)`
... and 6 more

... and 2 files with indirect coverage changes

This patch implements the `save` and `restore` functions in the `fusedev_upgrade` in the service create. To do this, - This patch add a new create named `nydus-upgrade` into the workspace. The `nydus-upgrade` create has some util functions help to do serialization and deserialization for the rust structs using the versionize and snapshot crates. The crate also has a trait named `StorageBackend` which can be used to store and restore fuse session fds and state data for the upgrade action, and there's also an implementation named `UdsStorageBackend` which uses unix domain socket to do this. - as we have to use the same fuse session connection, backend file system mount commands, Vfs to re-mount the rafs for the new daemon (created for "hot upgrade" or failover), this patch add a new struct named `FusedevState` to hold these information. The `FusedevState` is serialized and stored into the `UdsStorageBackend` (which happens in the `save` function in the `fusedev_upgrade` module) before the new daemon is created, and the `FusedevState` is deserialized and restored from the `UdsStorageBackend` (which happens in the `restore` function in the `fusedev_upgrade` module) when the new daemon is triggered by `takeover`. Signed-off-by: Nan Li <loheagn@icloud.com> Signed-off-by: linan.loheagn3 <linan.loheagn3@bytedance.com>

upgrade/Cargo.toml

upgrade/src/backend/mod.rs

src/bin/nydusd/api_server_glue.rs

service/src/singleton.rs

jiangliu · 2023-12-07T09:51:54Z

Otherwise LGTM, thanks for the great work:)

imeoer

Thanks for the work, let's add a smoke test case for validating fusedev failover later cc @loheagn .

refine the UpgradeManager, make it can also support store status for fscache daemon. And make the takeover feature applies to both fuse and fscache mode. Signed-off-by: Xin Yin <yinxin.x@bytedance.com>

Signed-off-by: Xin Yin <yinxin.x@bytedance.com>

delay test

kevinXYin requested a review from a team as a code owner November 18, 2023 06:45

kevinXYin requested review from bergwolf, liubin and adamqqqplay and removed request for a team November 18, 2023 06:45

kevinXYin marked this pull request as draft November 18, 2023 06:46

ccx1024cc suggested changes Nov 20, 2023

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

ccx1024cc previously requested changes Nov 20, 2023

View reviewed changes

kevinXYin force-pushed the takeover branch from 9bc531c to ad9d59b Compare November 26, 2023 12:07

hsiangkao requested a review from ccx1024cc December 6, 2023 10:00

kevinXYin force-pushed the takeover branch from ad9d59b to 50e9a03 Compare December 6, 2023 10:22

kevinXYin marked this pull request as ready for review December 6, 2023 10:27

kevinXYin force-pushed the takeover branch 3 times, most recently from 2bb9a7c to c7ed6f0 Compare December 6, 2023 12:56

hsiangkao changed the title ~~Takeover~~ implement takeover for nydus fscache mode Dec 6, 2023

kevinXYin force-pushed the takeover branch 2 times, most recently from a8ce6f9 to 2b0a2a6 Compare December 7, 2023 01:47