Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: nydus-snapshotter can not run cause of failing of postStart hook #209

Merged
merged 1 commit into from
Dec 11, 2023

Conversation

cheriL
Copy link
Contributor

@cheriL cheriL commented Nov 30, 2023

nydus-snapshotter can not run cause of failing of postStart hook when containerd config is disabled.

events:

  Warning  FailedPostStartHook  28s (x4 over 2m46s)  kubelet            Exec lifecycle hook ([/bin/sh -c # currently, without host pid in container, we can not nsenter with pid and can not invoke systemctl correctly.
nsenter -t 1 -m systemctl -- restart containerd.service
]) for Container "nydus-snapshotter" in Pod "nydus-snapshotter-kg8fs_nydus-snapshotter(6959e4c6-957e-4065-a7d9-b199c90a6535)" failed - error: command '/bin/sh -c # currently, without host pid in container, we can not nsenter with pid and can not invoke systemctl correctly.
nsenter -t 1 -m systemctl -- restart containerd.service
' exited with 127: nsenter: can't execute 'systemctl': No such file or directory
, message: "nsenter: can't execute 'systemctl': No such file or directory\n"
  Normal  Killing  28s (x4 over 2m46s)  kubelet  FailedPostStartHook

@gaius-qi
Copy link
Member

@cheriL Please fix lint.

@cheriL cheriL force-pushed the fix_nydus_ds branch 2 times, most recently from 3938ae1 to 767f971 Compare December 1, 2023 01:51
@cheriL
Copy link
Contributor Author

cheriL commented Dec 1, 2023

@gaius-qi done

Copy link
Member

@adamqqqplay adamqqqplay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your fix. I have a small question, can nydus-snapshotter work properly when containerd config is disabled?

@cheriL
Copy link
Contributor Author

cheriL commented Dec 1, 2023

Thanks for your fix. I have a small question, can nydus-snapshotter work properly when containerd config is disabled?

@adamqqqplay Actually, I'm new to nydus and still testing it. The main reason for this change is that I don't want to change the config of containerd by injecting when the daemonset pod starting. And would we consider separating the hostpid and containerd config injection?

@cheriL
Copy link
Contributor Author

cheriL commented Dec 1, 2023

@adamqqqplay I started the nydus daemon pod (with hostPid=true), and configured /etc/containerd/config.toml manually, but still encountered this problem with nydus image:

  • ✅ busybox:1.32 (traditional oci image)
  • ✅ busybox:1.32-esgz (nerdctl -n k8s.io image convert --estargz --oci busybox:1.32 busybox:1.32-esgz)
  • ❌ busybox:1.32-nydus (nerdctl -n k8s.io image convert --nydus --oci busybox:1.32 busybox:1.32-nydus)

Event:

Warning  Failed          13m (x12 over 15m)  kubelet  Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:49de949245b76bb0e2451821fb63dfd2c349decfd047e6a158a20a58a47e08c8: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown

nydus-snapshotter version: v0.13.3
nydus inside pod: v2.1.5

[root@worker12 ~]# ctr plugins ls | grep nydus
io.containerd.snapshotter.v1          nydus                    -              ok

@adamqqqplay
Copy link
Member

@adamqqqplay I started the nydus daemon pod (with hostPid=true), and configured /etc/containerd/config.toml manually, but still encountered this problem with nydus image:

  • ✅ busybox:1.32 (traditional oci image)
  • ✅ busybox:1.32-esgz (nerdctl -n k8s.io image convert --estargz --oci busybox:1.32 busybox:1.32-esgz)
  • ❌ busybox:1.32-nydus (nerdctl -n k8s.io image convert --nydus --oci busybox:1.32 busybox:1.32-nydus)

Event:

Warning  Failed          13m (x12 over 15m)  kubelet  Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:49de949245b76bb0e2451821fb63dfd2c349decfd047e6a158a20a58a47e08c8: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown

nydus-snapshotter version: v0.13.3 nydus inside pod: v2.1.5

[root@worker12 ~]# ctr plugins ls | grep nydus
io.containerd.snapshotter.v1          nydus                    -              ok

@cheriL I think your containerd configuration may be overwritten by /etc/containerd/config.toml in kind. Maybe you can remove the field in extraMounts.

      - hostPath: ./config.toml
        containerPath: /etc/containerd/config.toml

@cheriL
Copy link
Contributor Author

cheriL commented Dec 4, 2023

@adamqqqplay My k8s is not set up by kind, the reason is weired that the image converted by nerdctl image convert --nydus didn't work(nerdctl v1.7.0).

… when containerd config is disabled

Signed-off-by: soup <lqh348659137@outlook.com>
Copy link
Member

@adamqqqplay adamqqqplay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@adamqqqplay adamqqqplay merged commit 2f6d13f into dragonflyoss:main Dec 11, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants