Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

seccomp: Can't find entry on tid_real #2508

Closed
rst0git opened this issue Nov 1, 2024 · 1 comment · Fixed by #2514
Closed

seccomp: Can't find entry on tid_real #2508

rst0git opened this issue Nov 1, 2024 · 1 comment · Fixed by #2514
Labels
bug

Comments

@rst0git
Copy link
Member

rst0git commented Nov 1, 2024

Steps to reproduce the issue:

  1. Create a Pod with the following manifest:
apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 101
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: counter
      image: busybox:latest
      command:
        - "/bin/sh"
        - "-c"
        - "i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done"
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
      resources:
        requests:
          memory: "1Gi"
          cpu: "1"
        limits:
          memory: "1Gi"
          cpu: "1"
  1. Create a container checkpoint using the kubelet Checkpoint API

Describe the results you received:
CRIU fails with the following error:

(00.122028) Collecting pidns 9/501089
(00.122037) Error (criu/seccomp.c:61): seccomp: Can't find entry on tid_real 501089
(00.122039) Error (criu/seccomp.c:278): seccomp: Can't collect filter on leader tid_real 501089
(00.122098) net: Unlock network
(00.122100) Running network-unlock scripts
(00.122101) 	RPC
(00.196082) cuda_plugin: finished cuda_plugin stage 0 err -1
(00.196155) Unfreezing tasks into 1
(00.196157) 	Unseizing 501089 into 1
(00.196165) Error (compel/src/lib/infect.c:418): Unable to detach from 501089: No such process
(00.196174) Error (criu/cr-dump.c:2111): Dumping FAILED

dump.log
@rst0git
Copy link
Member Author

rst0git commented Nov 2, 2024 •
edited
Loading

@avagin, using git bisect, I was able to confirm that this error is caused by the changes introduced in #2475
If we remove /usr/lib/criu/cuda_plugin.so, the error does not appear.

@rst0git rst0git added the bug label Nov 2, 2024
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: fix error handling for check_freezer_cgroup
f0d9f0d 
When check_freezer_cgroup() has non-zero, `goto err` calls `return ret`.
However, the value of `ret` has been set to 0 in the lines above and
CRIU does not handle the error.

This problem is related to checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
@rst0git rst0git mentioned this issue Nov 4, 2024
Merged
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: fix error handling for check_freezer_cgroup
63dc0f7 
When check_freezer_cgroup() has non-zero, `goto err` calls `return ret`.
However, the value of `ret` has been set to 0 in the lines above and
CRIU does not handle the error.

This problem is related to checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: fix error handling for check_freezer_cgroup
baf6683 
When check_freezer_cgroup() has non-zero, `goto err` calls `return ret`.
However, the value of `ret` has been set to 0 in the lines above and
CRIU does not handle the error.

This problem is related to checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: fix error handling for check_freezer_cgroup
ea03746 
When `check_freezer_cgroup()` has non-zero return value, `goto err` calls
`return ret`. However, the value of `ret`` has been set to `0` in the lines
above and CRIU does not handle the error properly.

This problem is related to checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: fix error handling for check_freezer_cgroup
9537341 
When `check_freezer_cgroup()` has non-zero return value, `goto err` calls
`return ret`. However, the value of `ret` has been set to `0` in the lines
above and CRIU does not handle the error properly.

This problem is related to checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: enable support for frozen containers
70b3ad9 
Container runtimes like CRI-O and containerd utilize the freezer cgroup
to create a consistent snapshot of container rootfs changes. In this
case, the container is frozen before invoking CRIU. Once CRIU
successfully completes, a copy of the container rootfs diff is saved,
and then the container is unfrozen. To enable GPU checkpointing support
with these runtimes, we need to unfreeze the cgroup and restore it to
its original state at the end.

Fixes: checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
@rst0git rst0git mentioned this issue Nov 4, 2024
Merged
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: enable support for frozen containers
511d073 
Container runtimes like CRI-O and containerd utilize the freezer cgroup
to create a consistent snapshot of container rootfs changes. In this
case, the container is frozen before invoking CRIU. Once CRIU
successfully completes, a copy of the container rootfs diff is saved,
and then the container is unfrozen. To enable GPU checkpointing support
with these runtimes, we need to unfreeze the cgroup and restore it to
its original state at the end.

Fixes: checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: fix error handling for check_freezer_cgroup
3565136 
When `check_freezer_cgroup()` has non-zero return value, `goto err` calls
`return ret`. However, the value of `ret` has been set to `0` in the lines
above and CRIU does not handle the error properly.

This problem is related to checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: fix error handling for check_freezer_cgroup
beab2cb 
When `check_freezer_cgroup()` has non-zero return value, `goto err` calls
`return ret`. However, the value of `ret` has been set to `0` in the lines
above and CRIU does not handle the error properly.

This problem is related to checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 4, 2024
@rst0git
seize: enable support for frozen containers
f8ca757 
Container runtimes like CRI-O and containerd utilize the freezer cgroup
to create a consistent snapshot of container rootfs changes. In this
case, the container is frozen before invoking CRIU. Once CRIU
successfully completes, a copy of the container rootfs diff is saved,
and then the container is unfrozen. To enable GPU checkpointing support
with these runtimes, we need to unfreeze the cgroup and restore it to
its original state at the end.

Fixes: checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
avagin pushed a commit that referenced this issue Nov 6, 2024
@rst0git @avagin
seize: fix error handling for check_freezer_cgroup
216d804 
When `check_freezer_cgroup()` has non-zero return value, `goto err` calls
`return ret`. However, the value of `ret` has been set to `0` in the lines
above and CRIU does not handle the error properly.

This problem is related to #2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 7, 2024
@rst0git
seize: enable support for frozen containers
d39fc15 
Container runtimes like CRI-O and containerd utilize the freezer cgroup
to create a consistent snapshot of container root filesystem (rootfs)
changes. In this case, the container is frozen before invoking CRIU.
After CRIU successfully completes, a copy of the container rootfs diff
is saved, and the container is then unfrozen.

However, the `cuda-checkpoint` tool is not able to perform a 'lock'
action on frozen threads.  To support GPU checkpointing with these
container runtimes, we need to unfreeze the cgroup and return it to its
original state once the checkpointing is complete.

To reflect this new behavior, the following changes are applied:
 - `dont_use_freeze_cgroup(void)` -> `set_compel_interrupt_only_mode(void)`
 - `bool freeze_cgroup_disabled` -> `bool compel_interrupt_only_mode`
 - `check_freezer_cgroup(void)` -> `prepare_freezer_for_interrupt_only_mode(void)`

Note that when `compel_interrupt_only_mode` is set to `true`,
`compel_interrupt_task()` is used instead of `freeze_processes()`
to prevent tasks from running during `criu dump`.

Fixes: checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 7, 2024
@rst0git
seize: enable support for frozen containers
979e277 
Container runtimes like CRI-O and containerd utilize the freezer cgroup
to create a consistent snapshot of container root filesystem (rootfs)
changes. In this case, the container is frozen before invoking CRIU.
After CRIU successfully completes, a copy of the container rootfs diff
is saved, and the container is then unfrozen.

However, the `cuda-checkpoint` tool is not able to perform a 'lock'
action on frozen threads.  To support GPU checkpointing with these
container runtimes, we need to unfreeze the cgroup and return it to its
original state once the checkpointing is complete.

To reflect this new behavior, the following changes are applied:
 - `dont_use_freeze_cgroup(void)` -> `set_compel_interrupt_only_mode(void)`
 - `bool freeze_cgroup_disabled` -> `bool compel_interrupt_only_mode`
 - `check_freezer_cgroup(void)` -> `prepare_freezer_for_interrupt_only_mode(void)`

Note that when `compel_interrupt_only_mode` is set to `true`,
`compel_interrupt_task()` is used instead of `freeze_processes()`
to prevent tasks from running during `criu dump`.

Fixes: checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 8, 2024
@rst0git
seize: enable support for frozen containers
c204cf2 
Container runtimes like CRI-O and containerd utilize the freezer cgroup
to create a consistent snapshot of container root filesystem (rootfs)
changes. In this case, the container is frozen before invoking CRIU.
After CRIU successfully completes, a copy of the container rootfs diff
is saved, and the container is then unfrozen.

However, the `cuda-checkpoint` tool is not able to perform a 'lock'
action on frozen threads.  To support GPU checkpointing with these
container runtimes, we need to unfreeze the cgroup and return it to its
original state once the checkpointing is complete.

To reflect this new behavior, the following changes are applied:
 - `dont_use_freeze_cgroup(void)` -> `set_compel_interrupt_only_mode(void)`
 - `bool freeze_cgroup_disabled` -> `bool compel_interrupt_only_mode`
 - `check_freezer_cgroup(void)` -> `prepare_freezer_for_interrupt_only_mode(void)`

Note that when `compel_interrupt_only_mode` is set to `true`,
`compel_interrupt_task()` is used instead of `freeze_processes()`
to prevent tasks from running during `criu dump`.

Fixes: checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 8, 2024
@rst0git
seize: enable support for frozen containers
43b5845 
Container runtimes like CRI-O and containerd utilize the freezer cgroup
to create a consistent snapshot of container root filesystem (rootfs)
changes. In this case, the container is frozen before invoking CRIU.
After CRIU successfully completes, a copy of the container rootfs diff
is saved, and the container is then unfrozen.

However, the `cuda-checkpoint` tool is not able to perform a 'lock'
action on frozen threads.  To support GPU checkpointing with these
container runtimes, we need to unfreeze the cgroup and return it to its
original state once the checkpointing is complete.

To reflect this new behavior, the following changes are applied:
 - `dont_use_freeze_cgroup(void)` -> `set_compel_interrupt_only_mode(void)`
 - `bool freeze_cgroup_disabled` -> `bool compel_interrupt_only_mode`
 - `check_freezer_cgroup(void)` -> `prepare_freezer_for_interrupt_only_mode(void)`

Note that when `compel_interrupt_only_mode` is set to `true`,
`compel_interrupt_task()` is used instead of `freeze_processes()`
to prevent tasks from running during `criu dump`.

Fixes: checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
rst0git added a commit to rst0git/criu that referenced this issue Nov 10, 2024
@rst0git
seize: enable support for frozen containers
f2418e3 
Container runtimes like CRI-O and containerd utilize the freezer cgroup
to create a consistent snapshot of container root filesystem (rootfs)
changes. In this case, the container is frozen before invoking CRIU.
After CRIU successfully completes, a copy of the container rootfs diff
is saved, and the container is then unfrozen.

However, the `cuda-checkpoint` tool is not able to perform a 'lock'
action on frozen threads.  To support GPU checkpointing with these
container runtimes, we need to unfreeze the cgroup and return it to its
original state once the checkpointing is complete.

To reflect this new behavior, the following changes are applied:
 - `dont_use_freeze_cgroup(void)` -> `set_compel_interrupt_only_mode(void)`
 - `bool freeze_cgroup_disabled` -> `bool compel_interrupt_only_mode`
 - `check_freezer_cgroup(void)` -> `prepare_freezer_for_interrupt_only_mode(void)`

Note that when `compel_interrupt_only_mode` is set to `true`,
`compel_interrupt_task()` is used instead of `freeze_processes()`
to prevent tasks from running during `criu dump`.

Fixes: checkpoint-restore#2508

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
@avagin avagin closed this as completed in f8f0e1d Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

seize: enable support for frozen containers rst0git/criu
1 participant
@rst0git