Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Cannot build a tar layer on top of a squashfs layer #450

Closed
smoser opened this issue Mar 31, 2023 · 3 comments · Fixed by #505
Closed

Bug: Cannot build a tar layer on top of a squashfs layer #450

smoser opened this issue Mar 31, 2023 · 3 comments · Fixed by #505
Assignees
Labels

Comments

@smoser
Copy link
Contributor

smoser commented Mar 31, 2023

stacker version

v1.0.0-rc4-8e267fc

Describe the bug

Building the following stacker.yaml as root or non-root will fail with the stacktrace below.
The problem is with trying to build a tar layer from a source layer that is only published as squashfs.
You can avoid the problem by using --layer-type=squashfs

stacker.yaml:

base:
  from:
    type: docker
    url: "${{ZOTHUB_BASE:docker://zothub.io}}/machine/bootkit/rootfs:0.0.5.230327-squashfs"
  run: |
    echo hello world

To reproduce

$ stacker --debug build
stacker version v1.0.0-rc4-8e267fc
usernsexec-ing [u 0 1000 1 1 100001 65535 g 0 1000 1 1 100001 65535 -- /usr/local/bin/stacker --internal-userns --debug build]
stacker version v1.0.0-rc4-8e267fc
no previous storage type detected
initializing stacker recipe: stacker.yaml
substituting $STACKER_ROOTFS_DIR to /tmp/roots
substituting $STACKER_STACKER_DIR to /tmp/.stacker
substituting $STACKER_OCI_DIR to /tmp/oci
substituting $STACKER_WORK_DIR to 
stacker build order:
0 build /tmp/stacker.yaml: requires: []
building: 0 /tmp/stacker.yaml
substituting $STACKER_ROOTFS_DIR to /tmp/roots
substituting $STACKER_STACKER_DIR to /tmp/.stacker
substituting $STACKER_OCI_DIR to /tmp/oci
substituting $STACKER_WORK_DIR to 
Dependency Order [base]
preparing image base...
overlay-dirs, possibly modified after import: []
loading docker://zothub.io/machine/bootkit/rootfs:0.0.5.230327-squashfs
Copying blob c2c670f1c1af done  
Copying blob bcd9b263edd3 done  
Copying blob 25c2c00faedd done  
Copying config 3bd19ef6e2 done  
Writing manifest to image destination
Storing signatures
unpacking to /tmp/roots/base
maybeKernelSquashMount(/tmp/.stacker/layer-bases/oci/blobs/sha256/25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d) exited 32: mount: /tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay: failed to setup loop device for /tmp/.stacker/layer-bases/oci/blobs/sha256/25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d.
maybeKernelSquashMount(/tmp/.stacker/layer-bases/oci/blobs/sha256/c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9) exited 32: mount: /tmp/roots/sha256_c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9/overlay: failed to setup loop device for /tmp/.stacker/layer-bases/oci/blobs/sha256/c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9.
Extracting /tmp/.stacker/layer-bases/oci/blobs/sha256/25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d -> /tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay with /usr/bin/squashfuse_ll [/tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/.overlay-squashfuse.log]
maybeKernelSquashMount(/tmp/.stacker/layer-bases/oci/blobs/sha256/bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad) exited 32: mount: /tmp/roots/sha256_bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad/overlay: failed to setup loop device for /tmp/.stacker/layer-bases/oci/blobs/sha256/bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad.
Extracting /tmp/.stacker/layer-bases/oci/blobs/sha256/c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9 -> /tmp/roots/sha256_c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9/overlay with /usr/bin/squashfuse_ll [/tmp/roots/sha256_c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9/.overlay-squashfuse.log]
Extracting /tmp/.stacker/layer-bases/oci/blobs/sha256/bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad -> /tmp/roots/sha256_bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad/overlay with /usr/bin/squashfuse_ll [/tmp/roots/sha256_bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad/.overlay-squashfuse.log]
lxc rootfs overlay arg overlayfs:/tmp/roots/sha256_c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9/overlay:/tmp/roots/sha256_bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad/overlay:/tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay:/tmp/roots/base/overlay
stacker version v1.0.0-rc4-8e267fc
stacker subcommand: [/usr/local/bin/stacker --oci-dir /tmp/oci --roots-dir /tmp/roots --stacker-dir /tmp/.stacker --storage-type overlay --internal-userns --debug internal-go check-aa-profile lxc-container-default-cgns]
bind mounting /tmp/.stacker/imports/base into container
+ echo hello world
hello world
converting between {squashfs true} and {tar false}
error: chmod /tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay: function not implemented
chmod +r
github.com/opencontainers/umoci/pkg/unpriv.Open.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:134
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Open
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:125
github.com/opencontainers/umoci/pkg/unpriv.foreachSubpath
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:318
github.com/opencontainers/umoci/pkg/unpriv.walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:541
github.com/opencontainers/umoci/pkg/unpriv.Walk.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:574
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:569
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
unpriv.open
github.com/opencontainers/umoci/pkg/unpriv.Open
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:142
github.com/opencontainers/umoci/pkg/unpriv.foreachSubpath
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:318
github.com/opencontainers/umoci/pkg/unpriv.walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:541
github.com/opencontainers/umoci/pkg/unpriv.Walk.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:574
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:569
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
github.com/opencontainers/umoci/pkg/unpriv.foreachSubpath
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:320
github.com/opencontainers/umoci/pkg/unpriv.walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:541
github.com/opencontainers/umoci/pkg/unpriv.Walk.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:574
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:569
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
unpriv.walk
github.com/opencontainers/umoci/pkg/unpriv.Walk.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:579
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:569
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
generate insert layer
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1.1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:140
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
copy to temporary blob
github.com/opencontainers/umoci/oci/cas/dir.(*dirEngine).PutBlob
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/cas/dir/dir.go:173
stackerbuild.io/stacker/pkg/overlay.ociPutBlob
	/stacker-tree/pkg/overlay/pack.go:301
stackerbuild.io/stacker/pkg/overlay.ConvertAndOutput
	/stacker-tree/pkg/overlay/pack.go:152
stackerbuild.io/stacker/pkg/overlay.(*overlay).initializeBasesInOutput
	/stacker-tree/pkg/overlay/pack.go:236
stackerbuild.io/stacker/pkg/overlay.(*overlay).Repack
	/stacker-tree/pkg/overlay/pack.go:266
stackerbuild.io/stacker/pkg/stacker.(*Builder).build
	/stacker-tree/pkg/stacker/build.go:493
stackerbuild.io/stacker/pkg/stacker.(*Builder).BuildMultiple
	/stacker-tree/pkg/stacker/build.go:568
main.doBuild
	/stacker-tree/cmd/stacker/build.go:117
github.com/urfave/cli.HandleAction
	/stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.12/app.go:524
github.com/urfave/cli.Command.Run
	/stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.12/command.go:175
github.com/urfave/cli.(*App).Run
	/stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.12/app.go:277
main.main
	/stacker-tree/cmd/stacker/main.go:324
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
error: exit status 1
stackerbuild.io/stacker/pkg/container.MaybeRunInNamespace
	/stacker-tree/pkg/container/userns.go:102
main.main.func3
	/stacker-tree/cmd/stacker/main.go:319
github.com/urfave/cli.(*App).Run
	/stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.12/app.go:264
main.main
	/stacker-tree/cmd/stacker/main.go:324
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598

Expected behavior

No response

Additional context

No response

@hallyn
Copy link
Contributor

hallyn commented Sep 12, 2023

Honestly this makes no sense:

ociPutBlob: called on types.StackerConfig{WorkDir:"", StackerDir:"/home/serge/sandbox/smoser4/.stacker", OCIDir:"/home/serge/sandbox/smoser4/oci", RootFSDir:"/home/serge/sandbox/smoser4/roots", Debug:false, StorageType:"overlay", EmbeddedFS:embed.FS{files:(*[]embed.file)(0x3482da0)}}
ociPutBlob: opened layout "/home/serge/sandbox/smoser4/oci"
ociPutBlob: calling putblob on &io.PipeReader{p:(*io.pipe)(0xc000701f80)}
error: copy to temporary blob: generate layer: unpriv.walk: unpriv.open: chmod +r: chmod /home/serge/sandbox/smoser4/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay: function not implemented

We are opening the ${topdir}/oci as ocidir in umoci, then calling oci.PubBlob(), but it ends up trying to chown ${RootfsDir}/sha256_{blob}

@hallyn
Copy link
Contributor

hallyn commented Sep 13, 2023

Ok, yeah, so the problem is in pkg/overlay/pack.go:generateBlob(), we take in the target layerType. If we are building a tar layer, then we call umoci/oci/layer.GenerateInsertLayer(), if squash then mksquashfs. But if we are building a tar layer from a squashfs source layer, then the overlay/ dir where we start is a mountpoint. Now umoci tries to chmod the parent dir +r, which fails bc it's a mountpoint.

This only happens when we are re-building an intermediate layer - one which we already have as squash layer, but are now generating a tar layer for.

Easiest is probably to detect the situation (using IsMountpoint()) and then convert slightly differently.

hallyn added a commit to hallyn/umoci that referenced this issue Sep 13, 2023
If the inode is a mountpoint, chmod may just fail.

(See project-stacker/stacker#450)

Signed-off-by: Serge Hallyn <serge@hallyn.com>
@hallyn
Copy link
Contributor

hallyn commented Sep 13, 2023

Actually hallyn/umoci@922fdb3 fixes it for me

hallyn added a commit to hallyn/umoci that referenced this issue Sep 13, 2023
If the inode is a mountpoint, chmod may just fail.

(See project-stacker/stacker#450)

Signed-off-by: Serge Hallyn <serge@hallyn.com>
hallyn added a commit to hallyn/umoci that referenced this issue Sep 13, 2023
If the inode is a mountpoint, chmod may just fail.

(See project-stacker/stacker#450)

Signed-off-by: Serge Hallyn <serge@hallyn.com>
hallyn added a commit to hallyn/umoci that referenced this issue Sep 13, 2023
If the inode is a mountpoint, chmod may just fail.

(See project-stacker/stacker#450)

Signed-off-by: Serge Hallyn <serge@hallyn.com>
(cherry picked from commit ebe120e)
hallyn added a commit to hallyn/stacker-1 that referenced this issue Sep 13, 2023
There is a PR against umoci#main to avoid trying to chmod +r when we
don't need to.  This will avoid trying to chmod +r on a mountpoint, which
is the root cause of project-stacker#450.

While we can and should also forward port our umoci dep to point at
current main, we should not do that as part of this PR, as that will
risk confusion if there are regressions from just the fwd port.

Closes project-stacker#450

Signed-off-by: Serge Hallyn <serge@hallyn.com>
hallyn added a commit to hallyn/stacker-1 that referenced this issue Sep 13, 2023
There is a PR against umoci#main to avoid trying to chmod +r when we
don't need to.  This will avoid trying to chmod +r on a mountpoint, which
is the root cause of project-stacker#450.

While we can and should also forward port our umoci dep to point at
current main, we should not do that as part of this PR, as that will
risk confusion if there are regressions from just the fwd port.

Closes project-stacker#450

Signed-off-by: Serge Hallyn <serge@hallyn.com>
rchincha pushed a commit that referenced this issue Sep 13, 2023
There is a PR against umoci#main to avoid trying to chmod +r when we
don't need to.  This will avoid trying to chmod +r on a mountpoint, which
is the root cause of #450.

While we can and should also forward port our umoci dep to point at
current main, we should not do that as part of this PR, as that will
risk confusion if there are regressions from just the fwd port.

Closes #450

Signed-off-by: Serge Hallyn <serge@hallyn.com>
hallyn added a commit to hallyn/umoci that referenced this issue Sep 19, 2023
If the inode is a mountpoint, chmod may just fail.

(See project-stacker/stacker#450)

Signed-off-by: Serge Hallyn <serge@hallyn.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants