File ownership on files copied to the <app> dir are set to root when using Podman

[frayer]

Summary

Workspace files copied into the <app> directory of the Builder image have permissions set to root when pack is used in combination with Podman. When doing the same against a Docker daemon, the user:group permissions match ${CNB_USER_ID}:${CNB_GROUP_ID} of the selected Builder image.

This causes permission denied errors for buildpacks which attempt to write to the <app> directory such as paketo-buildpacks/maven since they are running as CNB_USER_ID and don't have permissions to write to the <app> directory.

---

Reproduction

Steps

I've created a minimal buildpack at https://github.com/frayer/details-buildpack to demonstrate the issue.

1. Install and configure Podman per the Building on Podman instructions.
2. Clone the repo at https://github.com/frayer/details-buildpack
3. Execute the pack build once using against Docker, and once against Podman to observe the difference

against Docker

git clone https://github.com/frayer/details-buildpack.git
cd details-buildpack
pack build test-app -B paketobuildpacks/builder:base -b ./buildpack

against Podman

export DOCKER_HOST="unix://$(podman info -f "{{.Host.RemoteSocket.Path}}")"
git clone https://github.com/frayer/details-buildpack.git
cd details-buildpack
pack build test-app -B paketobuildpacks/builder:base -b ./buildpack --docker-host=inherit

Current behavior

Look in the log output of pack build for the ---> Working directory details section.

when executed against Docker

---> Working directory details
pwd=/workspace
ls -al
total 28
drwsrwsrwt 4 cnb  cnb  4096 Jan  1  1970 .
drwxr-xr-x 1 root root 4096 Mar 20 18:38 ..
drwxrwxr-x 8 cnb  cnb  4096 Mar 20 18:08 .git
-rw-rw-r-- 1 cnb  cnb   709 Mar 20 17:40 README.md
drwxrwxr-x 3 cnb  cnb  4096 Mar 20 17:40 buildpack
-rw-rw-r-- 1 cnb  cnb    22 Mar 20 18:09 main.sh

when executed against Podman

---> Working directory details
pwd=/workspace
ls -al
total 28
drwxr-xr-x 4 root root 4096 Mar 20 18:36 .
dr-xr-xr-x 1 root root 4096 Mar 20 18:36 ..
drwxrwxr-x 8 root root 4096 Mar 20 18:07 .git
-rw-rw-r-- 1 root root  709 Mar 20 17:33 README.md
drwxr-xr-x 3 root root 4096 Mar 20 16:56 buildpack
-rw-rw-r-- 1 root root   25 Mar 20 18:10 main.sh

Expected behavior

I expect the user:group ownership on the files in /workspace to match ${CNB_USER_ID}:${CNB_GROUP_ID} of the Builder image when using Podman as they are when building against the Docker daemon. In this example, that would be cnb:cnb or 1000:1000.

---

Environment

pack info

Pack:
  Version:  0.24.0+git-79a40b7.build-3148
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.13.3

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8

Config:
  default-builder-image = "[REDACTED]"
  experimental = true

docker info

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.13.0-35-generic
 Operating System: Ubuntu 21.10
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 7.765GiB
 Name: vm-ubuntu
 ID: HVDO:NIOC:PAPJ:47QP:CQE7:K4FV:OPQX:4TE3:CBF2:327U:LARM:YORG
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

podman info

host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 6
  distribution:
    distribution: ubuntu
    version: "21.10"
  eventLogger: journald
  hostname: vm-ubuntu
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.0-35-generic
  linkmode: dynamic
  memFree: 6150553600
  memTotal: 8337092608
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.1-0ubuntu2
      spec: 1.0.2-dev
      go: go1.16.5
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.4.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 5h 14m 30.69s (Approximately 0.21 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/$USER/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/$USER/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/$USERl/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.2.1

[jromero]

@matejvasek would you have some insight or guidance here?

[hibell]

Is there any update for this issue? I am hitting the same issue on macOS when building with podman 4.1.0.

It looks like the app directory has the correct owner in the resulting image, and it is only wrong ownership during the build:

App dir owner during build

===> ANALYZING
[analyzer] Previous image with name "test" not found
===> DETECTING
[detector] examples/diag 0.0.1
===> RESTORING
===> BUILDING
[builder] ---> Building with diag buildpack
[builder] uid=1000(cnb) gid=1000(cnb) groups=1000(cnb)
[builder] drwxr-xr-x.   2 root root   22 Jun 10 12:30 workspace

Checking app dir in resulting image

$ podman run --entrypoint ls test -al /
drwxr-xr-x.   2 cnb  cnb    22 Jan  1  1980 workspace

[matejvasek]

Interesting I've seen and fixed similar issues long time ago: containers/podman#10801

[matejvasek]

When trying this on Linux with podman 4.2 it works as expected. The workspace belongs to cnb user.

[hibell]

@matejvasek Here is the verbose output forpack build on macOS + podman: diag-build.log

The diag buildpack I created prints out the owner if /workspace during the build:

[builder] Running build for buildpack  0.0.1
[builder] Creating plan directory
[builder] Preparing paths
[builder] Running build command
[builder] + echo '---> Building with diag buildpack'
[builder] ---> Building with diag buildpack
[builder] + id
[builder] uid=1000(cnb) gid=1000(cnb) groups=1000(cnb)
[builder] + grep workspace
[builder] + ls -al /
[builder] drwxr-xr-x.   2 root root   22 Jul 11 18:30 workspace
[builder] + touch /workspace/foo.txt
[builder] touch: cannot touch '/workspace/foo.txt': Permission denied
[builder] ERROR: failed to build: exit status 1
ERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 51

[matejvasek]

@frayer this was happening on Linux with podman 3.2.1, right?
What about newer versions of podman? 3.2.1 is quite old.

[matejvasek]

Is really old:
It doesn't contain
containers/podman#10905
containers/podman#10804

[matejvasek]

I recommend at least 3.3 for pack.

[matejvasek]

The guide says that v3.3+ should be used.

[frayer]

@matejvasek That is correct. It was Podman v3.2.1 on a Linux host system. Thank you for the v3.3+ call out as that addressed the issue for me. I am now seeing what I expect for file ownership when using my original steps above with Podman v4.1.0.

---> Working directory details
pwd=/workspace
ls -al
total 24
drwxrwxrwx  4 cnb  cnb  4096 Jan  1  1970 .
dr-xr-xr-x 27 root root 4096 Jul 12 23:17 ..
drwxrwxr-x  8 cnb  cnb  4096 Jul 12 23:11 .git
-rw-rw-r--  1 cnb  cnb   709 Jul 12 17:20 README.md
drwxrwxr-x  3 cnb  cnb  4096 Jul 12 17:20 buildpack
-rw-rw-r--  1 cnb  cnb    22 Jul 12 17:20 main.sh

I'm on an Ubuntu 21.10 distro which doesn't appear to have newer versions of podman in the registries yet.

Distributor ID: Ubuntu
Description:    Ubuntu 21.10
Release:        21.10
Codename:       impish

For anyone in a similar situation, you can build podman from source or there is a binary distribution here. Use the later at your own risk as it's not an official distribution.

Unfortunately I don't think this addresses what @hibell is seeing on macOS, but my original issue is resolved.

[natalieparellano]

Closing as this seems resolved, but please re-open if not

[matejvasek]

This should not be issue for trusetd builders. However this is still an issue if build in unstruster builder mode. See: containers/podman#19652

[natalieparellano]

Thanks @matejvasek - should this issue be blocked on containers/podman#19652?

[wdonne]

I have a strange situation that could be related. My pack build runs on Jenkins with podman 4.4.1. The build works fine and creates the target directory in the workspace. However, when it tries to remove the source files it gets a permission denied. So, Maven has the permission to write, but the layer creator does not.

[natalieparellano]

Closing as I believe this is fixed in pack 0.34 with #2129 (but please reopen if not)