distroless/static:nonroot with runc v1.0.0-rc93 - chdir to cwd: permission denied

[hasheddan]

This issue is mostly informational, and is meant to potentially facilitate conversation, but does not necessarily require any changes.

You can find a complete write-up and testing in crossplane/crossplane#2245.

Some folks have recently been running into issues with images using the distroless nonroot image as base, where they observe the following error:

level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"

To provide some background, gcr.io/distroless/static:nonroot image sets user as nonroot (uid=65532) and working directory to /home/nonroot, which has 0700 permissions. This means that only the nonroot user can chdir to this directory because the group and other permissions do not have execute bit. While it is expected that consumers of images built on the nonroot image run as the nonroot user, there are some environments, such as Openshift, which will set a random uid within a range for containers. This has not caused issues in the past because runc changed to the specified working directory for the image before it setup the container user. However, a recent change that was included in the v1.0.0-rc93 release switched to first setting up the container user, then changing to the working directory. This means that if a random uid was assigned, the container user likely does not have permissions to change to the working directory, causing the error mentioned above. As various hosted Kubernetes offerings have updated to the newest version of runc, folks have begun to see this more and more.

The error was prevalent enough that a workaround has now been merged, and will be present in the next release. However, it is likely that a number of platforms will be using v1.0.0-rc93 for the near term, requiring some projects to update their base image to accommodate for these scenarios.

Whether a change should be made to the nonroot image is mostly a matter of philosophy of the image purpose: Is the intention to force users to run as nonroot? If so, then it seems no changes are required. However, if the intention is rather to just default to users running as nonroot, then it may make sense to make the default permissions on /home/nonroot to 0755, or change the default working directory to /. I tend to lean towards no change as folks can accomplish the workaround by using the plain distroless/static image and set uid to 65532 in the top layer of their image build if they want to allow for running with other users, but default to running as nonroot.

Anyway, I wanted to share this write-up with folks in an area where they may come searching for it. It should also be helpful from a documentation perspective for just understanding how the distroless images are built. Let me know if there is anything else I can do to be of help here :)

[jonjohnsonjr]

@hasheddan I've edited the issue body to fix the recent change link, just FYI.

[jphx]

I have a scenario that is having difficulty because the permission bits on the /home/nonroot directory are 700 instead of 755 or even 775. Specifically, if my program needs to read a configuration file or something from the current directory, it doesn't work on OpenShift with the existing permissions. And since the Dockerfile spec doesn't give me a way to change the permission, difficulties ensue.

My Dockerfile is:

FROM golang:latest as builder

WORKDIR /workspace

COPY go.mod go.mod
COPY go.sum go.sum
RUN go mod download
COPY cmd/ cmd/
COPY pkg/ pkg/

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on \
    go build ./cmd/debug-http-server

FROM gcr.io/distroless/static:nonroot

EXPOSE 8443

COPY --from=builder --chown=nonroot:nonroot /workspace/debug-http-server ./
COPY --chown=nonroot:nonroot certificate.pem key.pem ./

CMD ["./debug-http-server"]

The resulting permission of the relevant files are:

drwxr-xr-x 65532/65532        0 1969-12-31 19:00 home/
drwx------ 65532/65532        0 2022-11-10 11:23 home/nonroot/
-rw-rw-r-- 65532/65532     1448 2020-01-10 14:04 home/nonroot/certificate.pem
-rwxr-xr-x 65532/65532 12645263 2022-11-10 11:23 home/nonroot/debug-http-server
-rw-rw-r-- 65532/65532     1704 2020-01-10 14:03 home/nonroot/key.pem

When it runs under OpenShift, however, using the restricted SecurityContextConstraint, it runs with a random uid and a gid of 0, neither of which has permission to read the files under /home/nonroot, so the program fails at runtime with an error trying to read the files.

But if the home directory had permissions of 755, the program would be able to read the files.