OpenShift compatibility

[eed3si9n]

OpenShift is a flavor of Kubernetes maintained by Red Hat, which has bit more tighter security defaults and requires some unique settings, described in Guidelines.

It would be good if sbt-native-packager could generate Docker container that is compatible with both vanilla Kubernetes and OpenShift for daemonUser.

current

FROM openjdk:8
WORKDIR /opt/docker
ADD --chown=daemon:daemon opt /opt
USER daemon
ENTRYPOINT ["/opt/docker/bin/hello"]
CMD []

OpenShift compatible Dockerfile

The following Dockerfile implements non-root container, but with OpenShift compatibility:

FROM openjdk:8
WORKDIR /opt/docker
RUN useradd --system --create-home --uid 1001 --gid 0 nonroot
ADD opt /opt
RUN chgrp -R 0 /opt && chmod -R g=u /opt
USER 1001
ENTRYPOINT ENTRYPOINT ["/opt/docker/bin/hello"]

[seglo]

This would be enormously helpful. With sbt-native-packager today you have to override much of the Dockerfile generation to build compatible images for OpenShift.

[muuki88]

The implementation should be straightforward. Adding a new archetype plugin (like the AshScriptPlugin) that depends on the DockerPlugin and replaces / removes the incomaptible commands.

[eed3si9n]

@muuki88 Given that sbt-native-packager already implements non-root by default (which is awesome btw), what do you think about making the implementation OpenShift compatible by default too?
Most people probably aren't concerned about this detail, but having this enabled means that the Docker images created this way can be consumed easier by OpenShift users. (See for example https://medium.com/bitnami-perspectives/non-root-containers-to-show-openshift-some-love-3b32d7218ac6)

We can provide global setting if people want to opt-out for some reason:

ThisBuild / dockerOpenShiftCompatible := false

[jw3]

Beware of a regression of #883 here.

[muuki88]

Thanks for clarifying why I made this suggestion @jw3 ❤️

#883 is the reason I would favor a separate configuration plugin that can be added if necessary. The open shift compatible mode will add this additional layer we removed with #883. Unless docker supports chgrp as well in the ADD command, e.g. something like

ADD --chgrp -R 0 --chown -R g=u /opt

So the open shift AutoPlugin will remove these enhancements if I understood everything correctly so far.

[jw3]

An ADD --chown=daemon:root will set your ownership correctly.

But the chmod to g+rwX for root group will add the extra layer, and no there is no existing docker support (afaik) on ADD or COPY for a chmod.

An approach I use for OpenShift images from CI is to

• docker:stage with native packager
• chmod on the host filesystem
• docker build with the native packager produced Dockerfile

So if that on-host chmod could be integrated into the plugin you might work around #883.

[eed3si9n]

I added chmod to sbt/io at some point, but I think it only works for macOS and Linux - sbt/io#76

[TimMoore]

Would it be possible to use a multi-stage build to prepare the image? Rather than chmod on the host filesystem (which has some host OS issues) you would do it in the builder image.

For example http://www.garbers.co.za/2017/11/15/reduce-docker-image-size-chmod-chown/

[muuki88]

An ADD --chown=daemon:root will set your ownership correctly.

This should be possible today with daemonUser in Docker := "root" 👍

An approach I use for OpenShift images from CI is to

• docker:stage with native packager
• chmod on the host filesystem
• docker build with the native packager produced Dockerfile

So if that on-host chmod could be integrated into the plugin you might work around #883.

We something similar in the DebianPlugin.

I'm fine with putting this extra logic into a separate OpenShiftDockerPlugin that overrides the necessary tasks to perform these special steps and keep the DockerPlugin simple.

@TimMoore I haven't tried the multi-stage feature yet, but the example link you shared looks very much like what we need 😎 The big question is: does the open shift docker version support multi stage builds ? If so, then a OpenShiftDockerPlugin wouldn't need custom chown / chmod logic, which would be very neat.

[jw3]

@TimMoore yes great idea. Ive used multi-stage build a few times with good results.

does the open shift docker version support multi stage builds

Only the build host has to support it, the execution environment wont know the difference.

[eed3si9n]

@jroper reminded me something important.

Why do we need any files in any image to be owned by any particular user? Making an applications own deployment files writable by the same user that runs the application is a security vulnerability just waiting to happen.

I blindly translated existing ADD --chown=daemon:daemon opt /opt to OpenShift style implementation, but it's actually worth questioning whether /opt should be writable at all.

In fact the OpenShift guideline says:

For an image to support running as an arbitrary user, directories and files that may be written to by processes in the image should be owned by the root group and be read/writable by that group.

In other words, OpenShift defaults to secure read-only files and directories (assuming most file mores are r-- for o?). So we should be more cautious to opening this thing up with chown or chgrp, undoing non-root. Maybe we can add settings to express:

1. If we want /opt to be writable. (default to false)
2. Additional directories that are set to be writable.

[jw3]

So we should be more cautious to opening this thing up with chown or chgrp, undoing non-root.

Ah, yes please dont let my mention of g+rwX muddy the water. I didnt realize they had updated their docs regarding that practice from the original text. The g=u is much saner.

Maybe we can add settings to express:

1. If we want /opt to be writable. (default to false)
2. Additional directories that are set to be writable.

IMO you could add settings for both and be OK. (2) would probably end up being very helpful.

Comments on the second Dockerfile from the first post.

1. eliminate the RUN useradd, it will be fine to stick with the daemon user
2. let the USER command remain on daemon

There is no need to tweak the user from what native packager currently sets, OpenShift is going to run it on a arbitrary user anyhow (eg. 1000340000).

The big question is: does the open shift docker version support multi stage builds ?

Reading this again I see your point, and no doesnt look like it would be supported, OpenShift is at docker version 1.13, where multistage was in 17.05. And I guess the ADD --chown is likewise off limits.

So perhaps the following, other than it will still regress #883, but should let the image work securely and smoothly in or outside of OpenShift.

FROM openjdk:8
WORKDIR /opt/docker
ADD opt /opt
RUN chgrp -R 0 /opt \
 && chmod -R g=u /opt
USER daemon
ENTRYPOINT ENTRYPOINT ["/opt/docker/bin/hello"]

Then it sounds like to avoid regression of #883 you would have to decide

1. Have windows support, using multistage and add --chown but not be able to build on a docker daemon within OpenShift
2. Lose windows support by doing the chgrp and chmod on the host with sbt/io, but be able to build within OpenShift.

Being primarily an S2I image user I think (1) is the way to go. I dont know what the common practice is out there, maybe people do build images on Jenkins deployed in OpenShift or something.

Also, an outdated blast from the past for reference; https://github.com/jw3/sbt-openshift. Abandoned it for the leaner approach I mentioned earlier. I would have done a thing or two differently today (leave daemon user) but it worked well for most of a 2017 work project.

[eed3si9n]

OpenShift compatible Dockerfile (read-only /opt)

I haven't tested it but the following might outline the spirit of OpenShift default:

FROM openjdk:8
WORKDIR /opt/docker
RUN useradd --system --create-home --uid 1001 --gid 0 daemon
ADD opt /opt
RUN chgrp -R 0 /opt/docker/bin/hello \
  && chmod -R g+x /opt/docker/bin/hello
USER 1001
ENTRYPOINT ["/opt/docker/bin/hello"]

numeric UID

Guideline says:

Lastly, the final USER declaration in the Dockerfile should specify the user ID (numeric value) and not the user name.

writable /opt

dockerWritablePaths += "/opt"

Maybe the tradeoff to get /opt writable is you regress on #883.

[jw3]

RUN useradd --system --create-home --uid 1001 --gid 0 daemon

If you add this to the plugin and a user wants to set a base image that already has 1001, eg. any other OpenShift ready image, its going to fail the build.

Perhaps a configuration option openshiftCreateUser?

Maybe the tradeoff to get /opt writable is you regress on #883.

You need at a minimum a chgrp 0 somewhere, and if you do that in the docker build it will cost you the duplicated layer from #883, regardless of write status of /opt.

If files are staged with group rx and you dont need /opt writable then an ADD --chown should suffice. Perhaps to keep this simple the use of the --chown is the configuration point? If you dont have --chown or want writable /opt, then you have a larger image? It could of course be optimized with the approaches discussed earlier.

Created a repo that illustrates the approach I mentioned earlier, and fact checks some thoughts I shared https://github.com/jw3/example-openshift-snp-tests. Multistage looks like requirement if wanting to avoid #883

[eed3si9n]

If you add this to the plugin and a user wants to set a base image that already has 1001, eg. any other OpenShift ready image, its going to fail the build.

The defense against that would be id:

RUN id -u daemon || useradd --system --create-home --uid 1001 --gid 0 daemon

You need at a minimum a chgrp 0 somewhere, and if you do that in the docker build it will cost you the duplicated layer from #883, regardless of write status of /opt.

Docker reference on ADD says:

All new files and directories are created with a UID and GID of 0, unless the optional --chown flag specifies ...

source of incompatibility

For the record, here's what I think is happening.

Docker reference on USER says:

Warning: When the user doesn’t have a primary group then the image (or the next instructions) will be run with the root group.

So USER daemon effectively is same as USER daemon:root. On OpenShift it's replaced with USER random_user:root.

This is all ok as long as we stick to plain ADD. But calling ADD --chown=daemon:daemon opt /opt makes:

1. user bits available for non-OpenShift, but still can't be used by OpenShift's random_user.
2. group bits unavailable for both non-OpenShift and OpenShift.
3. other bit becomes the only recourse for OpenShift.

ensuring read-only-ness

Suppose we drop --chown daemon:daemon from ADD so we can all use group file mode bits, there's still the issue of ensuring that it's either read-only for writable since macOS seems to use r-- but on Ubuntu machine I see rw-.

So that would require calling chmod -R g=rX somewhere (another layer, host machine, or multi stage) for read-only, and similarly chmod -R g=rwX for writable directory.

[jw3]

@eed3si9n That all sound right to me, and clearly expressed :) Two things

1. I had mentioned ADD --chown=daemon:root opt /opt above. What are your thoughts on that?
2. Ive found during some testing on an ubuntu machine that a regular user cannot do a chmod g=0. So this pushes further towards using multi stage imo.
   • The on-host chmod is done on a CI machine, and it was running as root. Didnt recall that until digging in.

[eed3si9n]

1. ADD --chown=daemon:root opt /opt I think is still problematic since it depends on Docker server version, which is not available on older Minikube or current OpenShift as you mentioned. Also user bits are still unavailable to OpenShift, so that becomes a source of "works on my machine".
2. Maybe this strategy is the part that needs to be expressed as some setting:
   • do nothing
   • RUN and pay Docker Image 'chown -R daemon:daemon' duplicates application layer #883 overhead
   • run it on host (requires macOS or root on Ubuntu?)
   • multi-stage (requires Docker 17.05)

[jw3]

Concur.

I really like multi-stage to solve both issues. Perhaps can keep it simple and provide a flag to enable it.

• if you can enable it you save the cost of the extra layer
• if you dont enable it you have backwards compatibility

A minimal multi-stage for this would look something like the following.

FROM openjdk:8
WORKDIR /opt/docker
RUN id -u default || useradd --system --create-home --uid 1001 --gid 0 default
ADD opt /opt
RUN chmod -R g=u /opt

FROM openjdk:8
RUN id -u default || useradd --system --create-home --uid 1001 --gid 0 default
COPY --from=0 --chown=1001:0 /opt/docker /opt/docker
USER 1001

Changed from daemon user because it was conflicting with user id 1 named daemon that already exists. Using default because that is what OpenShift uses in their images for 1001.

I tested this locally and in OpenShift and it runs as expected

• locally under user 1001
• in OpenShift user id 1000340000

[dwickern]

My understanding is that images built on a newer docker version will still run on an older version. So we could use multi-stage builds as long as the build-time daemon is up-to-date, which seems like a reasonable requirement.

[jw3]

@dwickern this is my experience too. I assumed the constraint was for when building images from OpenShift via an exposed docker socket.

[eed3si9n]

Sent #1190 as a proposal to implement dockerPermissionStrategy, defaulting to multi-stage build.