Running in Docker fails with "must be owned by the user identifier (uid=1000) or root"

[slhck]

I'm using APM Server 7.5.0 within a Docker Compose environment.

The YAML file looks basically like this; it bind-mounts the configuration into the container, so that it does not have to be baked into the image when building. This is common practice, and it works for Elasticsearch, Kibana, and Logstash without issues.

version: "3.5"

services:
  apm-server:
    image: docker.elastic.co/apm/apm-server:7.5.0
    volumes:
      - type: bind
        source: ./config/apm-server.yml
        target: /usr/share/apm-server/apm-server.yml
        read_only: true

When launching the container, it immediately exits with:

Exiting: error loading config file: config file ("apm-server.yml") must be owned by the user identifier (uid=1000) or root

The issue seems to be that the file is mounted by the user calling Docker Compose, which is not always the user having ID 1000.

➜ docker-compose run apm-server ls -lha
total 63M
drwxr-x--- 1 root apm-server 4.0K Nov 25 23:11 .
drwxr-xr-x 1 root root       4.0K Nov 25 23:11 ..
-rw-r----- 1 root apm-server   41 Nov 25 23:11 .build_hash.txt
-rw-r----- 1 root apm-server  14K Nov 25 23:11 LICENSE.txt
-rw-r----- 1 root apm-server 212K Nov 25 23:11 NOTICE.txt
-rw-r----- 1 root apm-server  660 Nov 25 23:11 README.md
-rwxr-x--- 1 root apm-server  63M Nov 25 23:11 apm-server
-rw-r--r-- 1 1001       1003  605 Dec 11 09:12 apm-server.yml
drwxrwx--- 2 root apm-server 4.0K Nov 25 23:11 data
-rw-r----- 1 root apm-server 115K Nov 25 23:11 fields.yml
drwxr-x--- 1 root apm-server 4.0K Nov 25 23:11 ingest
drwxr-x--- 1 root apm-server 4.0K Nov 25 23:11 kibana
drwxrwx--- 2 root apm-server 4.0K Nov 25 23:11 logs

Since it is not possible to change the user with whose permissions the file is mounted, the only solution would be to:

1. Disable the requirement for the file owner, possibly via a CLI option to the apm-server command
2. Fix the Dockerfile so that a proper user for the apm-server is created

I see that the Elasticsearch Dockerfile adds its own user, as is recommended, whereas the apm-server Dockerfile does not. Could this be the cause?

[slhck]

Note that if I wrote my own Dockerfile and did:

COPY --chown=0:1000 ./config/apm-server.yml /usr/share/apm-server/apm-server.yml

the server would run, but this still does not allow bind-mount-type configs as you'd expect from Elastic, Kibana or Logstash.

[graphaelli]

I'm unable to reproduce this behavior - What are the OS of the host system and versions of docker cilent and server?

bind mounted apm-server.yml on my machine gets actually gets uid 1000 (apm-server) and gid 1000, as opposed to the original values of 0:1000 - my uid outside of docker is 501.

Also what does docker inspect of the running container report about the mount? Mine says:

"Mounts": [
    {
        "Type": "bind",
        "Source": "/go/src/github.com/elastic/apm-server/apm-server.yml",
        "Destination": "/usr/share/apm-server/apm-server.yml",
        "Mode": "",
        "RW": false,
        "Propagation": "rprivate"
    }
]

[slhck]

Hmm. I'm using Docker 18.09.7, build 2d0083d on Ubuntu 18.04, with Docker Compose 1.25.0.

The uid of my currently logged in user is 1001. The mounts show:

        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/www/analytics/apm-server/config/apm-server.yml",
                "Destination": "/usr/share/apm-server/apm-server.yml",
                "Mode": "",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],

I have another user on the server with uid 1000, but the problem is the same when creating the container with that one.

I also don't see how a user apm-server can exist when it is not explicitly created anywhere?

[slhck]

I'm guessing your host system is macOS since your default user gets uid 501?

I can confirm that it works under macOS, where I get:

➜ docker-compose run --rm apm-server ls -lha
total 63M
drwxr-x--- 1 root       apm-server 4.0K Nov 25 23:11 .
drwxr-xr-x 1 root       root       4.0K Nov 25 23:11 ..
-rw-r----- 1 root       apm-server   41 Nov 25 23:11 .build_hash.txt
-rw-r----- 1 root       apm-server  14K Nov 25 23:11 LICENSE.txt
-rw-r----- 1 root       apm-server 212K Nov 25 23:11 NOTICE.txt
-rw-r----- 1 root       apm-server  660 Nov 25 23:11 README.md
-rwxr-x--- 1 root       apm-server  63M Nov 25 23:11 apm-server
-rw-r--r-- 1 apm-server apm-server  605 Dec 10 19:22 apm-server.yml
drwxrwx--- 2 root       apm-server 4.0K Nov 25 23:11 data
-rw-r----- 1 root       apm-server 115K Nov 25 23:11 fields.yml
drwxr-x--- 1 root       apm-server 4.0K Nov 25 23:11 ingest
drwxr-x--- 1 root       apm-server 4.0K Nov 25 23:11 kibana
drwxrwx--- 2 root       apm-server 4.0K Nov 25 23:11 logs

[graphaelli]

I also don't see how a user apm-server can exist when it is not explicitly created anywhere?

It's created here. The Dockerfile you linked earlier is actually only used by the testsuite, we should really make that more clear.

I can confirm that it works under macOS

Yeah, my test host was macOS. Trying ubuntu 18.04 now.

[slhck]

I see, thanks!

I have a hunch. The project was initially cloned as root, later chowned to the user with the other ID. It might be an ACL issue? I can check tomorrow.

[graphaelli]

Confirmed the behavior you're seeing on ubuntu. Two options to get this working immediately:

1. set file perms on the host to group 0:1000 or 1000:1000
2. disable strict perms (docs), eg:

services:
  apm-server:
    command: --strict.perms=false
    image: docker.elastic.co/apm/apm-server:7.5.0
    volumes:
      - ./apm-server.yml:/usr/share/apm-server/apm-server.yml

I'll leave this open while we investigate other options.

[slhck]

Thanks for the documentation link; I didn't spot this earlier as I didn't have to do this while setting up the rest of the ELK stack (with user-mounted configs).

Indeed the documentation actually provides another workaround for building the image with the config embedded, similar to what I proposed above:

FROM docker.elastic.co/apm/apm-server:7.5.0
COPY apm-server.yml /usr/share/apm-server/apm-server.yml
USER root
RUN chown root:apm-server /usr/share/apm-server/apm-server.yml
USER apm-server

[graphaelli]

didn't spot this earlier as I didn't have to do this while setting up the rest of the ELK stack (with user-mounted configs).

Sure, that make sense. We should consider including a pointer to strict.perms in the error message while running in a container.

Closing this out now as a solution exists and we haven't tracked down another option. Thanks @slhck

[slhck]

I think the strict.perms error message gets shown when the perms are group/others-writable, so that'd make sense. Thanks!