Unable to create directory for account (dokku in docker) #235

BilliAlpha · 2021-05-28T15:54:20Z

Description of problem

Letsencrypt plugin seems unable to create certificates for apps when running from within docker. This seems to be an issue because the running user for dokku inside the container is always dokku (uid: 200) and never root (even when launched as root).

This leads to creating the lego container with the dokku user id (200) which can't create folders on / because it is owned by root.

How reproducible

Everytime

Steps to Reproduce

Install dokku using docker (https://dokku.com/docs/getting-started/install/docker/)
Install letsencrypt plugin
Try to enable letsencrypt for an app, be faced with the same error

Actual Results

# dokku letsencrypt:enable drive
=====> Enabling letsencrypt for drive
-----> Enabling ACME proxy for drive...
       ok: run: nginx: (pid 62) 11602s
-----> Getting letsencrypt certificate for drive...
        - Domain 'drive.dokku.me'
       DEBUG: docker run --rm --user 200:200 -p '61729:61729' -v '/home/dokku/drive/letsencrypt/certs/277c26e8d3c98526567f723718d821b5d11ebde3:/certs' 'goacme/lego:v4.3.1' '--pem --http --http.port :61729 --accept-tos --cert.timeout 2592000 --path /certs --server https://acme-v02.api.letsencrypt.org/directory --email <censored> --domains drive.dokku.me' run
       2021/05/28 15:36:35 No key found for account <censored>. Generating a P256 key.
       2021/05/28 15:36:35 Could not check/create directory for account <censored>: mkdir /certs/accounts: permission denied
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for drive...
       ok: run: nginx: (pid 62) 11604s
 !     Failed to setup letsencrypt
 !     Check log output for further information on failure

I added debug to see what was the command used to launch the lego container

Expected Results

The letsencrypt plugin successfully generates a certificate for the selected app

Environment Information

`dokku report` output

https://gist.github.com/BilliAlpha/5af332f4f23d1e3fb815ecb797b26bfe

How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

Installed using docker on a VPS (Debian)

The text was updated successfully, but these errors were encountered:

BilliAlpha · 2021-06-14T09:29:01Z

I proceeded to generate certificates manually following the process of this plugin and discovered there is another issue with the usage in a docker install, the letsencrypt configuration for nginx uses 127.0.0.1 as the address of the upstream server in the proxy_pass directive, which isn't valid when running inside docker. I solved the issue using the external ip-address of the host.

See https://github.com/dokku/dokku-letsencrypt/blob/master/templates/letsencrypt.conf.sigil#L8

BilliAlpha · 2021-09-03T12:36:49Z

Further investigation led me to understand the main issue comes from the path of the volume used with the lego container, instead of -v '/home/dokku/drive/letsencrypt/certs/277c26e8d3c98526567f723718d821b5d11ebde3:/certs' which is the path inside the dokku container, we have to use -v '/var/lib/dokku/home/dokku/drive/letsencrypt/certs/277c26e8d3c98526567f723718d821b5d11ebde3:/certs' (path on the host).

Closes #235

BilliAlpha · 2022-06-23T21:29:42Z

@josegonzalez Thanks for fixing this, however your change impacted the enable command :
/var/lib/dokku/plugins/enabled/letsencrypt/subcommands/enable: line 88: No such file or directory

The issue comes from this read -r -a config <"$config_dir/config" being run in the container but trying to use the new config_dir that uses the host path.

To fix this the letsencrypt_configure_and_get_dir function should return both path, for example separated by a newline.
The resulting function would then use the right path depending on whether it is inside or outside the container.

I'll try to drop a PR for this later

c4lliope · 2022-07-07T13:38:13Z

Hello all, I'm seeing a really similar error on dokku-in-docker.
The process is unable to make the /.lego directory based on permissions.

[core@localhost ~]$ dokku config:export check
export APPLICATION_HOST='check.silicon.am'
export DOKKU_APP_RESTORE='1'
export DOKKU_APP_TYPE='dockerfile'
export DOKKU_DOCKERFILE_PORTS='80'
export DOKKU_LETSENCRYPT_EMAIL='domains@assembled.app'
export DOKKU_PROXY_PORT_MAP='http:80:80  '
export GIT_REV='5538cc86426253dbc9e71be64df4aab098f63bba'

[core@localhost ~]$ dokku letsencrypt:enable check
=====> Enabling letsencrypt for check
-----> Enabling ACME proxy for check...
       ok: run: nginx: (pid 5656) 25045s
-----> Getting letsencrypt certificate for check...
        - Domain 'check.silicon.am'
/var/lib/dokku/plugins/enabled/letsencrypt/subcommands/enable: line 88: /var/lib/dokku/home/dokku/check/letsencrypt/certs/c5c30b9f619ead60e6252975b28d3686fc36701e/config: No such file or directory
       2022/07/07 13:23:07 Could not check/create path: mkdir /.lego: permission denied
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for check...
       ok: run: nginx: (pid 5656) 25049s
 !     Failed to setup letsencrypt
 !     Check log output for further information on failure

The other error message here regarding the config file, could be a path mapping issue?
The mapping is /var/lib/dokku:/mnt/dokku inside the dokku pod.
The file is present at the displayed path on my host machine:

[core@localhost ~]$ cat /var/lib/dokku/home/dokku/check/letsencrypt/certs/c5c30b9f619ead60e6252975b28d3686fc36701e/config
--http.port :22793 --pem --http --accept-tos --cert.timeout 2592000 --path /certs --server https://acme-v02.api.letsencrypt.org/directory --email domains@assembled.app   --domains check.silicon.am

c4lliope · 2022-07-07T13:46:20Z

Ah - there is an error in the documentation for the docker install, on:
https://dokku.com/docs/getting-started/install/docker/

DOKKU_HOST_ROOT should be changed in the docker container run command:

docker container run \
  --env DOKKU_HOSTNAME=dokku.me \
  --env DOKKU_HOST_ROOT=/var/lib/dokku/home/dokku \
  # ...

docker container run \
  --env DOKKU_HOSTNAME=dokku.me \
  --env DOKKU_HOST_ROOT=/mnt/dokku/home/dokku \
  # ...

Once I made this change I see the same error as @BilliAlpha ,
regarding mkdir /certs/accounts: permission denied.

c4lliope · 2022-07-07T14:24:43Z

Sorry, seems like I had been going backwards.
Keeping the DOKKU_HOST_ROOT=/var/lib/dokku/...,
I am seeing the error described from L88:
https://github.com/dokku/dokku-letsencrypt/blob/master/subcommands/enable#L88

I made a fork of dokku-letsencrypt, and I'll see if I can manage a proper build.

c4lliope · 2022-07-07T15:29:25Z

Here's some progress -
https://base.assembled.app/code/dokku-letsencrypt/src/branch/master/subcommands/enable#L83-L104

I bypassed one error by breaking up two function calls,
letsencrypt_configure_dir_on_host "$app" "$acme_port", and
letsencrypt_configure_dir_in_plugin_pod "$app" "$acme_port".
(see https://base.assembled.app/code/dokku-letsencrypt/src/branch/master/functions#L191-L201)

Running basic dokku letsencrypt:enable check produces:

-----> Getting letsencrypt certificate for check...
        - Domain 'check.silicon.am'

--------------------------------------------------------------------------------
config_dir in pod: /mnt/dokku/home/dokku/check/letsencrypt/certs/c5c30b9f619ead60e6252975b28d3686fc36701e
--------------------------------------------------------------------------------

        - Domain 'check.silicon.am'

--------------------------------------------------------------------------------
config_dir on main machine: /var/lib/dokku/home/dokku/check/letsencrypt/certs/c5c30b9f619ead60e6252975b28d3686fc36701e
--------------------------------------------------------------------------------

       2022/07/07 15:18:19 No key found for account domains@assembled.app. Generating a P256 key.
       2022/07/07 15:18:19 Could not check/create directory for account domains@assembled.app: mkdir /certs/accounts: permission denied
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for check...

So, back on a permissions error on /certs/accounts inside our acme pod.
A simple permissions check:

[core@localhost ~]$ ls -l /var/lib/dokku/home/dokku/check/letsencrypt/certs/
total 0
drwxrwxr-x. 2 200 200 20 Jul  7 14:20 c5c30b9f619ead60e6252975b28d3686fc36701e

[core@localhost ~]$ ls -l /var/lib/dokku/home/dokku/check/letsencrypt/certs/c5c30b9f619ead60e6252975b28d3686fc36701e/
total 4
-rw-rw-r--. 1 200 200 197 Jul  7 15:18 config

c4lliope · 2022-07-07T15:51:40Z

Ah, repaired this issue (SELinux concern) by adding --privileged.
Running again, I see:

-----> Getting letsencrypt certificate for check...
        - Domain 'check.silicon.am'

--------------------------------------------------------------------------------
config_dir in pod: /mnt/dokku/home/dokku/check/letsencrypt/certs/c5c30b9f619ead60e6252975b28d3686fc36701e
--------------------------------------------------------------------------------

        - Domain 'check.silicon.am'

--------------------------------------------------------------------------------
config_dir on main machine: /var/lib/dokku/home/dokku/check/letsencrypt/certs/c5c30b9f619ead60e6252975b28d3686fc36701e
--------------------------------------------------------------------------------

       2022/07/07 15:42:34 No key found for account domains@assembled.app. Generating a P256 key.
       2022/07/07 15:42:34 Saved key to /certs/accounts/acme-v02.api.letsencrypt.org/domains@assembled.app/keys/domains@assembled.app.key
       2022/07/07 15:42:35 [INFO] acme: Registering account for domains@assembled.app
       !!!! HEADS UP !!!!
       
       Your account credentials have been saved in your Let's Encrypt
       configuration directory at "/certs/accounts".
       
       You should make a secure backup of this folder now. This
       configuration directory will also contain certificates and
       private keys obtained from Let's Encrypt so making regular
       backups of this folder is ideal.
       2022/07/07 15:42:35 [INFO] [check.silicon.am] acme: Obtaining bundled SAN certificate
       2022/07/07 15:42:35 [INFO] [check.silicon.am] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/127973510106
       2022/07/07 15:42:35 [INFO] [check.silicon.am] acme: Could not find solver for: tls-alpn-01
       2022/07/07 15:42:35 [INFO] [check.silicon.am] acme: use http-01 solver
       2022/07/07 15:42:35 [INFO] [check.silicon.am] acme: Trying to solve HTTP-01
       2022/07/07 15:42:41 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/127973510106
       2022/07/07 15:42:41 Could not obtain certificates:
        error: one or more domains had a problem:
       [check.silicon.am] acme: error: 400 :: urn:ietf:params:acme:error:connection :: 98.218.220.151: Fetching http://check.silicon.am/.well-known/acme-challenge/UmI86DKyS2B9NbuS3DWoicV58S8WACXO4oKK6WbP6c4: Connection refused
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for check...
       ok: run: nginx: (pid 6293) 17s
 !     Failed to setup letsencrypt

My certs came up missing:

[core@localhost ~]$ sudo tree /var/lib/dokku/home/dokku/check/letsencrypt/certs/c5c30b9f619ead60e6252975b28d3686fc36701e/
/var/lib/dokku/home/dokku/check/letsencrypt/certs/c5c30b9f619ead60e6252975b28d3686fc36701e/
├── accounts
│   └── acme-v02.api.letsencrypt.org
│       └── domains@assembled.app
│           ├── account.json
│           └── keys
│               └── domains@assembled.app.key
├── certificates
└── config

So, beyond basic permissioning errors - here is an acme "400 connection refused".

yorch · 2022-10-05T12:29:51Z

I'm still seeing this permission denied error when using the most updated version of the plugin

❯ dokku letsencrypt:enable my-app
=====> Enabling letsencrypt for my-app
-----> Enabling ACME proxy for my-app...
       ok: run: nginx: (pid 42265) 537s
-----> Getting letsencrypt certificate for my-app...
        - Domain 'my-app.dokku.me'
       2022/10/05 12:16:33 No key found for account myemail@domain.com. Generating a P256 key.
       2022/10/05 12:16:33 Could not check/create directory for account myemail@domain.com: mkdir /certs/accounts: permission denied
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for my-app...
       ok: run: nginx: (pid 42265) 538s
 !     Failed to setup letsencrypt
 !     Check log output for further information on failure

BilliAlpha · 2022-10-10T10:51:42Z

@yorch Make sure you set the DOKKU_HOST_ROOT env var on the dokku container.

josegonzalez added a commit that referenced this issue May 12, 2022

fix: respect a mapped dokku root when running dokku in a container

8b0b5a0

Closes #235

josegonzalez mentioned this issue May 12, 2022

fix: respect a mapped dokku root when running dokku in a container #261

Merged

josegonzalez closed this as completed in #261 May 12, 2022

josegonzalez reopened this Jun 25, 2022

c4lliope mentioned this issue Jul 7, 2022

Change an erroneous DOKKU_HOST_ROOT in docker. dokku/dokku#5243

Closed

BilliAlpha mentioned this issue Jul 8, 2022

fix: use container-local config dir path when reading arguments #270

Merged

josegonzalez closed this as completed in #270 Jul 10, 2022

c4lliope mentioned this issue Jul 11, 2022

LetsEncrypt inside Docker; 502 error. #274

Open

BilliAlpha mentioned this issue Jul 11, 2022

Return both host and container config dir #275

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to create directory for account (dokku in docker) #235

Unable to create directory for account (dokku in docker) #235

BilliAlpha commented May 28, 2021 •

edited

Loading

BilliAlpha commented Jun 14, 2021 •

edited

Loading

BilliAlpha commented Sep 3, 2021

BilliAlpha commented Jun 23, 2022 •

edited

Loading

c4lliope commented Jul 7, 2022

c4lliope commented Jul 7, 2022

c4lliope commented Jul 7, 2022

c4lliope commented Jul 7, 2022

c4lliope commented Jul 7, 2022

yorch commented Oct 5, 2022

BilliAlpha commented Oct 10, 2022

Unable to create directory for account (dokku in docker) #235

Unable to create directory for account (dokku in docker) #235

Comments

BilliAlpha commented May 28, 2021 • edited Loading

Description of problem

How reproducible

Steps to Reproduce

Actual Results

Expected Results

Environment Information

dokku report output

How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

BilliAlpha commented Jun 14, 2021 • edited Loading

BilliAlpha commented Sep 3, 2021

BilliAlpha commented Jun 23, 2022 • edited Loading

c4lliope commented Jul 7, 2022

c4lliope commented Jul 7, 2022

c4lliope commented Jul 7, 2022

c4lliope commented Jul 7, 2022

c4lliope commented Jul 7, 2022

yorch commented Oct 5, 2022

BilliAlpha commented Oct 10, 2022

BilliAlpha commented May 28, 2021 •

edited

Loading

`dokku report` output

BilliAlpha commented Jun 14, 2021 •

edited

Loading

BilliAlpha commented Jun 23, 2022 •

edited

Loading