App not reachable: Lets encrypt plugin fails & ports of container not exposed

[JanRuettinger]

Description of the problem

I installed dokku, created a new app try to generate an SSL certificate with Let's encrypt which fails.

Command: dokku letsencrypt:enable project-explorer-backend
Error: Could not obtain certificates: error: one or more domains had a problem: [api.backend.dataleap.dev] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 65.109.23.49: Invalid response from ...

Environment Information

My setup is rather simple:

• Host system is an Ubuntu 22.04.
• Vercel's DNS server for the subdomain: api.backend.xxx.dex
• Standard dokku installation from the Getting Started page
• Deployed app: simple hello world Go web app with // +heroku goVersion 1.18 in go.mod (no docker file).
• Configured the domain for the dokku app

Diagnostics:

Command: dig -t A api.backend.xxx.dev

; <<>> DiG 9.10.6 <<>> -t A api.backend.xxx.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64513
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;api.backend.xxx.dev. IN A

;; ANSWER SECTION:
api.backend.xxx.dev. 60 IN A XX.XXX.XX.XX

;; Query time: 72 msec
;; SERVER: 192.168.178.1#53(192.168.178.1)
;; WHEN: Wed Jan 18 19:04:20 CET 2023
;; MSG SIZE rcvd: 69

=> DNS settings seem to be correct. Subdomain is reachable.

Command: docker ps

=> I think it's odd that no port is exposed of the app container.

Command: dokku nginx:show-config project-explorer-backend
Output:

server {
  listen      [::]:80;
  listen      80;
  server_name api.backend.xxx.dev;
  access_log  /var/log/nginx/project-explorer-backend-access.log;
  error_log   /var/log/nginx/project-explorer-backend-error.log;

  location    / {

    gzip on;
    gzip_min_length  1100;
    gzip_buffers  4 32k;
    gzip_types    text/css text/javascript text/xml text/plain text/x-component application/javascript application/x-javascript application/json application/xml  application/rss+xml font/truetype application/x-font-ttf font/opentype application/vnd.ms-fontobject image/svg+xml;
    gzip_vary on;
    gzip_comp_level  6;

    proxy_pass  http://project-explorer-backend-5000;
    proxy_http_version 1.1;
    proxy_read_timeout 60s;
    proxy_buffer_size 4096;
    proxy_buffering on;
    proxy_buffers 8 4096;
    proxy_busy_buffers_size 8192;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $http_connection;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Request-Start $msec;

  }



  error_page 400 401 402 403 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 /400-error.html;
  location /400-error.html {
    root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
    internal;
  }

  error_page 404 /404-error.html;
  location /404-error.html {
    root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
    internal;
  }

  error_page 500 501 502 503 504 505 506 507 508 509 510 511 /500-error.html;
  location /500-error.html {
    root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
    internal;
  }
  include /home/dokku/project-explorer-backend/nginx.conf.d/*.conf;

}

upstream project-explorer-backend-5000 {

  server 172.17.0.3:5000;
}

Command: dokku proxy:ports project-explorer-backend
Output:

Port mappings for project-explorer-backend
 -----> scheme  host port  container port
    http           80         5000

dokku report project-explorer-backend output

----> uname: Linux jans 5.15.0-43-generic dokku/dokku#46-Ubuntu SMP Tue Jul 12 10:30:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
                      total        used        free      shared  buff/cache   available
       Mem:          128729       30208         795          15       97725       97278
       Swap:          20479       20337         142
-----> docker version:
       Client: Docker Engine - Community
        Version:           20.10.17
        API version:       1.41
        Go version:        go1.17.11
        Git commit:        100c701
        Built:             Mon Jun  6 23:02:46 2022
        OS/Arch:           linux/amd64
        Context:           default
        Experimental:      true

       Server: Docker Engine - Community
        Engine:
         Version:          20.10.17
         API version:      1.41 (minimum version 1.12)
         Go version:       go1.17.11
         Git commit:       a89b842
         Built:            Mon Jun  6 23:00:51 2022
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.6.6
         GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
        runc:
         Version:          1.1.2
         GitCommit:        v1.1.2-0-ga916309
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info:
       Client:
        Context:    default
        Debug Mode: true
        Plugins:
         app: Docker App (Docker Inc., v0.9.1-beta3)
         buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
         compose: Docker Compose (Docker Inc., v2.6.0)
         scan: Docker Scan (Docker Inc., v0.17.0)

       Server:
        Containers: 46
         Running: 11
         Paused: 0
         Stopped: 35
        Images: 86
        Server Version: 20.10.17
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
        runc version: v1.1.2-0-ga916309
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: default
         cgroupns
        Kernel Version: 5.15.0-43-generic
        Operating System: Ubuntu 22.04 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 32
        Total Memory: 125.7GiB
        Name: jansquared
        ID: GQNY:2PA6:3FOD:XL55:NP4U:QZ7F:TEV4:6NON:Y6EP:CFXR:D62S:OIBR
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false

-----> git version: git version 2.34.1
-----> sigil version: 0.9.0build+bc921b7
-----> herokuish version:
       herokuish: v0.5.40
       buildpacks:
         heroku-buildpack-multi     v1.2.0
         heroku-buildpack-ruby      v244
         heroku-buildpack-nodejs    v202
         heroku-buildpack-clojure   v90
         heroku-buildpack-python    v223
         heroku-buildpack-java      v72
         heroku-buildpack-gradle    v38
         heroku-buildpack-scala     v96
         heroku-buildpack-play      v26
         heroku-buildpack-php       v227
         heroku-buildpack-go        v169
         heroku-buildpack-nginx     v22
         buildpack-null             v3
-----> dokku version: dokku version 0.29.4
-----> plugn version: plugn: 0.12.0build+3a27594
-----> dokku plugins:
         00_dokku-standard    0.29.4 enabled    dokku core standard plugin
         20_events            0.29.4 enabled    dokku core events logging plugin
         app-json             0.29.4 enabled    dokku core app-json plugin
         apps                 0.29.4 enabled    dokku core apps plugin
         builder              0.29.4 enabled    dokku core builder plugin
         builder-dockerfile   0.29.4 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.29.4 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.29.4 enabled    dokku core builder-lambda plugin
         builder-null         0.29.4 enabled    dokku core builder-null plugin
         builder-pack         0.29.4 enabled    dokku core builder-pack plugin
         buildpacks           0.29.4 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.29.4 enabled    dokku core caddy-vhosts plugin
         certs                0.29.4 enabled    dokku core certificate management plugin
         checks               0.29.4 enabled    dokku core checks plugin
         common               0.29.4 enabled    dokku core common plugin
         config               0.29.4 enabled    dokku core config plugin
         cron                 0.29.4 enabled    dokku core cron plugin
         docker-options       0.29.4 enabled    dokku core docker-options plugin
         domains              0.29.4 enabled    dokku core domains plugin
         enter                0.29.4 enabled    dokku core enter plugin
         git                  0.29.4 enabled    dokku core git plugin
         letsencrypt          0.18.1 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.29.4 enabled    dokku core logs plugin
         network              0.29.4 enabled    dokku core network plugin
         nginx-vhosts         0.29.4 enabled    dokku core nginx-vhosts plugin
         plugin               0.29.4 enabled    dokku core plugin plugin
         proxy                0.29.4 enabled    dokku core proxy plugin
         ps                   0.29.4 enabled    dokku core ps plugin
         registry             0.29.4 enabled    dokku core registry plugin
         repo                 0.29.4 enabled    dokku core repo plugin
         resource             0.29.4 enabled    dokku core resource plugin
         run                  0.29.4 enabled    dokku core run plugin
         scheduler            0.29.4 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.29.4 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.29.4 enabled    dokku core scheduler-null plugin
         shell                0.29.4 enabled    dokku core shell plugin
         ssh-keys             0.29.4 enabled    dokku core ssh-keys plugin
         storage              0.29.4 enabled    dokku core storage plugin
         trace                0.29.4 enabled    dokku core trace plugin
         traefik-vhosts       0.29.4 enabled    dokku core traefik-vhosts plugin
=====> project-explorer-backend app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:
=====> project-explorer-backend app information
       App created at:                1674059448
       App deploy source:             project-explorer-backend
       App deploy source metadata:    project-explorer-backend
       App dir:                       /home/dokku/project-explorer-backend
       App locked:                    false
=====> project-explorer-backend builder information
       Builder build dir:
       Builder computed build dir:
       Builder computed selected:
       Builder global build dir:
       Builder global selected:
       Builder selected:
=====> project-explorer-backend builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile
       Builder dockerfile global dockerfile path: Dockerfile
       Builder dockerfile dockerfile path:
=====> project-explorer-backend builder-lambda information
       Builder lambda computed lambdayml path: lambda.yml
       Builder lambda global lambdayml path: lambda.yml
       Builder lambda lambdayml path:
=====> project-explorer-backend builder-pack information
       Builder pack computed projecttoml path: project.toml
       Builder pack global projecttoml path: project.toml
       Builder pack projecttoml path:
=====> project-explorer-backend buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-20
       Buildpacks global stack:
       Buildpacks list:
       Buildpacks stack:
=====> project-explorer-backend ssl information
       Ssl dir:                       /home/dokku/project-explorer-backend/tls
       Ssl enabled:                   false
       Ssl hostnames:
       Ssl expires at:
       Ssl issuer:
       Ssl starts at:
       Ssl subject:
       Ssl verified:
=====> project-explorer-backend checks information
       Checks disabled list:          none
       Checks skipped list:           none
       Checks computed wait to retire: 60
       Checks global wait to retire:  60
       Checks wait to retire:
=====> project-explorer-backend cron information
       Cron task count:               0
=====> project-explorer-backend docker options information
       Docker options build:
       Docker options deploy:         --restart=on-failure:10
       Docker options run:
=====> project-explorer-backend domains information
       Domains app enabled:           true
       Domains app vhosts:            api.backend.xxx.dev
       Domains global enabled:        true
       Domains global vhosts:         backend.xxx.dev
=====> project-explorer-backend git information
       Git deploy branch:             main
       Git global deploy branch:      master
       Git keep git dir:              false
       Git rev env var:               GIT_REV
       Git sha:
       Git source image:
       Git last updated at:           1674059489
Could not open file or uri for loading certificate from /home/dokku/project-explorer-backend/tls/server.crt
4047CB89A67F0000:error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file
4047CB89A67F0000:error:80000002:system library:file_open:No such file or directory:../providers/implementations/storemgmt/file_store.c:267:calling stat(/home/dokku/project-explorer-backend/tls/server.crt)
Unable to load certificate
=====> project-explorer-backend letsencrypt information
       Letsencrypt active:            false
       Letsencrypt autorenew:         false
       Letsencrypt email:             xxx@xxxx
       Letsencrypt expiration:        1674000000
=====> project-explorer-backend logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:
       Logs max size:
       Logs vector sink:
=====> project-explorer-backend network information
       Network attach post create:
       Network attach post deploy:
       Network bind all interfaces:          false
       Network computed attach post create:
       Network computed attach post deploy:
       Network computed bind all interfaces: false
       Network computed initial network:
       Network computed tld:
       Network global attach post create:
       Network global attach post deploy:
       Network global bind all interfaces:   false
       Network global initial network:
       Network global tld:
       Network initial network:
       Network static web listener:
       Network tld:
       Network web listeners:                172.17.0.3:5000
=====> project-explorer-backend nginx information
       Nginx access log format:
       Nginx access log path:         /var/log/nginx/project-explorer-backend-access.log
       Nginx bind address ipv4:
       Nginx bind address ipv6:       ::
       Nginx client max body size:
       Nginx disable custom config:   false
       Nginx error log path:          /var/log/nginx/project-explorer-backend-error.log
       Nginx global hsts:             true
       Nginx computed hsts:           true
       Nginx hsts:
       Nginx hsts include subdomains: true
       Nginx hsts max age:            15724800
       Nginx hsts preload:            false
       Nginx computed nginx conf sigil path: nginx.conf.sigil
       Nginx global nginx conf sigil path: nginx.conf.sigil
       Nginx nginx conf sigil path:
       Nginx proxy buffer size:       4096
       Nginx proxy buffering:         on
       Nginx proxy buffers:           8 4096
       Nginx proxy busy buffers size: 8192
       Nginx proxy read timeout:      60s
       Nginx last visited at:         1674065687
       Nginx x forwarded for value:   $remote_addr
       Nginx x forwarded port value:  $server_port
       Nginx x forwarded proto value: $scheme
       Nginx x forwarded ssl:
=====> project-explorer-backend proxy information
       Proxy enabled:                 true
       Proxy port map:                http:80:5000
       Proxy type:                    nginx
=====> project-explorer-backend ps information
       Deployed:                      true
       Processes:                     1
       Ps can scale:                  true
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:
       Ps restart policy:             on-failure:10
       Restore:                       true
       Running:                       true
       Status web 1:                  running (CID: 28a16c50c73)
=====> project-explorer-backend registry information
       Registry computed image repo:      dokku/project-explorer-backend
       Registry computed push on release: false
       Registry computed server:
       Registry global push on release:
       Registry global server:
       Registry image repo:
       Registry push on release:
       Registry server:
       Registry tag version:
=====> project-explorer-backend resource information
=====> project-explorer-backend scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:
=====> project-explorer-backend scheduler-docker-local information
       Scheduler docker local disable chown:
       Scheduler docker local init process: true
       Scheduler docker local parallel schedule count:
=====> project-explorer-backend storage information
       Storage build mounts:
       Storage deploy mounts:
       Storage run mounts:

[josegonzalez]

I think the .dev subdomain is special and needs ssl by default. The LE enable command usually has a url in the output we can look at to see why it failed. Can you take a look at that url and see what exactly failed? You didnt show that part so its impossible for me to investigate.

[JanRuettinger]

Here is the output:

{
  "identifier": {
    "type": "dns",
    "value": "api.backend.xxx.dev"
  },
  "status": "invalid",
  "expires": "2023-01-25T19:29:40Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "65.109.23.49: Invalid response from http://api.backend.xxx.dev/.well-known/acme-challenge/ZUFiCyNMMzmS90EMl8kOQGbatEX3kYU_fwkiAwqJ60c: 504",
        "status": 403
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5000779933/NNcMXw",
      "token": "ZUFiCyNMMzmS90EMl8kOQGbatEX3kYU_fwkiAwqJ60c",
      "validationRecord": [
        {
          "url": "http://api.backend.xxx.dev/.well-known/acme-challenge/ZUFiCyNMMzmS90EMl8kOQGbatEX3kYU_fwkiAwqJ60c",
          "hostname": "api.backend.xxx.dev",
          "port": "80",
          "addressesResolved": [
            "65.109.23.49"
          ],
          "addressUsed": "65.109.23.49"
        }
      ],
      "validated": "2023-01-18T19:29:41Z"
    }
  ]
}

[josegonzalez]

What is the ip of your dokku host?

[josegonzalez]

For some reason, it was not possible to route to the lego container for the acme challenge. I was able to get around this by rewriting the plugin a bit. A fix will be coming, but I'm moving this to the letsencrypt repository for now.