lemmy ports

[mcmikemn]

I'm trying to add Lemmy (lemmy branch). The lemmy service is failing to connect to the postgres service. It was connecting before I turned the nginx ("proxy") service on, so I think it has something to do with ports now that both nginx and Traefik are reverse-proxying. The lemmy log says:

thread 'main' panicked at 'Error connecting to postgres://lemmy:86ruE60ExlfUQwzAS7e6IGqOB8Z93JkP@postgres:5432
/lemmy: could not connect to server: Connection refused
lemmy-lemmy-1     |     Is the server running on host "postgres" (192.168.208.4) and accepting
lemmy-lemmy-1     |     TCP/IP connections on port 5432?

But docker inspect lemmy-postgres-1 shows:

"Ports": {
"5432/tcp": null
...
"IPAddress": "192.168.208.4",

[mcmikemn]

I can ping postgres from the lemmy container.

[mcmikemn]

But I still get "bad gateway". The router and service look error free in Traefik.

[mcmikemn]

If I was wrong about the {{ }} expressions in config.hjson - if they are replaced a la jinja or are read as variables, then if I removed the curly braces and replaced the entire expression with my values, might that be causing this type of error?

Their original file:

{
  # for more info about the config, check out the documentation
  # https://join-lemmy.org/docs/en/administration/configuration.html

  database: {
    host: postgres
    password: "{{ postgres_password }}"
  }
  hostname: "{{ domain }}"
  pictrs: {
    url: "http://pictrs:8080/"
    api_key: "{{ postgres_password }}"
  }
  email: {
    smtp_server: "postfix:25"
    smtp_from_address: "noreply@{{ domain }}"
    tls_type: "none"
  }
}

[EnigmaCurry]

bad gateway is a separate issue, and it may be coming from nginx or it make be coming from traefik, check the logs of each to know for sure.

[mcmikemn]

My nginx log never actually says nginx up has started up, only that it's ready to:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/07/30 21:07:14 [notice] 1#1: using the "epoll" event method
2023/07/30 21:07:14 [notice] 1#1: nginx/1.25.1
2023/07/30 21:07:14 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r4)
2023/07/30 21:07:14 [notice] 1#1: OS: Linux 5.4.0-153-generic
2023/07/30 21:07:14 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/07/30 21:07:14 [notice] 1#1: start worker processes
2023/07/30 21:07:14 [notice] 1#1: start worker process 30
2023/07/30 21:07:14 [notice] 1#1: start worker process 31

[mcmikemn]

In my Traefik logs, I get only 1 line when I grep for "lemmy":

$ docker logs traefik-traefik-1 | grep -i lemmy
2023/07/29 12:32:23 reverseproxy.go:667: httputil: ReverseProxy read error during body copy: unexpected EOF

[EnigmaCurry]

so I think it has something to do with ports now that both nginx and Traefik are reverse-proxying.

every service listed in the docker-compose is a part of the same docker network, (ie. each docker-compose project gets a default network, and unless you specify explicit networks (you haven't) they are all on one network.) So there is no traefik or nginx to worry about, lemmy container should be able to connect to postgres container directly, no middleman.

[EnigmaCurry]

they are replaced a la jinja or are read as variables,

config/setup.sh uses envsubst which want to replace environment variables eg. ${VARIABLE} - not jinga style {{ VARIABLE }}

[mcmikemn]

Yes, but do any of Lemmy's services need them in jinja style?

[EnigmaCurry]

I'm not sure, but this is why I like to name templates as such, for example I would name the file config.template.hjson

[EnigmaCurry]

it did seem to know what it was talking about when it errored tho:

thread 'main' panicked at 'Error connecting to postgres://lemmy:86ruE60ExlfUQwzAS7e6IGqOB8Z93JkP@postgres:5432/lemmy: could not connect to server: Connection refused
Is the server running on host "postgres" (192.168.224.5) and accepting

this has all the password info from your var, so I think that part is working.

[mcmikemn]

Without knowing what they're doing with this file, there's no way to know whether I should replacepassword: "{{ postgres_password }}" with

• password: "{{ ${MY_PW_ENV_VAR} }}"
• password: "${MY_PW_ENV_VAR}"
• password: ${MY_PW_ENV_VAR}
  ...or whether I need to leave it as it is and define an env var named postgres_password.

[mcmikemn]

this has all the password info from your var, so I think that part is working.

That's a good point.

[EnigmaCurry]

@mcmikemn I fixed this and pushed in 37a2b09 - you hadn't written the nginx config file to the right location, so it was running with the default config on port 80. Thats why it was a Bad Gateway.

I replaced the nginx-config container with a custom nginx build with a Dockerfile that adds the static nginx config to /etc/nginx/nginx.conf - since the config is static, it does not need to be run through envsubst and, in fact it fails if you do this because there are some variables that are existing that need to stay put $http_connection and the like, envsubst can be configured to ignore all variables except the ones we want, but then we have to be explicit. Turns out we don't need to because its static and we don't need any variables replaced, so I ripped out the nginx-config container and replaced it with the nginx Dockerfile that adds the static config. It now starts up correctly on port 8536 and theres no more Bad Gateway.

[mcmikemn]

you hadn't written the nginx config file to the right location

Doh! Where was I saving it to? In both of my setup.sh I was trying to write the envsubst-ed file to a specific directory, but the files ended up in the root directory of the docker volume (/var/lib/docker/volumes/<config-dir>/_data/), so I must be mistunderstanding something there.

But it doesn't matter because you simplified this by using a static nginx config file. Yay!

[EnigmaCurry]

Where was I saving it to?

you had the right directory just the wrong filename - you had /etc/nginx/nginx-internal.conf instead of /etc/nginx/nginx.conf

[mcmikemn]

Ah. I wouldn't have figured that one out, so thanks for the better solution!