Multi-user Che on Kubernetes doesn't work

[sleshchenko]

Description

Multi-user Che on Kubernetes doesn't work.

Postgresql image failed to start with the following error:

Initial call to run-postgresql...
=> sourcing /opt/app-root/src/postgresql-pre-start/init-debug.sh ...
/opt/app-root/src/postgresql-pre-start/init-debug.sh: line 13: POSTGRESQL_LOG_DEBUG: unbound variable
pg_ctl: directory "/var/lib/pgsql/data/userdata" is not a database cluster directory
Initializing Che user...
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Warning: Failed initializing the Che user and database
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
pg_ctl: directory "/var/lib/pgsql/data/userdata" is not a database cluster directory
=> sourcing /opt/app-root/src/postgresql-pre-start/init-debug.sh ...
/opt/app-root/src/postgresql-pre-start/init-debug.sh: line 13: POSTGRESQL_LOG_DEBUG: unbound variable

It is related to #9343

Reproduction Steps

1. Deploy Mulit-user Che on Kubernetes.
   Expected: Che is deployed and available.
   Actual: Che is not deployed because of failing Postgresql start.

OS and version:

Diagnostics:

[monaka]

I've already succeeded to deploy Che into Azure AKS. I'll send some more patches for Helm charts after merged PR #11761 #11820 #11835 .

[mikekuzak]

I still have the same problem when deploying Che to K8s.

Initial call to run-postgresql... mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied pg_ctl: directory "/var/lib/pgsql/data/userdata" does not exist Initializing Che user... psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? Warning: Failed initializing the Che user and database psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? pg_ctl: directory "/var/lib/pgsql/data/userdata" does not exist mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

Used helm chart from master:
helm upgrade --install che-eclipse --namespace che-eclipse -f ./values/multi-user.yaml ./

postgres-data PVC has been bound

[CyborTronik]

Getting same error on Windows Server 2019:

kubectl logs --namespace=che postgres-656859d986-w4shn
Initial call to run-postgresql...
mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied
pg_ctl: directory "/var/lib/pgsql/data/userdata" does not exist
Initializing Che user...
psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Warning: Failed initializing the Che user and database
psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
pg_ctl: directory "/var/lib/pgsql/data/userdata" does not exist
mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

Other related info:

kubectl describe --namespace=che pod postgres-656859d986-w4shn
Name:               postgres-656859d986-w4shn
Namespace:          che
Priority:           0
PriorityClassName:  <none>
Node:               docker-desktop/192.168.65.3
Start Time:         Sun, 19 Jan 2020 07:34:17 -0800
Labels:             app=che
                    component=postgres
                    io.kompose.service=postgres
                    pod-template-hash=656859d986
Annotations:        <none>
Status:             Running
IP:                 10.1.0.13
Controlled By:      ReplicaSet/postgres-656859d986
Containers:
  postgres:
    Container ID:   docker://2eab1e53be06e424a864997f01f63cd4697a77f62c592a67a32aa9c57ec56f76
    Image:          quay.io/eclipse/che-postgres:nightly
    Image ID:       docker-pullable://quay.io/eclipse/che-postgres@sha256:3f4c5fc4b98a951f1c9ee8f622563b2ba0cc6d1183d9c2502ebc7ad207e258c2
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 19 Jan 2020 07:57:01 -0800
      Finished:     Sun, 19 Jan 2020 07:57:01 -0800
    Ready:          False
    Restart Count:  9
    Liveness:       tcp-socket :5432 delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:      exec [bash -c psql -h 127.0.0.1 -U ${POSTGRESQL_USER} -q -d $POSTGRESQL_DATABASE -c "SELECT 1"] delay=5s timeout=1s period=3s #success=1 #failure=10
    Environment:
      POSTGRESQL_USER:        keycloak
      POSTGRESQL_PASSWORD:    keycloak
      POSTGRESQL_DATABASE:    keycloak
      CHE_POSTGRES_USERNAME:  pgche
      CHE_POSTGRES_PASSWORD:  pgchepassword
      CHE_POSTGRES_DATABASE:  dbche
    Mounts:
      /var/lib/pgsql/data from postgres-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xknff (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  postgres-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  postgres-data
    ReadOnly:   false
  default-token-xknff:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xknff
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                   From                     Message
  ----     ------            ----                  ----                     -------
  Warning  FailedScheduling  27m (x3 over 27m)     default-scheduler        pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled         27m                   default-scheduler        Successfully assigned che/postgres-656859d986-w4shn to docker-desktop
  Normal   Pulling           25m (x4 over 27m)     kubelet, docker-desktop  Pulling image "quay.io/eclipse/che-postgres:nightly"
  Normal   Pulled            25m (x4 over 25m)     kubelet, docker-desktop  Successfully pulled image "quay.io/eclipse/che-postgres:nightly"
  Normal   Created           25m (x4 over 25m)     kubelet, docker-desktop  Created container postgres
  Normal   Started           25m (x4 over 25m)     kubelet, docker-desktop  Started container postgres
  Warning  BackOff           2m6s (x117 over 25m)  kubelet, docker-desktop  Back-off restarting failed container

I was trying: chectl server:start -m -p docker-desktop
@monaka shall be this issue reopened or is another case?