Skip to content

troubleshooting

Jump to bottom Edit New page
Marc Boorshtein edited this page Oct 30, 2020 · 17 revisions

FAQ and Troubleshooting

Kubernetes

I can't access the portal after its deployed

Do I need to use Helm to deploy?

Not entirely. The Helm charts are mainly for templating. We're not using any dependency management. If you're using another management solution or just want to use manifests, use the helm cli to generate a manifest using the template option instead of install. As an example, when deploying the OpenUnison operator use helm template openunison tremolo/openunison-operator --namespace openunison to generate your manifests that can then be used in your cluster without helm interacting directly with your cluster.

Making Configuration Changes After Deployment

Once Orchestra is deployed, make updates to the orchestra openunison object (kubectl edit openunison orchestra -n openunison). This object is designed to be storable in source control/configuration management. Whenever you make changes to the metadata.annotations, metadata.labels, or spec sections of this object the operator will update OpenUnison's configuration appropriately and rollout new containers by updating your Deployment object.

How do I change OpenUnison's certificates?

OpenUnison's certificate when deployed in Kubernetes is hosted by the Ingress controller, not by the OpenUnison container its self. When used for the login portal, we want to supply the CA certificate for two reasons:

  1. So it can be embedded in the kubectl command correctly
  2. So that the dashboard SSO works properly when validating the login process

Before moving forward you'll need:

  1. A certificate with subject alternative names for your portal (OU_HOST / network.openunison_host) and your dashboard (K8S_DASHBOARD_HOST / network.dashboard_host). If using impersonation, the impersonation host is needed too (K8S_API_HOST / network.api_server_host)
  2. The certificate authority (CA) certificate that signed your certificate from #1
  3. Any intermediate certs needed to complete the chain

Once you have your certificates and keys:

  1. Delete the ou-tls-certificate secret in the openunison namespace - kubectl delete secret ou-tls-certificate -n openunison
  2. Recreate the ou-tls-certificate secret - kubectl create secret tls ou-tls-certificate --cert=/path/to/chain.pem --key=/path/to/key.pem -n openunison - NOTE chain.pem should be your entire certificate chain, including the CA and all intermediate certs.
  3. Edit the orchestra openunison object - kubectl edit openunison orchestra -n openunison. Look for tls_secret_name: ou-tls-certificate and delete the entire array entry its a part of:
      - create_data:
          ca_cert: true
          key_size: 2048
          server_name: k8sou.apps.192-168-159-168.nip.io
          sign_by_k8s_ca: false
          subject_alternative_names:
          - k8sdb.apps.192-168-159-168.nip.io
        import_into_ks: certificate
        name: unison-ca
        tls_secret_name: ou-tls-certificate

Continue editing the CR, find the section called trusted_certificates (you may already have an entry). Add an an item called unison-ca to the list with the value of pem_data being the base64 encoded PEM for your CA's certificate. For example:

    trusted_certificates:
    - name: ldaps
      pem_data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURORENDQWh5Z0F3SUJBZ0lRYlJOajZSS3F0cVZQdlc2NXFaeFhYakFOQmdrcWhraUc5dzBCQVFVRkFEQWkKTVNBd0hnWURWUVFEREJkQlJFWlRMa1ZPVkRKTE1USXVSRTlOUVVsT0xrTlBUVEFlRncweE5EQXpNamd3TVRBMQpNek5hRncweU5EQXpNalV3TVRBMU16TmFNQ0l4SURBZUJnTlZCQU1NRjBGRVJsTXVSVTVVTWtzeE1pNUVUMDFCClNVNHVRMDlOTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUEyczlKa2VOQUhPa1EKMVFZSmdqZWZVd2Nhb2dFTWNhVy9rb0ErYnU5eGJyNHJIeS8yZ04va2M4T2tvUHV3Si9uTmxPSU8rcytNYm5YUwpMOW1VVEM0T0s3dHJrRWppS1hCK0QrVlNZeTZpbVhoNnpwQnROYmVaeXgrcmRCbmFPdjNCeVpSbm5FQjhMbWhNCnZIQSs0Zi90OWZ4LzJ2dDZ3UHgvL1ZnSXE5eXVZWVVRUkxtMVdqeVVCRnJaZUdvU3BQbTBLZXdtK0IwYmhtTWIKZHlDKzNmaGFLQytVazFOUG9kRTI5NzNqTEJaSmVsWnhzWlk0MFd3OHpZUXdkR1lJYlhxb1RjKzFhL3g0ZjFFbgptNEFOcWdnSHR3K05xOHpoc3MzeVR0WStVWUtEUkJJTGRMVlpRaEhKRXhlMGtBZWlzZ014SS9iQndPMUhickZWCit6U25rK252Z1FJREFRQUJvMll3WkRBekJnTlZIU1VFTERBcUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SUcKQ2lzR0FRUUJnamNVQWdJR0NDc0dBUVVGQndNRE1CMEdBMVVkRGdRV0JCVHlKVWZZNjZ6WWJtOWkweGVZSHVGSQo0TU43dURBT0JnTlZIUThCQWY4RUJBTUNCU0F3RFFZSktvWklodmNOQVFFRkJRQURnZ0VCQU01a3o5T0tOU3VYCjh3NE5PZ25mSUZkYXpkMG5QbElVYnZEVmZRb055OVEwUzFTRlVWTWVrSVBOaVZoZkd6eWE5SXdSdEdiMVZhQlEKQVEyT1JJekhyOEEycjVVTkx4M21GanBKbWVPeFF3bFYwWCtnOHMrMjUzS1ZGeE9wUkU2eXlhZ24vQnh4cHRUTAphMVo0cWVRSkxENDJsZDFxR2xSd0Z0VlJtVkZaelZYVnJwdTdOdUZkM3Zsbm5PL3FLV1hVK3VNc2ZYdHNsMTNuCmVjMWt3MUV3cTJqbks4V0ltS1RRNy85V2JhSVkwZ3g4bW93Q0pTT3NScTBURTd6Sy9ONTVkck4xd1hKVnhXZTUKNE4zMmVDcW90WHk5ajlsemRrTmE3YXdiOXEzOG5XVnhQK3ZhNWpxTklEbGxqQjZ0RXh5NW4zczd0NktLNmc1agpUWmdWcXJaMyttcz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    - name: unison-ca
      pem_data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNvRENDQVlnQ0NRRGhlNzN5cDQwQjNEQU5CZ2txaGtpRzl3MEJBUXNGQURBU01SQXdEZ1lEVlFRRERBZHIKZFdKbExXTmhNQjRYRFRJd01ETXlPVEUzTXpJeE1Wb1hEVE13TURNeU56RTNNekl4TVZvd0VqRVFNQTRHQTFVRQpBd3dIYTNWaVpTMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFLRFpSNmRqCm56czB2ZGhPdzFRdGIwWm9rb2xxTUJwYlpkcGMrY3EyRWFxRThCWEtTZDJMOGFSQldPVVdBNFhVMDRPclEyUjIKUU83b0NybERBR3dMRmpORktXZlBDNG5PMFhYcmVydEN2UEdrNGFKWndmSDArSThnTmw4TjhwL09QQklQOVFaaQp4K0RYekJibGFkRlZydGVtVTh5L1JqMGZXb1pJY0NPdzk5bDBhTGtxb21jSGo0UFBtYTZqYWYrOFZxSnZDT0ZMCktBUEVoeVEzYWFIYlZGNFV3TGlDMWNjenhFWWxXcnNIOUFleVpzdGovdG5FcEl3MzlVQkNMUVZLaHFFZVl5a0YKU3VGekNTcFJZZmNoUFBtK3NMQjY3cmoyc01KUFhyU1pDZHdjR0NSeW5GUDNjcnE3bkpRdndmM2tZV21vTDc5bQo3T2ZsamlweTJzeC81RThDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFnWFlsZmNpNjhHMmFMbkhqCjFjcTZmNmgvZGdEbWxXeUpGZG9qYnNBRUFCb0plUHhxY1dWOGZsVGZTTkRYY25hWmtQOEdhQXo2T2NpbXEwYlgKVWdEUnVIWlM5ZmZBQXJrK29DZTVxZVhNN3IvQUJrdWVZTDBiSVN5elowQjBBUnBtVldRd2MrS1E3TFltYWQxVQpiVVU3Qi9seHdIcDYrcXA1RVFrQ1hWeWRNanc5TS9pU3VoTXhYZjdkRGZ4R0hJSHgwZU9kVjhFQ0cyc29xTUw4ClZPU1E4OEZ1YlZQVWVBOHhTWURIWVYwaWZVTW4wOFUyY1FmeHd2Tlp0Y3FhTDlmUWZQTFExYU9NWDV2S0lUdkwKS29QczJUUkhKek44WTJENVNxdThTckRDY25DQXN4RFNMUXR6Q2cxelY1V3VCWXM3UHdnWExyZExBcVdUb3UvbAptN0VJMVE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

Save and close your editor so the CR can update. Make sure to update your API servers to use the new certificate.

Using a Commercially Signed Ingress Certificate

OpenUnison deploy's a self-signed certificate for your Ingress to make it easy to start using. Most production deployments do not use this certificate but instead use a wildcard or commercially signed certificate. To stop using the self signed certificate:

  1. Delete the ou-tls-certificate secret in the openunison namespace - kubectl delete secret ou-tls-certificate -n openunison
  2. Edit the orchestra openunison object - kubectl edit openunison orchestra -n openunison. Look for tls_secret_name: ou-tls-certificate and delete the entire array entry its a part of:
      - create_data:
          ca_cert: true
          key_size: 2048
          server_name: k8sou.apps.192-168-159-168.nip.io
          sign_by_k8s_ca: false
          subject_alternative_names:
          - k8sdb.apps.192-168-159-168.nip.io
        import_into_ks: certificate
        name: unison-ca
        tls_secret_name: ou-tls-certificate

Save and close your editor so the CR can update. Make sure to update your API server to remove the --oidc-ca-file flag if not using impersonation. Once the OpenUnison pods are redeployed there should no longer see the OpenUnison certificate when you login to the token screen. Also, the oulogin plugin will work now as well.

How do I monitor OpenUnison with Prometheus?

All of the Orchestra portals in https://github.com/OpenUnison have a /metrics endpoint that exposes all of the built in Java metrics from the prometheus simple client as well as the number of open sessions. This endpoint is secured as OpenUnison is part of security and you don't want to leak information about your security systems. By default, the service account for the Kube Prometheus project (https://github.com/coreos/kube-prometheus) is authorized but that can be changed.

First, update the orchestra openunison object in the openunison namespace (kubectl edit openunison orchestrra -n openunison), updating

  - name: PROMETHEUS_SERVICE_ACCOUNT
    value: system:serviceaccount:monitoring:prometheus-k8s

to the service account you want to authorize. Next, create an RBAC role and binding to allow the monitoring stack to query the openunison-orchestra Service:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
    audit2rbac.liggitt.net/version: v0.7.0
  labels:
    audit2rbac.liggitt.net/generated: "true"
    audit2rbac.liggitt.net/user: system-serviceaccount-monitoring-prometheus-k8s
  name: monitoring-list-services
  namespace: openunison
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - pods
  - services
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    audit2rbac.liggitt.net/version: v0.7.0
  labels:
    audit2rbac.liggitt.net/generated: "true"
    audit2rbac.liggitt.net/user: system-serviceaccount-monitoring-prometheus-k8s
  name: monitoring-list-services
  namespace: openunison
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: monitoring-list-services
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring

Finally, create a service monitor in the monitoring namespace:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: prometheus-operator
  name: orchestra
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: openunison-secure-orchestra
    scheme: https
    targetPort: 8443
    tlsConfig:
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - openunison
  selector:
    matchLabels:
      app: openunison-orchestra

Wait a few minutes, and you should see something similar to this in your openunison logs:

[2020-04-02 10:21:51,121][XNIO-1 task-10] INFO  AccessLog - [AuSuccess] - metrics - https://10.244.0.20:8443/metrics - username=system:serviceaccount:monitoring:prometheus-k8s,ou=oauth2,o=Tremolo - 20 / oauth2k8s [10.244.0.14] - [fefe1c571fbe12338abf0d5ba2fe4283a4a6d0def]
[2020-04-02 10:21:51,121][XNIO-1 task-10] INFO  AccessLog - [AzSuccess] - metrics - https://10.244.0.20:8443/metrics - username=system:serviceaccount:monitoring:prometheus-k8s,ou=oauth2,o=Tremolo -
           [10.244.0.14] - [fefe1c571fbe12338abf0d5ba2fe4283a4a6d0def]

You'll also see the metrics available in your Prometheus dashboard.

Customizing Orchestra

Orchestra is an application built on OpenUnison with several "opinions" on how you should manage authentication in your cluster. These opinions my be close to what you need, but not exact. In order to customize Orchestra you'll need:

  1. git
  2. OpenJDK 8
  3. Apache Maven
  4. Docker registry

First, fork the GitHub project from https://github.com/openunison that you want to start with. Next, make your edits. To deploy to a local Docker daemon that you want to then use to push to a registry:

mvn clean package
mvn compile jib:dockerBuild
docker tag image:version registry/image:version
docker push registry/image:version

If you have credentials to access a registry remotely and are not running docker locally, you can push the image directly to your registry:

mvn clean package
export OU_CONTAINER_DEST=registry/image:version
export OU_REG_USER=registry_user
export OU_REG_PASSWORD=registry_password
mvn compile jib:build

Air-Gap Implementation

To deploy OpenUnison in an air-gap environment you need to import for the following images into your local repository. The table notes the image, its function, the chart its referenced in and the Helm chart values to set to customize which image to use.

Image Function Chart Value Name
docker.io/tremolosecurity/openunison-k8s-operator OpenUnison Operator tremolo/openunison-operator image
docker.io/tremolosecurity/activemq-docker:latest ActiveMQ Image Maintained by Tremolo Security tremolo/openunison-* amq_image
docker.io/tremolosecurity/kubernetes-artifact-deployment:1.1.0 Check certificates, update them as needed tremolo/openunison-* cert_update_image
docker.io/tremolosecurity/openunison-* Orchestra Image tremolo/openunison-* image

Istio Ingress

Using OpenUnison with Istio instead of NGINX is supported. First create a Gateway object with a TLS certificate that covers all of the hosts OpenUnison is using. For instance:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: openunison-gateway
  namespace: openunison
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "ou-tls-certificate"
    hosts:
    - k8sou.apps.192-168-2-144.nip.io
    - k8sdb.apps.192-168-2-144.nip.io
    - k8sapi.apps.192-168-2-144.nip.io

The important parts of this configuration are that tls is SIMPLE, do NOT use passthrough SSL with sni. Also, make sure credentialName is a Secret in the istio-system namespace that has a valid certificate and key for the OpenUnison hosts. Next, create a DestinationRule that will tell Istio to send all traffic from the Gateway to OpenUnison via TLS and to create a cookie to maintain sticky sessions:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: openunison-tls-rule
  namespace: openunison
spec:
  host: openunison-orchestra
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: oustickysession
          path: /
          ttl: 0s
    tls:
      mode: SIMPLE

Finally, create VirtualService that will link the Gateway to OpenUnison:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: openunison-orchestra
  namespace: openunison
spec:
  gateways:
  - openunison-gateway
  hosts:
  - k8sou.apps.192-168-2-144.nip.io
  - k8sdb.apps.192-168-2-144.nip.io
  - k8sapi.apps.192-168-2-144.nip.io
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: openunison-orchestra
        port:
          number: 443

Supporting Red Hat Marketplace

Red Hat Marketplace doesn't support your Red Hat login information for pulling images. We're going to update the operator shortly to get around this issue but until then, you can follow this process:

If you already deployed an OpenUnison instance

Make a backup of your openunison object then delete it to clear it out. This will make the import easier.

Copy the OpenShift pull secret into your namespace

There's a global pull Secret stored in the openshift-config namespace. The ImageStream can't reference a Secret outside it's own namespace. Assuming you're deploying your OpenUnison into the openunison project, run the following command to create the pull Secret (if using a different project, go into that project first):

$ oc project openunison
$ oc get secret pull-secret -n openshift-config -o json | jq -M 'del(.metadata.namespace)' | oc create -f -

Once the Secret is created, create your openunison object. The operator will create the ImageStream for the builder image and use the correct Secret to pull the image.

Update the pull secret for the BuildConfig

Once BuildConfig is created by the operator, it needs to be told to use the pull-secret Secret to get the builder image from Red Hat. Assuming your openunison object is called test-openunison your patch command would be:

$ oc patch buildconfig openunison-test-openunison  -p '{"spec":{"strategy":{"sourceStrategy":{"pullSecret":{"name":"pull-secret"}}}}}'

At this point you can start a new build.

Add a custom footer
Add a custom sidebar
Clone this wiki locally