Exploration for Kubeflow Dashboard (HTTPs) how-to guide

[orfeas-k]

Context

Explore the implementation needed for https://warthogs.atlassian.net/browse/KF-5665

What needs to get done

Explore how the implementation needed can be achieved in order to be able then to write a how-to guide

Definition of Done

Exploration has been done and findings are documented in the issue.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5919.

This message was autogenerated

[misohu]

Today I went through this guide and made some more experiments on my side to answer some questions from the parent epic. The guide was nicely written but still does not answer all the questions from Acceptance criteria section (listed and described bellow).

To sum up there are following charms:

• self-signed-certificates-operator. This one generates the certificates and sends those to related charms. (this is only for development so I will skip it in the answers bellow).
• tsl-certificates-operator. This one allows us to send custom certificates to related charms.
• httpreq-acme-operator, namecheap-acme-operator, or route53-acme-operator. These are used with ACME servers for validation (like Let’s encrypt).

Today I went through the first two options. To test the third option we need to register a name for our kubeflow deployment with the CA authority which will provide the ACME servers. Should we do this in our explanation guide? Rest on the answers today are based on the first two charms.

I can enable and disable the TLS

This is done easily by relating and un-relating the provider charm with istio-pilot. We can write a short section about that in final doc.

I can read about the benefits of using the TLS

We can add section explaining benefits of HTTPs over HTTP (e.g. from here).

I can read about the versions of Kubeflow that TLS works

Today I tested on 1.8 and latest/beta

I can read about the environment (on prem, cloud) which TLS works

In both cases the setup can be simply accomplished with the second charm by providing the certificates manually. The setting with ACME might be problematic, but each of those ACME charms have tutorials explaining details (e.g. for route 53 AWS).

I can change the TLS certificate if needed

The same way I provide certificate in the second charm the same way I can update. For the ACME this is handled by the ACME server.

I can read about for how long it is available (or when it expire)

This is hard to answer in second charm user creates the certificate and they decide how long it will be available. In third group of charms its the setting of the ACME server.

I can read about possible constraints

Not sure about constraints will need to sync with team

I can integrate with a TLS provider

What provider? Should we really do this in kubeflow guide? The setting of provider can be really specific.

I can read about examples of TLS providers that can be integrated with / used

Same as above

Test 1 self-signed-certificates-operator

• I have deployed kubeflow
• I have deployed self-signed-certificates-operator
• I related those together

juju relate istio-pilot:certificates self-signed-certificates:certificates

• I could connect to dashboard url prefixed with https:// (after enabling unsecure certificate in browser)

Test 2 self-signed-certificates-operator

• I have deployed kubeflow
• I have deployed tsl-certificates-operator.
• Relate

juju relate istio-pilot:certificates manual-tls-certificates:certificates

• Get csr and relation info

juju run --wait=2m  manual-tls-certificates/leader get-outstanding-certificate-requests

• Save csr to file csr.pem
• Run this

# Install OpenSSL
sudo apt update
sudo apt install openssl

# Create a Private Key for the CA
openssl genrsa -out ca.key 2048

# Create a Self-Signed Certificate for the CA
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.crt

# Sign the CSR with the CA Certificate and Key
openssl x509 -req -in csr.pem -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256

# Verify the Certificate
openssl x509 -in server.crt -text -noout

• base encode

base64 -w0 server.crt > certificate.pem.b64
base64 -w0 ca.crt > ca-chain.pem.b64
base64 -w0 ca.crt > ca-certificate.pem.b64
base64 -w0 csr.pem > csr.pem.b64

• Manual submit

juju run manual-tls-certificates/leader provide-certificate \
  relation-id=<id> \
  certificate="$(cat certificate.pem.b64)" \
  ca-chain="$(cat ca-chain.pem.b64)" \
  ca-certificate="$(cat ca-certificate.pem.b64)" \
  certificate-signing-request="$(cat csr.pem.b64)"

• config dex

juju config dex-auth public-url=https://10.64.140.43.nip.io
juju config oidc-gatekeeper public-url=https://10.64.140.43.nip.io

• I could connect to dashboard url prefixed with https:// (after enabling unsecure certificate in browser)

[misohu]

Sync with @kimwnasptd

Using Clouds k8s (EKS AKS)

• for now we assume its on k8s service not vm + microk8s (sync with andreea)

• Tell user to create a domain name for the kubeflow dashboard (point to public cloud docs)
  • give them overview what is happening.
• Tell them to get the certificate for that domain name (point to AWS Certificate Manager guide and sth for azure or just tell to use the once they have in company).
• Show user how to use manual-tls-charm to provide the certificate to istio-pilot.
• No need to talk about acme servers and charms

• sync with Diko about how he setup the TLS (what is the info he would need for azure)

[misohu]

So how managed solution handles HTTPS in Azure

• deploy cos and kubeflow
• deploy VM with nginx
• register dynamic DNS record <NAME>.<REGION>.cloudapp.azure.com and first they register it with the VM's ip
• register certificate form the VM (with nginx) to the DNS (bash script here :) )
• sent the certificates to istio and traefik
• point dns record to kubeflow + traefik

Rotation:

• change dns to vm with nginx
• regenerate certificate
• send certificates to istio traefik
• change dns to istio traefik

The TLS for dynamic dns in azure is 12 second ... so every 3 months there is a chance for 12 seconds that the user will see welcome to the nginx message :)

Notes
Answer why cannot we use manual TLS charm for distributing lets encrypt certificates.
Answer why we cant use HTTPRequest LEGO K8s charm

AWS
They do do it there rn and they have some concerns about route53 service. From my digging the same functionality is supported for route 53.

[misohu]

At the end we agreed

• to move away the information about tls charms as they are currently not usable because of the things from previous comment
• keep just the section about istio config options for tls setup on istio
• add prerequisites section about FQDN dns creation and TLS certificate creation with links for different cloud providers.

[DnPlas]

At the end we agreed

* to move away the information about tls charms as they are currently not usable because of the things from previous comment

* keep just the section about istio config options for tls setup on istio

* add prerequisites section about FQDN dns creation and TLS  certificate creation with links for different cloud providers.

Hey @misohu @kimwnasptd I can see the section of how to integrate with TLS certificates providers was removed, but it is not clear to me why it was removed. Do you mind sharing the arguments in favour of not exposing this information?

Also, I have some notes for the published guide:

1. Passing the certificates directly to the istio-pilot charm is a feature that will only be supported until istio-operators 1.22, meaning that by CKF 1.10 (when we bump Istio), this feature will not be available any more. This has been communicated everywhere in our docs and contributions (see here, here). It was also part of an epic, but I cannot seem to find it. It was also on the old guide:

@kimwnasptd did this changed recently?

2. We do not have any other way to tell our users how to use TLS certificate providers, which is something some users (customers included) have used in the past. Is this information placed somewhere else?

3. Tiny note on the guide:

This section is confusing. I'd suggest we change the title to "Migrating from configuration to juju secret".

Also, this paragraph is not clear:

Passing the TLS certificate and key using Juju secrets means replacing the ssl-* configuration options for the istio-pilot . This migration is as simple as doing:

This section is supposed to help people migrate from the old configuration implementation (passing values to ssl-key and ssl-crt config options) to the new one, passing those values as a juju secret, but this first paragraph does not make this clear (at least it does not to me). Maybe rephrase it a bit so it is clear why and when people have to do it? You can get inspiration from the old guide.

[misohu]

Hey @DnPlas ... The charms section was removed because currently none of them are usable (tested on azure cloud).

Here I will put again the issues describing the problems with those charms

• Charm should have the option to provide externally issued certificates manual-tls-certificates-operator#251 for manueal tls
• Charm should have the option to provide externally issued certificates manual-tls-certificates-operator#251 (comment) for lego charm

Moreover none of the charms provides guides on how to set them up with any certificate provider.

The only way to setup https which was tested was through the configuration on istio-pilot. With @kimwnasptd we decided to keep only that part