Adding SSL enabled Kafka Cluster (Bitnami) with PEM certificate in KLAW kubernetes deployment.

[ss85705]

Hi @aindriu-aiven @muralibasani

I am using kubernetes deployment file for deploying KLAW. I am able to connect PLAINTEXT kafka clusters but unable to deploy Cluster api and klaw core with PEM environment variables.

i am using below env in my kubernetes deployment.

• name: DEV1_KAFKASSL_KEYSTORE_TYPE
  value: "PEM"
• name: DEV1_KAFKASSL_KEYSTORE_CERTIFICATE_CHAIN
  value: "klaw-tls-cert"
• name: DEV1_KAFKASSL_KEYSTORE_KEY
  value: "klaw-tls-cert"

This config is giving parsing error in deployment file.
I am taking reference from one of the post about PEM support
#511

[aindriu-aiven]

Hi @ss85705 I am just looking at this now to see if I can spot any issues.

[aindriu-aiven]

@ss85705 Can I just check where you got dev1 from?

That prefix is normally from
the "Clusters" page as superadmin

Select the clipboard icon on the right of the kafka cluster you want to configure, that will allow you to get the prefix required for the configuration.

Also the environment variables need to be attached to the cluster api as that is what will communicate with the kafka clusters.

Aside from that the configuration looks correct although I will revert to @muralibasani to see if he can add any additional information.

If you see any error in the log when you test the connection please revert that back to us (minus any urls) and I will see if we can make any more progress diagnosing this issue.

[ss85705]

@aindriu-aiven THanks for your response.
I understood your point here.

But , not sure if the configuration is valid for kubernetes deployments. I tried that but then Pod is going into waiting state and right after removing the configuration pods become fine.

So not sure, if we need to have some kind of configuration changes or somethig.

Also, for container deployments, can we check for config map sort of things where we can deploy seperate config files without modifying the deployments.

[aindriu-aiven]

@ss85705 You should be able to make alterations to the configmap without redeploying the instance however to getKlaw to pick up those changes it would need to restart to pull in those differences.

Is it possible the pods are unable to access the certificate or other configured resource after you make the change leaving it in a waiting state?

I will try to setup a k8s deployment myself and see if I can navigate through a few of these issues.

[ss85705]

Thanks @aindriu-aiven
I will try but if you can provide some k8s reference , would be really helpful. i am working on it and need to finalize. Thanks

[aindriu-aiven]

Hey @ss85705 I spent some time yesterday to find the best way to externalize configuration for Klaw in K8s.

Their are a few ways to do so, but the cleanest way available to us at the moment, is to use the envFrom method.

e.g. create your configmap with your configuration the core-config is an example of how to define the variables you want to alter in this case it is the postgres connection..
core-config.txt

Then in the deploy-kubernetes.yaml the environment variables are replaced with envFrom this loads all of the data from the config maps making changes to the configmap will require you to scale your pods to zero and then back to 1

Deployment Klaw-core e.g.

apiVersion: apps/v1
kind: Deployment
metadata:
name: klaw-core
spec:
replicas: 1
selector:
matchLabels:
app: klaw-core
template:
metadata:
labels:
app: klaw-core
spec:
containers:
- name: klaw-core
image: aivenoy/klaw-core:latest
imagePullPolicy: Never
envFrom:
- configMapRef:
name: core-config
volumeMounts:
- name: klaw-data
mountPath: /klaw
volumes:
- name: klaw-data
persistentVolumeClaim:
claimName: klaw-data