Provide support for using Prometheus scaler with Azure Monitor managed service for Prometheus (preview)

[tomkerkhove]

Proposal

Provide support for using Prometheus scaler with Azure Monitor managed service for Prometheus (preview) which relies on Azure AD authentication.

Use-Case

Customers using Azure Monitor managed service for Prometheus (preview) can scale their applications based on existing metrics.

Is this a feature you are interested in implementing yourself?

No

Anything else?

This requires the addition of Azure AD authentication, I'll dig in to the requirements for this.

[raggupta-ms]

Thankyou @tomkerkhove for opening this issue. I am starting to look into this support and would need some time to go through the details and come up with a plan. Will share the proppsal/idea for further discussion/brainstorming once I am there!

[tomkerkhove]

Great, thank you @raggupta-ms!

[raggupta-ms]

I spent some time on KEDA code and this is what I am thinking:

• Add support for Azure Workload Identity in the first iteration i.e. hoping to get released in March's release.

• Add support for Azure AD App Auth iteratively after the first version is shipped.

• Do not add support for Azure Pod Identity Auth since Pod Identity is deprecated in lieu of Workload identity. See these links for reference: https://learn.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity and https://github.com/Azure/aad-pod-identity#-announcement

• As a sample of ScaledObject and TriggerAuthtication, it will be very similar to how it looks today for Prometheus scaler and a sample is:
  

• KEDA will need label/annotation which is pre-existing to enable azure workload identity for any azure scaler, as described here: https://keda.sh/docs/2.9/authentication-providers/azure-ad-workload-identity/

• For the implementation of workload identity support:
  -- Use/enhance existing Azure workload identity code to get the token in required format (https://github.com/kedacore/keda/blob/main/pkg/scalers/azure/azure_aad_workload_identity.go)
  -- Add a new http round tripper for Azure auth that will utilize the code in above mentioned file azure_aad_workload_identity.go.
  -- The round tripper will ensure to inject the workload identity token in authorization header.
  -- Use this new Azure round tripper in existing Prometheus scaler i.e. if the auth in scaler config is set to azure workload identity (

  keda/pkg/scalers/scaler.go

  Line 95 in 22a5111

  PodIdentity kedav1alpha1.AuthPodIdentity

  ), use this azure round tripper instead of the one used currently in code (https://github.com/kedacore/keda/blob/main/pkg/scalers/prometheus_scaler.go#L94).
  -- Might need to change some code here and there such as auth config parsing in Prometheus scaler etc.
  -- Overall, doesn't seem to be a lot of changes.

@tomkerkhove, please share your comments on the above thought. Also, please add/tag other folks who can help reviewing.

[tomkerkhove]

• Add support for Azure Workload Identity in the first iteration i.e. hoping to get released in March's release.
• Add support for Azure AD App Auth iteratively after the first version is shipped.

This is OK for me, if we agree on it with other maintainers then we should track a new issue for it.

• Do not add support for Azure Pod Identity Auth since Pod Identity is deprecated in lieu of Workload identity. See these links for reference: learn.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity and Azure/aad-pod-identity#-announcement

I am on the fence on this one since we do support it for other Azure scalers and implementation should be minimal. Pod Identity is deprecated, but not archived yet so I'm leaning toward adding it. @JorTurFer @zroubalik?

• As a sample of ScaledObject and TriggerAuthtication, it will be very similar to how it looks today for Prometheus scaler and a sample is:
  

Correct, this should work as an end outcome.

• KEDA will need label/annotation which is pre-existing to enable azure workload identity for any azure scaler, as described here: keda.sh/docs/2.9/authentication-providers/azure-ad-workload-identity

This is already a pre-requisite so a non-action for you.

• For the implementation of workload identity support:
  -- Use/enhance existing Azure workload identity code to get the token in required format (main/pkg/scalers/azure/azure_aad_workload_identity.go)
  -- Add a new http round tripper for Azure auth that will utilize the code in above mentioned file azure_aad_workload_identity.go.
  -- The round tripper will ensure to inject the workload identity token in authorization header.
  -- Use this new Azure round tripper in existing Prometheus scaler i.e. if the auth in scaler config is set to azure workload identity (

  keda/pkg/scalers/scaler.go

  Line 95 in 22a5111

  PodIdentity kedav1alpha1.AuthPodIdentity

  
  ), use this azure round tripper instead of the one used currently in code (main/pkg/scalers/prometheus_scaler.go#L94).
  -- Might need to change some code here and there such as auth config parsing in Prometheus scaler etc.

@JorTurFer @zroubalik can you validate this, Sounds legit to me though.

-- Overall, doesn't seem to be a lot of changes.

I agree, changes should be easy.

[JorTurFer]

• Add support for Azure Workload Identity in the first iteration i.e. hoping to get released in March's release.
• Add support for Azure AD App Auth iteratively after the first version is shipped.

This is OK for me, if we agree on it with other maintainers then we should track a new issue for it.

I have no idea about what is Azure AD App auth, but it looks good to me

I am on the fence on this one since we do support it for other Azure scalers and implementation should be minimal. Pod Identity is deprecated, but not archived yet so I'm leaning toward adding it. @JorTurFer @zroubalik?

Agree with @tomkerkhove , if AAD-Pod-Identity is still alive (its EOL is December if I'm not wrong), we should support both. The best part is that we already have the code to get the token (like for AAD-WI), so the extra required code should be low

About the implementation, sound good in my mind, but I need to see the code to have a real opinion about it. From my side, you can go ahead.

Remember that e2e test is also required as part of the feature. You will need to do another PR to the testing infra repository, adding the needed terraform to deploy the managed prometheus and performing the role assignment required to grant permissions to the e2e managed identity

[raggupta-ms]

Thankyou @tomkerkhove and @JorTurFer for reviweing. I didn't have strong opinion on not adding support for Pod Identity. Since you feel it should be there (I also agree somewhat that it is going to stick in for probably around an year more), I will add that as well.

Will proceed to make code changes at my end. Noted the testing part, I may have more questions, especially around testing when I reach there. Thanks for the support!

[tomkerkhove]

I have no idea about what is Azure AD App auth, but it looks good to me

@JorTurFer This is just Azure AD authentication with a client app

[raggupta-ms]

@tomkerkhove @JorTurFer I am facing an issue while trying to setup my VSCode for debugging and could use your help, in case you know.

I am following the steps listed here: https://github.com/kedacore/keda/blob/main/BUILD.md#operator but when I launch VS Code, I get following error:

{"level":"error","ts":"2023-02-12T00:28:16-08:00","logger":"setup","msg":"problem running manager","error":"open /certs/ca.crt: no such file or directory","stacktrace":"main.main\n\t/home/raggupta/code/keda/cmd/operator/main.go:284\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

Looks like the operator is trying to get hold of ca and tls certs. Is this expected when debugging locally? Is there a way to disable tls on local setup? I am using minikube cluster. I could find some annotation related to tls cert in keda-metrics-apiserver yaml file and removed all of it, annotated with --kubelet-insecure-tls and --kubelet-preferred-address-types="InternalIP", but very likely I am looking at wrong place since I want to debug operator but these config is for api server.

I tried to supply a ca.crt at the mentioned location but then it asked for tls.crt etc. and I could not resolve that completely - am no expert on tls and certs :(

Any idea what I might be doing wrong or to get it working? Note that when I deploy by scaling keda operator to 0 and then make deploy, it works fine - I don't see any cert issues in logs and keda containers are up and running fine. Just that VS Code launch isn't working.

[JorTurFer]

Oh! Sorry, this has been my fault :(
I'll update the docs asap, in the meantime, you can get the certificates that the operator has created in the cluster (it's a secret in KEDA namespace, you will find them together inside the secret) and create the files in that folder
In the cluster it works because the secret is automatically mounted into the pod

[JorTurFer]

Hi @raggupta-ms ,
I have just opened a PR with the information you need. It sin't merged yet, but you can check the process directly there.

Thanks for notifying the documentation gap :)

[raggupta-ms]

@JorTurFer thankyou for the instructions. It did the resolve the issue :)

[JorTurFer]

no worries, it was missing due to my fault xD

[raggupta-ms]

created a PR: #4256
@tomkerkhove @JorTurFer Please take a look.

[tomkerkhove]

Docs and feature are in, we are good to go!

[tomkerkhove]

Thank you @raggupta-ms !