Deployment is not scaling down when using podidentity for Azure Eventhubs #2808

Harisyam · 2022-03-23T11:04:07Z

Report

We have created a java app on aks based on eventhub trigger, and we are using keda for autoscaling.

When we used the connection strings directly in the deployments the autoscaling and scale down worked properly for eventhub trigger. But when we used the podidentity for scaled object, it is scaling up but scaling down is not happening and the HPA metric is showing higher number even after all the checkpoints are done.

find the below yml files

Deployment.yml

kind: Deployment
metadata:
  name: events-processor-deployment
spec:
  selector:
    matchLabels:
      app: events-processor
  template:
    metadata:
      labels:
        app: events-processor
        # aadpodidbinding: function-apps-binding
    spec:
    
      containers:
      - name: events-processor-container
        image: kubepocmyacr.azurecr.io/sample-image:1.0
        env:
        - name: BLOB_STORAGE_CONNECTION
          value: <Placeholder Connection String>
        # JAVA OPTS
        - name: JAVA_OPTS
          value: "-XX:MaxRAMPercentage=75.0"
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "2.5Gi"
            cpu: "1500m"`

TriggerAuthentication.yml

kind: TriggerAuthentication
metadata:
  name: azure-eventhub-trigger-auth
  namespace: keda-system

spec:
  podIdentity:
    provider: azure

ScaledObject.yml

kind: ScaledObject
metadata:
  name: azure-eventhub-spring-scaledobject
  namespace: keda-system
spec:
  scaleTargetRef:
    name: events-processor-deployment
  minReplicaCount: 1
  maxReplicaCount: 5
  cooldownPeriod: 30
  pollingInterval: 15
  triggers:
  - type: azure-eventhub
    metadata:
      # Required
      storageConnectionFromEnv: BLOB_STORAGE_CONNECTION
      eventHubNamespace: poceventhubcosmos
      eventHubName: demo-analysis-eventhub
      consumerGroup: $Default
      unprocessedEventThreshold: '50'
      checkpointStrategy: 'blobMetadata'
      blobContainer: 'kubernetes-poc'
    authenticationRef:
      name: azure-eventhub-trigger-auth

### Expected Behavior

Keda should be able to scale down the deployment even when we use pod identity

### Actual Behavior

Keda is not able to scale down the deployment when using podidentity, even though we are checkpoint every message, And the HPA metrics are showing higher number.



### Steps to Reproduce the Problem


This is working as expected when we use below metadata in scaledobject

metadata:
  storageConnectionFromEnv: BLOB_STORAGE_CONNECTION
  connectionFromEnv: EVENTHUB_CONNECTION
  consumerGroup: $Default
  unprocessedEventThreshold: '50'
  checkpointStrategy: 'blobMetadata'
  blobContainer: 'kubernetes-poc'


But when we use the below metadata for pod identity, its not working as expected

metadata:
  storageConnectionFromEnv: BLOB_STORAGE_CONNECTION
  eventHubNamespace: poceventhubcosmos
  eventHubName: demo-analysis-eventhub
  consumerGroup: $Default
  unprocessedEventThreshold: '50'
  checkpointStrategy: 'blobMetadata'
  blobContainer: 'kubernetes-poc'
authenticationRef:
  name: azure-eventhub-trigger-auth


### Logs from KEDA operator

Keda operator logs

1.648033250782659e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.648033250782781e+09 INFO controller.scaledobject Creating a new HPA {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system", "HPA.Namespace": "keda-system", "HPA.Name": "keda-hpa-azure-eventhub-spring-scaledobject"}

1.648033251146449e+09 INFO controller.scaledobject Initializing Scaling logic according to ScaledObject Specification {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.6480332511549592e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.6480332557244997e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.6480332696407406e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.6480332853822763e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}




Keda operator metrics logs

2022-03-23T16:31:09+05:30 I0323 11:01:09.589547 1 trace.go:205] Trace[937030477]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/keda-system/s0-azure-eventhub-$default,user-agent:kube-controller-manager/v1.21.9 (linux/amd64) kubernetes/37f338a/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:35c82ed1-8f1f-4669-a9a9-b12b90ed2930,client:172.31.58.69,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (23-Mar-2022 11:01:06.213) (total time: 3375ms):

2022-03-23T16:31:09+05:30 Trace[937030477]: ---"Listing from storage done" 3375ms (11:01:09.589)

2022-03-23T16:31:09+05:30 Trace[937030477]: [3.375537885s] [3.375537885s] END

2022-03-23T16:31:25+05:30 I0323 11:01:25.367790 1 trace.go:205] Trace[407449632]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/keda-system/s0-azure-eventhub-$default,user-agent:kube-controller-manager/v1.21.9 (linux/amd64) kubernetes/37f338a/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:4cc1c7be-41c0-43a2-9113-77324ca64220,client:172.31.58.69,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (23-Mar-2022 11:01:24.691) (total time: 676ms):

2022-03-23T16:31:25+05:30 Trace[407449632]: ---"Listing from storage done" 676ms (11:01:25.367)

2022-03-23T16:31:25+05:30 Trace[407449632]: [676.120576ms] [676.120576ms] END

2022-03-23T16:31:40+05:30 I0323 11:01:40.982286 1 trace.go:205] Trace[1154341616]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/keda-system/s0-azure-eventhub-$default,user-agent:kube-controller-manager/v1.21.9 (linux/amd64) kubernetes/37f338a/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:0d0f7cf8-e718-4333-b5e8-3cef73bb8eca,client:172.31.58.69,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (23-Mar-2022 11:01:40.421) (total time: 560ms):

2022-03-23T16:31:40+05:30 Trace[1154341616]: ---"Listing from storage done" 560ms (11:01:40.982)


### KEDA Version

2.6.1

### Kubernetes Version

1.21

### Platform

Microsoft Azure

### Scaler Details

Azure Eventhub

### Anything else?

We think that it is not able to use the storage connection environment variable when we use pod identity for eventhub credentials

We followed below documentation for implementing this:
https://keda.sh/docs/2.6/scalers/azure-event-hub/

The text was updated successfully, but these errors were encountered:

JorTurFer · 2022-03-23T19:07:50Z

hi @Harisyam
Could you try querying the metric manually to double-check that it's working and getting the values?
It'd be nice if you could try in both scenarios

Harisyam · 2022-03-24T12:03:22Z

Hi @JorTurFer

Please find the below metrics

Metrics with pod identity approach

{"kind":"ExternalMetricValueList","apiVersion":"external.metrics.k8s.io/v1beta1","metadata":{},"items":[{"metricName":"s0-azure-eventhub-read","metricLabels":null,"timestamp":"2022-03-24T11:31:37Z","value":"2k"}]}

Metrics with connection string approach

{"kind":"ExternalMetricValueList","apiVersion":"external.metrics.k8s.io/v1beta1","metadata":{},"items":[{"metricName":"s0-azure-eventhub-read","metricLabels":null,"timestamp":"2022-03-24T11:55:08Z","value":"0"}]}

JorTurFer · 2022-03-24T18:06:03Z

This is weird...
This could sound super obvious, but, have you double-check if both cases use the same event hub? I mean, in both cases you have to provide the event hub name and namespace, maybe by error connection string uses another. If they are the same, doesn't make sense to get different values :/

Harisyam · 2022-03-24T18:20:39Z

We have used the same event hub in both the cases.
Only difference was in scaled object we haven't mentioned event hub name and namespace for the connection string approach, But that scaled down properly.

v-shenoy · 2022-03-25T07:14:09Z

This seems to be a combination of the two following issues.
#2781
#2363

When using podIdentity, the scaler is unable to find the storage blob, and the logs are missing in release 2.6.1 so the operator doesn't report anything. Changes for both have been merged as parts of #2788 and #2717, and the error should be fixed in the upcoming release.

Harisyam · 2022-03-25T07:23:07Z

Hi @v-shenoy,

If it is resolved when are we expecting a release for it

v-shenoy · 2022-03-25T07:24:41Z

@JorTurFer @kedacore/keda-contributors can answer that.

zroubalik · 2022-03-25T08:06:35Z

2.7.0 is due early May.

Harisyam · 2022-03-25T11:49:00Z

Hi @zroubalik,

Thanks for the update.

Is there any workaround for this issue? if not can it get fixed on any minor version?

zroubalik · 2022-03-25T12:13:30Z

I am not sure about the workaround, but unfortunately we don't plan to release any minor version.

You can use images with main tag which follows the main git branch, at your own risk, but I am not sure this is something you'd like to run on production.

JorTurFer · 2022-03-25T13:40:05Z

do you think @v-shenoy that this issue can be closed because it's already solved with the other PRs?

v-shenoy · 2022-03-25T13:47:27Z

@JorTurFer I think so.

@Harisyam It seems that you're doing a poc first. So, would it be possible for you to test if the scaling works with a KEDA image from the main branch? It would help us confirm that the issue is resolved.

Harisyam · 2022-03-28T05:21:31Z

@v-shenoy @JorTurFer

Sure, We will do a test with the image from main branch and let you know about this

Harisyam · 2022-03-31T09:45:00Z

Hi @v-shenoy, @JorTurFer ,

We tried using the image from main and it is working fine.

JorTurFer · 2022-03-31T10:14:04Z

nice! In that case I close this ticket.
Thanks for checking it

Harisyam added the bug Something isn't working label Mar 23, 2022

JorTurFer closed this as completed Mar 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deployment is not scaling down when using podidentity for Azure Eventhubs #2808

Deployment is not scaling down when using podidentity for Azure Eventhubs #2808

Harisyam commented Mar 23, 2022 •

edited

Loading

JorTurFer commented Mar 23, 2022

Harisyam commented Mar 24, 2022

JorTurFer commented Mar 24, 2022

Harisyam commented Mar 24, 2022 •

edited

Loading

v-shenoy commented Mar 25, 2022 •

edited

Loading

Harisyam commented Mar 25, 2022

v-shenoy commented Mar 25, 2022 •

edited

Loading

zroubalik commented Mar 25, 2022

Harisyam commented Mar 25, 2022

zroubalik commented Mar 25, 2022

JorTurFer commented Mar 25, 2022

v-shenoy commented Mar 25, 2022

Harisyam commented Mar 28, 2022

Harisyam commented Mar 31, 2022

JorTurFer commented Mar 31, 2022

Deployment is not scaling down when using podidentity for Azure Eventhubs #2808

Deployment is not scaling down when using podidentity for Azure Eventhubs #2808

Comments

Harisyam commented Mar 23, 2022 • edited Loading

Report

JorTurFer commented Mar 23, 2022

Harisyam commented Mar 24, 2022

JorTurFer commented Mar 24, 2022

Harisyam commented Mar 24, 2022 • edited Loading

v-shenoy commented Mar 25, 2022 • edited Loading

Harisyam commented Mar 25, 2022

v-shenoy commented Mar 25, 2022 • edited Loading

zroubalik commented Mar 25, 2022

Harisyam commented Mar 25, 2022

zroubalik commented Mar 25, 2022

JorTurFer commented Mar 25, 2022

v-shenoy commented Mar 25, 2022

Harisyam commented Mar 28, 2022

Harisyam commented Mar 31, 2022

JorTurFer commented Mar 31, 2022

Harisyam commented Mar 23, 2022 •

edited

Loading

Harisyam commented Mar 24, 2022 •

edited

Loading

v-shenoy commented Mar 25, 2022 •

edited

Loading

v-shenoy commented Mar 25, 2022 •

edited

Loading