Failed
=== RUN   TestAWSSuite/TestALBExperimentStepNoSetWeightMultiIngress
time="2023-11-08T01:03:41Z" level=info msg="Deleting e2e-test-name=TestALBExperimentStepNoSetWeightMultiIngress"
time="2023-11-08T01:03:41Z" level=info msg="service/alb-rollout-root created\n" rollout=alb-rollout
time="2023-11-08T01:03:41Z" level=info msg="service/alb-rollout-canary created\n" rollout=alb-rollout
time="2023-11-08T01:03:42Z" level=info msg="service/alb-rollout-stable created\n" rollout=alb-rollout
time="2023-11-08T01:03:42Z" level=info msg="ingress.networking.k8s.io/alb-rollout-multi-ingress-1 created\n" rollout=alb-rollout
time="2023-11-08T01:03:42Z" level=info msg="ingress.networking.k8s.io/alb-rollout-multi-ingress-2 created\n" rollout=alb-rollout
time="2023-11-08T01:03:42Z" level=info msg="rollout.argoproj.io/alb-rollout created\n" rollout=alb-rollout
time="2023-11-08T01:03:42Z" level=info msg="Waiting for condition: status=Healthy" rollout=alb-rollout
time="2023-11-08T01:03:44Z" level=info msg="Condition 'status=Healthy' met after 1s" rollout=alb-rollout
time="2023-11-08T01:03:44Z" level=info msg="Experiment expectation 'experiment count == 0' met" rollout=alb-rollout
time="2023-11-08T01:03:44Z" level=info msg="Updated rollout pod spec: 2023-11-08T01:03:44.268068438Z" rollout=alb-rollout
time="2023-11-08T01:03:44Z" level=info msg="Sleeping 10s" rollout=alb-rollout
    require.go:72: 
        	Error Trace:	/home/runner/go/pkg/mod/github.com/tj/assert@v0.0.3/require.go:72
        	            				/home/runner/work/argo-rollouts/argo-rollouts/test/e2e/aws_test.go:298
        	            				/home/runner/work/argo-rollouts/argo-rollouts/test/fixtures/then.go:28
        	            				/home/runner/work/argo-rollouts/argo-rollouts/test/e2e/aws_test.go:287
        	Error:      	Not equal: 
        	            	expected: "{\"Type\":\"forward\",\"ForwardConfig\":{\"TargetGroups\":[{\"ServiceName\":\"alb-rollout-canary\",\"ServicePort\":\"80\",\"Weight\":0},{\"ServiceName\":\"\",\"ServicePort\":\"80\",\"Weight\":20},{\"ServiceName\":\"\",\"ServicePort\":\"80\",\"Weight\":20},{\"ServiceName\":\"alb-rollout-stable\",\"ServicePort\":\"80\",\"Weight\":60}]}}"
        	            	actual  : "{\"Type\":\"forward\",\"ForwardConfig\":{\"TargetGroups\":[{\"ServiceName\":\"alb-rollout-canary\",\"ServicePort\":\"80\",\"Weight\":0},{\"ServicePort\":\"80\",\"Weight\":20},{\"ServicePort\":\"80\",\"Weight\":20},{\"ServiceName\":\"alb-rollout-stable\",\"ServicePort\":\"80\",\"Weight\":60}]}}"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-{"Type":"forward","ForwardConfig":{"TargetGroups":[{"ServiceName":"alb-rollout-canary","ServicePort":"80","Weight":0},{"ServiceName":"","ServicePort":"80","Weight":20},{"ServiceName":"","ServicePort":"80","Weight":20},{"ServiceName":"alb-rollout-stable","ServicePort":"80","Weight":60}]}}
        	            	+{"Type":"forward","ForwardConfig":{"TargetGroups":[{"ServiceName":"alb-rollout-canary","ServicePort":"80","Weight":0},{"ServicePort":"80","Weight":20},{"ServicePort":"80","Weight":20},{"ServiceName":"alb-rollout-stable","ServicePort":"80","Weight":60}]}}
        	Test:       	TestAWSSuite/TestALBExperimentStepNoSetWeightMultiIngress
Name:            alb-rollout
Namespace:       default
Status:          �[94m◌�[0m Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/1
  SetWeight:     0
  ActualWeight:  0
Images:          nginx:1.19-alpine (�[32mstable�[0m, �[0mΣ:experiment-alb-canary�[0m, �[0mΣ:experiment-alb-stable�[0m)
Replicas:
  Desired:       1
  Current:       1
  Updated:       0
  Ready:         1
  Available:     1

NAME                                                                  KIND        STATUS         AGE  INFO
⟳ alb-rollout                                                         Rollout     �[94m◌�[0m Progressing  12s  
├──# revision:2                                                                                       
│  ├──⧉ �[33malb-rollout-6d6c8558b4�[0m                                        ReplicaSet  �[0m•�[0m ScaledDown   10s  �[33mcanary�[0m
│  └──Σ �[94malb-rollout-6d6c8558b4-2-0�[0m                                    Experiment  �[94m◌�[0m Running      10s  
│     ├──⧉ alb-rollout-6d6c8558b4-2-0-experiment-alb-canary           ReplicaSet  �[32m✔�[0m Healthy      10s  
│     │  └──□ alb-rollout-6d6c8558b4-2-0-experiment-alb-canary-54pn8  Pod         �[32m✔�[0m Running      10s  ready:1/1
│     └──⧉ alb-rollout-6d6c8558b4-2-0-experiment-alb-stable           ReplicaSet  �[32m✔�[0m Healthy      10s  
│        └──□ alb-rollout-6d6c8558b4-2-0-experiment-alb-stable-z7db5  Pod         �[32m✔�[0m Running      10s  ready:1/1
└──# revision:1                                                                                       
   └──⧉ �[32malb-rollout-d6f4fdcdc�[0m                                         ReplicaSet  �[32m✔�[0m Healthy      12s  �[32mstable�[0m
      └──□ alb-rollout-d6f4fdcdc-447gn                                Pod         �[32m✔�[0m Running      12s  ready:1/1

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    rollout.argoproj.io/revision: "2"
  creationTimestamp: "2023-11-08T01:03:42Z"
  generation: 3
  labels:
    argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e
    e2e-test-name: TestALBExperimentStepNoSetWeightMultiIngress
  name: alb-rollout
  namespace: default
  resourceVersion: "4296"
  uid: 7d48d1fc-4e74-40c6-97ce-fc5ec14de487
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alb-rollout
  strategy:
    canary:
      canaryService: alb-rollout-canary
      stableService: alb-rollout-stable
      steps:
      - experiment:
          duration: 15s
          templates:
          - metadata: {}
            name: experiment-alb-canary
            specRef: canary
            weight: 20
          - metadata: {}
            name: experiment-alb-stable
            specRef: stable
            weight: 20
      trafficRouting:
        alb:
          ingresses:
          - alb-rollout-multi-ingress-1
          - alb-rollout-multi-ingress-2
          rootService: alb-rollout-root
          servicePort: 80
  template:
    metadata:
      annotations:
        update: "2023-11-08T01:03:44.268068438Z"
      creationTimestamp: null
      labels:
        app: alb-rollout
    spec:
      containers:
      - image: nginx:1.19-alpine
        name: alb-rollout
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 16Mi
status:
  HPAReplicas: 1
  availableReplicas: 1
  blueGreen: {}
  canary:
    currentExperiment: alb-rollout-6d6c8558b4-2-0
    weights:
      additional:
      - weight: 20
      - weight: 20
      canary:
        podTemplateHash: 6d6c8558b4
        serviceName: alb-rollout-canary
        weight: 0
      stable:
        podTemplateHash: d6f4fdcdc
        serviceName: alb-rollout-stable
        weight: 60
  conditions:
  - lastTransitionTime: "2023-11-08T01:03:44Z"
    lastUpdateTime: "2023-11-08T01:03:44Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  - lastTransitionTime: "2023-11-08T01:03:44Z"
    lastUpdateTime: "2023-11-08T01:03:44Z"
    message: Rollout is not healthy
    reason: RolloutHealthy
    status: "False"
    type: Healthy
  - lastTransitionTime: "2023-11-08T01:03:44Z"
    lastUpdateTime: "2023-11-08T01:03:44Z"
    message: RolloutCompleted
    reason: RolloutCompleted
    status: "False"
    type: Completed
  - lastTransitionTime: "2023-11-08T01:03:42Z"
    lastUpdateTime: "2023-11-08T01:03:44Z"
    message: Created new replica set "alb-rollout-6d6c8558b4"
    reason: NewReplicaSetCreated
    status: "True"
    type: Progressing
  currentPodHash: 6d6c8558b4
  currentStepHash: 558cc8bf4d
  currentStepIndex: 0
  message: more replicas need to be updated
  observedGeneration: "3"
  phase: Progressing
  readyReplicas: 1
  replicas: 1
  selector: app=alb-rollout
  stableRS: d6f4fdcdc

2023-11-08T01:03:42Z         Normal    RolloutAddedToInformer    Rollout resource added to informer: default/alb-rollout
2023-11-08T01:03:42Z         Normal    RolloutNotCompleted       Rollout not completed, started update to revision 1 (d6f4fdcdc)
2023-11-08T01:03:42Z         Normal    RolloutUpdated            Rollout updated to revision 1
2023-11-08T01:03:42Z         Normal    NewReplicaSetCreated      Created ReplicaSet alb-rollout-d6f4fdcdc (revision 1)
2023-11-08T01:03:42Z (x2)    Normal    TrafficWeightUpdated      Traffic weight updated to 0
2023-11-08T01:03:42Z         Normal    ScalingReplicaSet         Scaled up ReplicaSet alb-rollout-d6f4fdcdc (revision 1) from 0 to 1
2023-11-08T01:03:42Z         Normal    RolloutCompleted          Rollout completed update to revision 1 (d6f4fdcdc): Initial deploy
2023-11-08T01:03:44Z         Normal    SwitchService             Switched selector for service 'alb-rollout-stable' from '' to 'd6f4fdcdc'
2023-11-08T01:03:44Z         Normal    SwitchService             Switched selector for service 'alb-rollout-canary' from '' to 'd6f4fdcdc'
2023-11-08T01:03:44Z         Normal    RolloutNotCompleted       Rollout not completed, started update to revision 2 (6d6c8558b4)
2023-11-08T01:03:44Z         Normal    RolloutUpdated            Rollout updated to revision 2
2023-11-08T01:03:44Z         Normal    NewReplicaSetCreated      Created ReplicaSet alb-rollout-6d6c8558b4 (revision 2)
2023-11-08T01:03:44Z (x2)    Normal    TrafficWeightUpdated      Traffic weight updated 
2023-11-08T01:03:44Z         Normal    ExperimentCreated         Created Experiment 'alb-rollout-6d6c8558b4-2-0'
2023-11-08T01:03:46Z (x2)    Normal    PatchingALBIngress        Updating Ingress `alb-rollout-multi-ingress-1` to desiredWeight '0'
2023-11-08T01:03:46Z (x2)    Normal    PatchingALBIngress        Updating Ingress `alb-rollout-multi-ingress-2` to desiredWeight '0'
2023-11-08T01:03:46Z         Normal    TrafficWeightUpdated      Traffic weight updated additional: [{20  } {20  }]

time="2023-11-08T01:03:54Z" level=info msg="Deleting e2e-test-name=TestALBExperimentStepNoSetWeightMultiIngress"
    --- FAIL: TestAWSSuite/TestALBExperimentStepNoSetWeightMultiIngress (13.57s)

Failed
=== RUN   TestAWSSuite
--- FAIL: TestAWSSuite (118.15s)

Failed
=== RUN   TestCanarySuite/TestCanaryScaleDownDelay
time="2023-11-08T01:09:50Z" level=info msg="Deleting e2e-test-name=TestCanaryScaleDownDelay"
W1108 01:09:50.946951   21896 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1108 01:09:50.949906   21896 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2023-11-08T01:09:51Z" level=info msg="service/canary-scaledowndelay-root created\n" rollout=canary-scaledowndelay
time="2023-11-08T01:09:51Z" level=info msg="service/canary-scaledowndelay-canary created\n" rollout=canary-scaledowndelay
time="2023-11-08T01:09:51Z" level=info msg="service/canary-scaledowndelay-stable created\n" rollout=canary-scaledowndelay
time="2023-11-08T01:09:51Z" level=info msg="ingress.networking.k8s.io/canary-scaledowndelay-ingress created\n" rollout=canary-scaledowndelay
time="2023-11-08T01:09:52Z" level=info msg="rollout.argoproj.io/canary-scaledowndelay created\n" rollout=canary-scaledowndelay
time="2023-11-08T01:09:52Z" level=info msg="Waiting for condition: status=Healthy" rollout=canary-scaledowndelay
time="2023-11-08T01:09:54Z" level=info msg="Condition 'status=Healthy' met after 2s" rollout=canary-scaledowndelay
time="2023-11-08T01:09:54Z" level=info msg="Updated rollout spec: {\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"rev\":\"two\"}}}}}" rollout=canary-scaledowndelay
time="2023-11-08T01:09:54Z" level=info msg="Waiting for condition: status=Healthy" rollout=canary-scaledowndelay
time="2023-11-08T01:09:55Z" level=info msg="Condition 'status=Healthy' met after 1s" rollout=canary-scaledowndelay
time="2023-11-08T01:09:55Z" level=info msg="Sleeping 2s" rollout=canary-scaledowndelay
time="2023-11-08T01:09:57Z" level=info msg="Updated rollout spec: {\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"rev\":\"three\"}}}}}" rollout=canary-scaledowndelay
time="2023-11-08T01:09:57Z" level=info msg="Waiting for condition: status=Healthy" rollout=canary-scaledowndelay
time="2023-11-08T01:09:58Z" level=info msg="Condition 'status=Healthy' met after 0s" rollout=canary-scaledowndelay
time="2023-11-08T01:09:59Z" level=info msg="Updated rollout spec: {\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"rev\":\"two\"}}}}}" rollout=canary-scaledowndelay
time="2023-11-08T01:09:59Z" level=info msg="Waiting for condition: status=Healthy" rollout=canary-scaledowndelay
time="2023-11-08T01:09:59Z" level=info msg="Condition 'status=Healthy' met after 0s" rollout=canary-scaledowndelay
    canary_test.go:529: 
        	Error Trace:	/home/runner/work/argo-rollouts/argo-rollouts/test/e2e/canary_test.go:529
        	            				/home/runner/work/argo-rollouts/argo-rollouts/test/fixtures/then.go:28
        	            				/home/runner/work/argo-rollouts/argo-rollouts/test/e2e/canary_test.go:517
        	Error:      	Should NOT be empty, but was 
        	Test:       	TestCanarySuite/TestCanaryScaleDownDelay
Name:            canary-scaledowndelay
Namespace:       default
Status:          �[32m✔�[0m Healthy
Strategy:        Canary
  Step:          
  SetWeight:     100
  ActualWeight:  100
Images:          nginx:1.19-alpine (�[32mstable�[0m)
Replicas:
  Desired:       1
  Current:       2
  Updated:       1
  Ready:         2
  Available:     2

NAME                                               KIND        STATUS        AGE  INFO
⟳ canary-scaledowndelay                            Rollout     �[32m✔�[0m Healthy     9s   
├──# revision:4                                                                   
│  └──⧉ �[32mcanary-scaledowndelay-7c74c66d54�[0m           ReplicaSet  �[32m✔�[0m Healthy     7s   �[32mstable�[0m
│     └──□ canary-scaledowndelay-7c74c66d54-jnmz9  Pod         �[32m✔�[0m Running     7s   ready:1/1
├──# revision:3                                                                   
│  └──⧉ canary-scaledowndelay-9586cf7d4            ReplicaSet  �[32m✔�[0m Healthy     4s   
│     └──□ canary-scaledowndelay-9586cf7d4-5nlqs   Pod         �[32m✔�[0m Running     4s   ready:1/1
└──# revision:1                                                                   
   └──⧉ canary-scaledowndelay-66597877b7           ReplicaSet  �[0m•�[0m ScaledDown  9s   

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    rollout.argoproj.io/revision: "4"
  creationTimestamp: "2023-11-08T01:09:52Z"
  generation: 5
  labels:
    argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e
    e2e-test-name: TestCanaryScaleDownDelay
  name: canary-scaledowndelay
  namespace: default
  resourceVersion: "7207"
  uid: 64904868-3757-4e59-be6e-21869a695171
spec:
  replicas: 1
  selector:
    matchLabels:
      app: canary-scaledowndelay
  strategy:
    canary:
      canaryService: canary-scaledowndelay-canary
      scaleDownDelayRevisionLimit: 1
      stableService: canary-scaledowndelay-stable
      trafficRouting:
        alb:
          ingress: canary-scaledowndelay-ingress
          rootService: canary-scaledowndelay-root
          servicePort: 80
  template:
    metadata:
      annotations:
        rev: two
      creationTimestamp: null
      labels:
        app: canary-scaledowndelay
    spec:
      containers:
      - image: nginx:1.19-alpine
        name: canary-scaledowndelay
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 16Mi
status:
  HPAReplicas: 2
  availableReplicas: 2
  blueGreen: {}
  canary:
    weights:
      canary:
        podTemplateHash: 7c74c66d54
        serviceName: canary-scaledowndelay-canary
        weight: 0
      stable:
        podTemplateHash: 9586cf7d4
        serviceName: canary-scaledowndelay-stable
        weight: 100
  conditions:
  - lastTransitionTime: "2023-11-08T01:09:54Z"
    lastUpdateTime: "2023-11-08T01:09:54Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  - lastTransitionTime: "2023-11-08T01:09:54Z"
    lastUpdateTime: "2023-11-08T01:09:54Z"
    message: Rollout is not healthy
    reason: RolloutHealthy
    status: "False"
    type: Healthy
  - lastTransitionTime: "2023-11-08T01:09:52Z"
    lastUpdateTime: "2023-11-08T01:09:59Z"
    message: ReplicaSet "canary-scaledowndelay-7c74c66d54" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  - lastTransitionTime: "2023-11-08T01:09:59Z"
    lastUpdateTime: "2023-11-08T01:09:59Z"
    message: RolloutCompleted
    reason: RolloutCompleted
    status: "True"
    type: Completed
  currentPodHash: 7c74c66d54
  currentStepHash: 5ffbfbbd64
  observedGeneration: "5"
  phase: Healthy
  readyReplicas: 2
  replicas: 2
  selector: app=canary-scaledowndelay
  stableRS: 7c74c66d54
  updatedReplicas: 1

2023-11-08T01:09:52Z         Normal    RolloutAddedToInformer    Rollout resource added to informer: default/canary-scaledowndelay
2023-11-08T01:09:52Z         Normal    RolloutNotCompleted       Rollout not completed, started update to revision 1 (66597877b7)
2023-11-08T01:09:52Z         Normal    RolloutUpdated            Rollout updated to revision 1
2023-11-08T01:09:52Z         Normal    NewReplicaSetCreated      Created ReplicaSet canary-scaledowndelay-66597877b7 (revision 1)
2023-11-08T01:09:52Z         Normal    PatchingALBIngress        Updating Ingress `canary-scaledowndelay-ingress` to desiredWeight '0'
2023-11-08T01:09:52Z (x2)    Normal    TrafficWeightUpdated      Traffic weight updated to 0
2023-11-08T01:09:52Z         Normal    ScalingReplicaSet         Scaled up ReplicaSet canary-scaledowndelay-66597877b7 (revision 1) from 0 to 1
2023-11-08T01:09:52Z         Normal    RolloutCompleted          Rollout completed update to revision 1 (66597877b7): Initial deploy
2023-11-08T01:09:54Z         Normal    SwitchService             Switched selector for service 'canary-scaledowndelay-stable' from '' to '66597877b7'
2023-11-08T01:09:54Z         Normal    SwitchService             Switched selector for service 'canary-scaledowndelay-canary' from '' to '66597877b7'
2023-11-08T01:09:54Z         Normal    RolloutNotCompleted       Rollout not completed, started update to revision 2 (7c74c66d54)
2023-11-08T01:09:54Z         Normal    RolloutUpdated            Rollout updated to revision 2
2023-11-08T01:09:54Z         Normal    NewReplicaSetCreated      Created ReplicaSet canary-scaledowndelay-7c74c66d54 (revision 2)
2023-11-08T01:09:54Z         Normal    ScalingReplicaSet         Scaled up ReplicaSet canary-scaledowndelay-7c74c66d54 (revision 2) from 0 to 1
2023-11-08T01:09:55Z         Normal    SwitchService             Switched selector for service 'canary-scaledowndelay-canary' from '66597877b7' to '7c74c66d54'
2023-11-08T01:09:55Z         Normal    RolloutCompleted          Rollout completed update to revision 2 (7c74c66d54): Completed all 0 canary steps
2023-11-08T01:09:55Z         Normal    SwitchService             Switched selector for service 'canary-scaledowndelay-stable' from '66597877b7' to '7c74c66d54'
2023-11-08T01:09:55Z (x4)    Normal    TrafficWeightUpdated      Traffic weight updated 
2023-11-08T01:09:57Z         Normal    RolloutNotCompleted       Rollout not completed, started update to revision 3 (9586cf7d4)
2023-11-08T01:09:57Z         Normal    RolloutUpdated            Rollout updated to revision 3
2023-11-08T01:09:57Z         Normal    NewReplicaSetCreated      Created ReplicaSet canary-scaledowndelay-9586cf7d4 (revision 3)

time="2023-11-08T01:10:01Z" level=info msg="Deleting e2e-test-name=TestCanaryScaleDownDelay"
W1108 01:10:01.228423   21896 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1108 01:10:01.232411   21896 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
    --- FAIL: TestCanarySuite/TestCanaryScaleDownDelay (10.37s)

Failed
=== RUN   TestCanarySuite
time="2023-11-08T01:07:12Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n"
--- FAIL: TestCanarySuite (261.43s)