Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestTriggerDependencyAnnotation failing on Mink fairly consistently #4294

Closed
vaikas opened this issue Oct 13, 2020 · 1 comment · Fixed by #4296
Closed

TestTriggerDependencyAnnotation failing on Mink fairly consistently #4294

vaikas opened this issue Oct 13, 2020 · 1 comment · Fixed by #4296
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@vaikas
Copy link
Contributor

vaikas commented Oct 13, 2020

Describe the bug

TestTriggerDependencyAnnotation seems to be failing fairly frequently on Mink using Github Actions on kind.

2020-10-12T21:22:07.4013714Z     creation.go:683: Creating pingsource &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:test-ping-source-annotation GenerateName: Namespace:test-trigger-dependency-annotation-n2l4r SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[]} Spec:{SourceSpec:{Sink:{Ref:0xc0012fdf80 URI:} CloudEventOverrides:<nil>} Schedule:*/1 * * * * Timezone: JsonData:{"msg":"Test trigger-annotation 7f42f3dd-a1ea-4102-a72e-d6fdad730739"}} Status:{SourceStatus:{Status:{ObservedGeneration:0 Conditions:[] Annotations:map[]} SinkURI: CloudEventAttributes:[]}}}
2020-10-12T21:22:07.4030528Z     operation.go:73: Object that did not become ready {Trigger eventing.knative.dev/v1beta1}-trigger-annotation when dumping error state: &{TypeMeta:{Kind:Trigger APIVersion:eventing.knative.dev/v1beta1} ObjectMeta:{Name:trigger-annotation GenerateName: Namespace:test-trigger-dependency-annotation-n2l4r SelfLink:/apis/eventing.knative.dev/v1beta1/namespaces/test-trigger-dependency-annotation-n2l4r/triggers/trigger-annotation UID:608e8961-92ae-4f99-aa87-2398e83211a0 ResourceVersion:14474 Generation:1 CreationTimestamp:2020-10-12 21:05:58 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[eventing.knative.dev/broker:default] Annotations:map[eventing.knative.dev/creator:kubernetes-admin eventing.knative.dev/lastModifier:kubernetes-admin knative.dev/dependency:{"kind":"PingSource","name":"test-ping-source-annotation","apiVersion":"sources.knative.dev/v1beta1"}] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[{Manager:e2e.test Operation:Update APIVersion:eventing.knative.dev/v1beta1 Time:2020-10-12 21:05:58 +0000 UTC FieldsType:FieldsV1 FieldsV1:&FieldsV1{Raw:*[123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 46 34 58 123 125 44 34 102 58 107 110 97 116 105 118 101 46 100 101 118 47 100 101 112 101 110 100 101 110 99 121 34 58 123 125 125 125 44 34 102 58 115 112 101 99 34 58 123 34 46 34 58 123 125 44 34 102 58 115 117 98 115 99 114 105 98 101 114 34 58 123 34 46 34 58 123 125 44 34 102 58 114 101 102 34 58 123 34 46 34 58 123 125 44 34 102 58 97 112 105 86 101 114 115 105 111 110 34 58 123 125 44 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 125 125 125 44 34 102 58 115 116 97 116 117 115 34 58 123 125 125],}} {Manager:webhook Operation:Update APIVersion:eventing.knative.dev/v1 Time:2020-10-12 21:05:59 +0000 UTC FieldsType:FieldsV1 FieldsV1:&FieldsV1{Raw:*[123 34 102 58 115 116 97 116 117 115 34 58 123 34 102 58 99 111 110 100 105 116 105 111 110 115 34 58 123 125 44 34 102 58 111 98 115 101 114 118 101 100 71 101 110 101 114 97 116 105 111 110 34 58 123 125 44 34 102 58 115 117 98 115 99 114 105 98 101 114 85 114 105 34 58 123 125 125 125],}}]} Status:{ObservedGeneration:1 Conditions:[{Type:BrokerReady Status:True Severity: LastTransitionTime:{Inner:2020-10-12 21:05:58 +0000 UTC} Reason: Message:} {Type:DependencyReady Status:False Severity: LastTransitionTime:{Inner:2020-10-12 21:05:59 +0000 UTC} Reason:MinimumReplicasUnavailable Message:Deployment does not have minimum availability.} {Type:Ready Status:False Severity: LastTransitionTime:{Inner:2020-10-12 21:05:59 +0000 UTC} Reason:MinimumReplicasUnavailable Message:Deployment does not have minimum availability.} {Type:SubscriberResolved Status:True Severity: LastTransitionTime:{Inner:2020-10-12 21:05:58 +0000 UTC} Reason: Message:} {Type:SubscriptionReady Status:True Severity: LastTransitionTime:{Inner:2020-10-12 21:05:59 +0000 UTC} Reason: Message:}] Annotations:map[]}}
2020-10-12T21:22:07.4041039Z     operation.go:75: Failed to get {Trigger eventing.knative.dev/v1beta1}-trigger-annotation ready: <nil>

However, the pingsource is ready as seen by the fact that when it goes to get cleaned up:

2020-10-12T21:22:07.4185987Z     tracker.go:129: Cleaning resource: "test-ping-source-annotation"
2020-10-12T21:22:07.4186567Z         {
2020-10-12T21:22:07.4187015Z           "apiVersion": "sources.knative.dev/v1beta1",
2020-10-12T21:22:07.4187518Z           "kind": "PingSource",
2020-10-12T21:22:07.4187856Z           "metadata": {
2020-10-12T21:22:07.4188173Z             "annotations": {
2020-10-12T21:22:07.4188899Z               "sources.knative.dev/creator": "kubernetes-admin",
2020-10-12T21:22:07.4189873Z               "sources.knative.dev/lastModifier": "kubernetes-admin"
2020-10-12T21:22:07.4190409Z             },
2020-10-12T21:22:07.4190961Z             "creationTimestamp": "2020-10-12T21:05:58Z",
2020-10-12T21:22:07.4191388Z             "finalizers": [
2020-10-12T21:22:07.4191978Z               "pingsources.sources.knative.dev"
2020-10-12T21:22:07.4192509Z             ],
2020-10-12T21:22:07.4192797Z             "generation": 1,
2020-10-12T21:22:07.4193274Z             "managedFields": [
2020-10-12T21:22:07.4193590Z               {
2020-10-12T21:22:07.4194036Z                 "apiVersion": "sources.knative.dev/v1beta1",
2020-10-12T21:22:07.4194572Z                 "fieldsType": "FieldsV1",
2020-10-12T21:22:07.4194937Z                 "fieldsV1": {
2020-10-12T21:22:07.4195240Z                   "f:spec": {
2020-10-12T21:22:07.4195513Z                     ".": {},
2020-10-12T21:22:07.4195814Z                     "f:jsonData": {},
2020-10-12T21:22:07.4196144Z                     "f:schedule": {},
2020-10-12T21:22:07.4196457Z                     "f:sink": {
2020-10-12T21:22:07.4196720Z                       ".": {},
2020-10-12T21:22:07.4196994Z                       "f:ref": {
2020-10-12T21:22:07.4197267Z                         ".": {},
2020-10-12T21:22:07.4197584Z                         "f:apiVersion": {},
2020-10-12T21:22:07.4197918Z                         "f:kind": {},
2020-10-12T21:22:07.4198214Z                         "f:name": {},
2020-10-12T21:22:07.4198540Z                         "f:namespace": {}
2020-10-12T21:22:07.4198843Z                       }
2020-10-12T21:22:07.4199097Z                     }
2020-10-12T21:22:07.4199348Z                   }
2020-10-12T21:22:07.4199594Z                 },
2020-10-12T21:22:07.4199899Z                 "manager": "e2e.test",
2020-10-12T21:22:07.4200264Z                 "operation": "Update",
2020-10-12T21:22:07.4200788Z                 "time": "2020-10-12T21:05:58Z"
2020-10-12T21:22:07.4201063Z               },
2020-10-12T21:22:07.4201386Z               {
2020-10-12T21:22:07.4201834Z                 "apiVersion": "sources.knative.dev/v1beta1",
2020-10-12T21:22:07.4202371Z                 "fieldsType": "FieldsV1",
2020-10-12T21:22:07.4202739Z                 "fieldsV1": {
2020-10-12T21:22:07.4203048Z                   "f:metadata": {
2020-10-12T21:22:07.4203390Z                     "f:finalizers": {
2020-10-12T21:22:07.4203700Z                       ".": {},
2020-10-12T21:22:07.4204261Z                       "v:\"pingsources.sources.knative.dev\"": {}
2020-10-12T21:22:07.4204817Z                     }
2020-10-12T21:22:07.4205067Z                   }
2020-10-12T21:22:07.4205315Z                 },
2020-10-12T21:22:07.4205612Z                 "manager": "mtping",
2020-10-12T21:22:07.4205978Z                 "operation": "Update",
2020-10-12T21:22:07.4206497Z                 "time": "2020-10-12T21:06:03Z"
2020-10-12T21:22:07.4206772Z               },
2020-10-12T21:22:07.4207015Z               {
2020-10-12T21:22:07.4207462Z                 "apiVersion": "sources.knative.dev/v1beta1",
2020-10-12T21:22:07.4207999Z                 "fieldsType": "FieldsV1",
2020-10-12T21:22:07.4208367Z                 "fieldsV1": {
2020-10-12T21:22:07.4208681Z                   "f:status": {
2020-10-12T21:22:07.4208962Z                     ".": {},
2020-10-12T21:22:07.4209290Z                     "f:ceAttributes": {},
2020-10-12T21:22:07.4209660Z                     "f:conditions": {},
2020-10-12T21:22:07.4210079Z                     "f:observedGeneration": {},
2020-10-12T21:22:07.4210501Z                     "f:sinkUri": {}
2020-10-12T21:22:07.4210789Z                   }
2020-10-12T21:22:07.4211036Z                 },
2020-10-12T21:22:07.4211341Z                 "manager": "webhook",
2020-10-12T21:22:07.4211709Z                 "operation": "Update",
2020-10-12T21:22:07.4212218Z                 "time": "2020-10-12T21:06:04Z"
2020-10-12T21:22:07.4212490Z               }
2020-10-12T21:22:07.4212730Z             ],
2020-10-12T21:22:07.4213356Z             "name": "test-ping-source-annotation",
2020-10-12T21:22:07.4214502Z             "namespace": "test-trigger-dependency-annotation-n2l4r",
2020-10-12T21:22:07.4215335Z             "resourceVersion": "14688",
2020-10-12T21:22:07.4217027Z             "selfLink": "/apis/sources.knative.dev/v1beta1/namespaces/test-trigger-dependency-annotation-n2l4r/pingsources/test-ping-source-annotation",
2020-10-12T21:22:07.4218658Z             "uid": "ac4fdc9c-288e-4558-82b9-26a3ac767de0"
2020-10-12T21:22:07.4219204Z           },
2020-10-12T21:22:07.4219463Z           "spec": {
2020-10-12T21:22:07.4220377Z             "jsonData": "{\"msg\":\"Test trigger-annotation 7f42f3dd-a1ea-4102-a72e-d6fdad730739\"}",
2020-10-12T21:22:07.4221064Z             "schedule": "*/1 * * * *",
2020-10-12T21:22:07.4221358Z             "sink": {
2020-10-12T21:22:07.4221626Z               "ref": {
2020-10-12T21:22:07.4222080Z                 "apiVersion": "eventing.knative.dev/v1",
2020-10-12T21:22:07.4222541Z                 "kind": "Broker",
2020-10-12T21:22:07.4222871Z                 "name": "default",
2020-10-12T21:22:07.4223839Z                 "namespace": "test-trigger-dependency-annotation-n2l4r"
2020-10-12T21:22:07.4224577Z               }
2020-10-12T21:22:07.4224819Z             }
2020-10-12T21:22:07.4225055Z           },
2020-10-12T21:22:07.4225313Z           "status": {
2020-10-12T21:22:07.4225640Z             "ceAttributes": [
2020-10-12T21:22:07.4225948Z               {
2020-10-12T21:22:07.4227249Z                 "source": "/apis/v1/namespaces/test-trigger-dependency-annotation-n2l4r/pingsources/test-ping-source-annotation",
2020-10-12T21:22:07.4228517Z                 "type": "dev.knative.sources.ping"
2020-10-12T21:22:07.4228958Z               }
2020-10-12T21:22:07.4229200Z             ],
2020-10-12T21:22:07.4229489Z             "conditions": [
2020-10-12T21:22:07.4229779Z               {
2020-10-12T21:22:07.4230346Z                 "lastTransitionTime": "2020-10-12T21:06:04Z",
2020-10-12T21:22:07.4230776Z                 "status": "True",
2020-10-12T21:22:07.4231191Z                 "type": "Deployed"
2020-10-12T21:22:07.4231486Z               },
2020-10-12T21:22:07.4231712Z               {
2020-10-12T21:22:07.4232288Z                 "lastTransitionTime": "2020-10-12T21:06:04Z",
2020-10-12T21:22:07.4233071Z                 "status": "True",
2020-10-12T21:22:07.4233389Z                 "type": "Ready"
2020-10-12T21:22:07.4233666Z               },
2020-10-12T21:22:07.4233908Z               {
2020-10-12T21:22:07.4234485Z                 "lastTransitionTime": "2020-10-12T21:05:58Z",
2020-10-12T21:22:07.4234919Z                 "status": "True",
2020-10-12T21:22:07.4235284Z                 "type": "SinkProvided"
2020-10-12T21:22:07.4235605Z               }
2020-10-12T21:22:07.4235830Z             ],
2020-10-12T21:22:07.4236195Z             "observedGeneration": 1,
2020-10-12T21:22:07.4238080Z             "sinkUri": "http://broker-ingress.mink-system.svc.c302888377.local/test-trigger-dependency-annotation-n2l4r/default"
2020-10-12T21:22:07.4239595Z           }
2020-10-12T21:22:07.4239837Z         }

Yet the Trigger is still looking like it has not observed the PingSource becoming ready:

2020-10-12T21:22:07.4241567Z     tracker.go:129: Cleaning resource: "trigger-annotation"
2020-10-12T21:22:07.4242036Z         {
2020-10-12T21:22:07.4242488Z           "apiVersion": "eventing.knative.dev/v1beta1",
2020-10-12T21:22:07.4242978Z           "kind": "Trigger",
2020-10-12T21:22:07.4243294Z           "metadata": {
2020-10-12T21:22:07.4243630Z             "annotations": {
2020-10-12T21:22:07.4244371Z               "eventing.knative.dev/creator": "kubernetes-admin",
2020-10-12T21:22:07.4245363Z               "eventing.knative.dev/lastModifier": "kubernetes-admin",
2020-10-12T21:22:07.4246787Z               "knative.dev/dependency": "{\"kind\":\"PingSource\",\"name\":\"test-ping-source-annotation\",\"apiVersion\":\"sources.knative.dev/v1beta1\"}"
2020-10-12T21:22:07.4248540Z             },
2020-10-12T21:22:07.4249169Z             "creationTimestamp": "2020-10-12T21:05:58Z",
2020-10-12T21:22:07.4249601Z             "generation": 1,
2020-10-12T21:22:07.4249913Z             "labels": {
2020-10-12T21:22:07.4250376Z               "eventing.knative.dev/broker": "default"
2020-10-12T21:22:07.4250807Z             },
2020-10-12T21:22:07.4251121Z             "managedFields": [
2020-10-12T21:22:07.4251425Z               {
2020-10-12T21:22:07.4251884Z                 "apiVersion": "eventing.knative.dev/v1beta1",
2020-10-12T21:22:07.4252429Z                 "fieldsType": "FieldsV1",
2020-10-12T21:22:07.4252917Z                 "fieldsV1": {
2020-10-12T21:22:07.4253241Z                   "f:metadata": {
2020-10-12T21:22:07.4253591Z                     "f:annotations": {
2020-10-12T21:22:07.4253909Z                       ".": {},
2020-10-12T21:22:07.4254278Z                       "f:knative.dev/dependency": {}
2020-10-12T21:22:07.4254635Z                     }
2020-10-12T21:22:07.4254888Z                   },
2020-10-12T21:22:07.4255157Z                   "f:spec": {
2020-10-12T21:22:07.4255435Z                     ".": {},
2020-10-12T21:22:07.4255743Z                     "f:subscriber": {
2020-10-12T21:22:07.4256141Z                       ".": {},
2020-10-12T21:22:07.4256417Z                       "f:ref": {
2020-10-12T21:22:07.4256695Z                         ".": {},
2020-10-12T21:22:07.4257013Z                         "f:apiVersion": {},
2020-10-12T21:22:07.4257347Z                         "f:kind": {},
2020-10-12T21:22:07.4257642Z                         "f:name": {}
2020-10-12T21:22:07.4257906Z                       }
2020-10-12T21:22:07.4258160Z                     }
2020-10-12T21:22:07.4258408Z                   },
2020-10-12T21:22:07.4258685Z                   "f:status": {}
2020-10-12T21:22:07.4258958Z                 },
2020-10-12T21:22:07.4259266Z                 "manager": "e2e.test",
2020-10-12T21:22:07.4259631Z                 "operation": "Update",
2020-10-12T21:22:07.4260163Z                 "time": "2020-10-12T21:05:58Z"
2020-10-12T21:22:07.4260441Z               },
2020-10-12T21:22:07.4260779Z               {
2020-10-12T21:22:07.4261218Z                 "apiVersion": "eventing.knative.dev/v1",
2020-10-12T21:22:07.4261745Z                 "fieldsType": "FieldsV1",
2020-10-12T21:22:07.4262108Z                 "fieldsV1": {
2020-10-12T21:22:07.4262419Z                   "f:status": {
2020-10-12T21:22:07.4262753Z                     "f:conditions": {},
2020-10-12T21:22:07.4263187Z                     "f:observedGeneration": {},
2020-10-12T21:22:07.4263651Z                     "f:subscriberUri": {}
2020-10-12T21:22:07.4263975Z                   }
2020-10-12T21:22:07.4264223Z                 },
2020-10-12T21:22:07.4264514Z                 "manager": "webhook",
2020-10-12T21:22:07.4264881Z                 "operation": "Update",
2020-10-12T21:22:07.4265399Z                 "time": "2020-10-12T21:05:59Z"
2020-10-12T21:22:07.4265674Z               }
2020-10-12T21:22:07.4265913Z             ],
2020-10-12T21:22:07.4266440Z             "name": "trigger-annotation",
2020-10-12T21:22:07.4267621Z             "namespace": "test-trigger-dependency-annotation-n2l4r",
2020-10-12T21:22:07.4268457Z             "resourceVersion": "14474",
2020-10-12T21:22:07.4269945Z             "selfLink": "/apis/eventing.knative.dev/v1beta1/namespaces/test-trigger-dependency-annotation-n2l4r/triggers/trigger-annotation",
2020-10-12T21:22:07.4271377Z             "uid": "608e8961-92ae-4f99-aa87-2398e83211a0"
2020-10-12T21:22:07.4271760Z           },
2020-10-12T21:22:07.4272014Z           "spec": {
2020-10-12T21:22:07.4272325Z             "broker": "default",
2020-10-12T21:22:07.4272639Z             "filter": {},
2020-10-12T21:22:07.4272952Z             "subscriber": {
2020-10-12T21:22:07.4273251Z               "ref": {
2020-10-12T21:22:07.4273561Z                 "apiVersion": "v1",
2020-10-12T21:22:07.4273904Z                 "kind": "Service",
2020-10-12T21:22:07.4274518Z                 "name": "subscriber-annotation",
2020-10-12T21:22:07.4275601Z                 "namespace": "test-trigger-dependency-annotation-n2l4r"
2020-10-12T21:22:07.4276344Z               }
2020-10-12T21:22:07.4276570Z             }
2020-10-12T21:22:07.4276805Z           },
2020-10-12T21:22:07.4277064Z           "status": {
2020-10-12T21:22:07.4277375Z             "conditions": [
2020-10-12T21:22:07.4277662Z               {
2020-10-12T21:22:07.4278229Z                 "lastTransitionTime": "2020-10-12T21:05:58Z",
2020-10-12T21:22:07.4278659Z                 "status": "True",
2020-10-12T21:22:07.4279015Z                 "type": "BrokerReady"
2020-10-12T21:22:07.4279434Z               },
2020-10-12T21:22:07.4279677Z               {
2020-10-12T21:22:07.4280258Z                 "lastTransitionTime": "2020-10-12T21:05:59Z",
2020-10-12T21:22:07.4280858Z                 "message": "Deployment does not have minimum availability.",
2020-10-12T21:22:07.4281575Z                 "reason": "MinimumReplicasUnavailable",
2020-10-12T21:22:07.4282114Z                 "status": "False",
2020-10-12T21:22:07.4282511Z                 "type": "DependencyReady"
2020-10-12T21:22:07.4282864Z               },
2020-10-12T21:22:07.4283107Z               {
2020-10-12T21:22:07.4283672Z                 "lastTransitionTime": "2020-10-12T21:05:59Z",
2020-10-12T21:22:07.4284258Z                 "message": "Deployment does not have minimum availability.",
2020-10-12T21:22:07.4284971Z                 "reason": "MinimumReplicasUnavailable",
2020-10-12T21:22:07.4285509Z                 "status": "False",
2020-10-12T21:22:07.4285830Z                 "type": "Ready"
2020-10-12T21:22:07.4286111Z               },
2020-10-12T21:22:07.4286352Z               {
2020-10-12T21:22:07.4286913Z                 "lastTransitionTime": "2020-10-12T21:05:58Z",
2020-10-12T21:22:07.4287342Z                 "status": "True",
2020-10-12T21:22:07.4287766Z                 "type": "SubscriberResolved"
2020-10-12T21:22:07.4288145Z               },
2020-10-12T21:22:07.4288387Z               {
2020-10-12T21:22:07.4288946Z                 "lastTransitionTime": "2020-10-12T21:05:59Z",
2020-10-12T21:22:07.4289450Z                 "status": "True",
2020-10-12T21:22:07.4289863Z                 "type": "SubscriptionReady"
2020-10-12T21:22:07.4290231Z               }
2020-10-12T21:22:07.4290472Z             ],
2020-10-12T21:22:07.4290834Z             "observedGeneration": 1,
2020-10-12T21:22:07.4293588Z             "subscriberUri": "http://subscriber-annotation.test-trigger-dependency-annotation-n2l4r.svc.c302888377.local/"
2020-10-12T21:22:07.4296102Z           }
2020-10-12T21:22:07.4296348Z         }

Expected behavior
Test to pass consistently.

To Reproduce
You can observe this in the mink e2e eventing tests. Some examples here:
https://github.com/mattmoor/mink/runs/1245661099?check_suite_focus=true

https://github.com/mattmoor/mink/actions?query=workflow%3A%22MinKinD+Upstream+E2Es%22

Knative release version

Additional context
Possibly related to: #3578
We don't see these failing on prow, or at least very very rarely as pointed out in #3578 but seems to fail consistenly on kind.
@vaikas vaikas added the kind/bug Categorizes issue or PR as related to a bug. label Oct 13, 2020
@vaikas
Copy link
Contributor Author

vaikas commented Oct 13, 2020

Oh, due to the way the tests are ran, the timestamps are meaningless since all the output from the test gets buffered until the test fails.

@vaikas vaikas mentioned this issue Oct 13, 2020
Merged
This was referenced Oct 13, 2020
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

use Source informer instead of conditions vaikas/eventing
1 participant
@vaikas