deploymentconfigs when run iteratively [Conformance] should only deploy the last deployment

[bparees]

/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:188
2017-12-01 00:48:56.786002895 +0000 UTC: Deployer pod invariant broken! More than one unterminated deployer pod exists for DC extended-test-cli-deployment-jmxgc-zbvps/deployment-simple!
		List of unterminated pods: ([]*v1.Pod)[<*>(0xc420ce0a80)&Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:deployment-simple-3-deploy,GenerateName:,Namespace:extended-test-cli-deployment-jmxgc-zbvps,SelfLink:/api/v1/namespaces/extended-test-cli-deployment-jmxgc-zbvps/pods/deployment-simple-3-deploy,UID:5eaa1077-d631-11e7-bfc1-0edcbcf5e722,ResourceVersion:19952,Generation:0,CreationTimestamp:2017-12-01 00:48:38 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{openshift.io/deployer-pod-for.name: deployment-simple-3,},Annotations:map[string]string{openshift.io/deployment-config.name: deployment-simple,openshift.io/deployment.name: deployment-simple-3,openshift.io/scc: restricted,},OwnerReferences:[{v1 ReplicationController deployment-simple-3 5ea84f10-d631-11e7-bfc1-0edcbcf5e722 <nil> <nil>}],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[{deployer-token-z25j7 {nil nil nil nil nil SecretVolumeSource{SecretName:deployer-token-z25j7,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{deployment openshift/origin-deployer:bc3cd34 [] []  [] [] [{KUBERNETES_MASTER https://ip-172-18-14-174.ec2.internal:8443 nil} {OPENSHIFT_MASTER https://ip-172-18-14-174.ec2.internal:8443 nil} {BEARER_TOKEN_FILE /var/run/secrets/kubernetes.io/serviceaccount/token nil} {OPENSHIFT_CA_DATA -----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE1MTIwODY4ODUwHhcNMTcxMjAxMDAwODA0WhcNMjIxMTMw
MDAwODA1WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTIwODY4ODUw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCqKIu61x0kcVgY1vtdx0bn
DuE9wBZlGn7hNIh6ZlEHVtX0/O2ZbrK/Dl8d1vxPVkwvI59EHay+rp8BgeeQw2Ax
Yl3ofouK3bs02/nIwPM55pf1Bn3gfddVXFvOoazTcIB0a2CQro9A++i115GpZTfn
JvcqlWEWPesAj+B2sXT9gNMu7+w8hhRoHDG00tJQGydVI8rcpYl+xxzNCcLkh7lU
NXodkzOecnTTvhxwSpt7weXnhRVQTFbeyvOrYQQfaN4oT3+zDt4QgZ+sW2xT4QcG
j4u2wVgh8iBmvnhucVyn8RfSclHLfdrGuJxs/4leQJsn9AQNz7PYbJt963agHZi/
AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQB1FHEiXTIMssKdqKrbk6RQoMCtu2CTbqUbgmPd0efgRG9F
oS5DVsJBNr0okzoKZm62RyzDn4/Lv80MxTt3vBYu1uOGlhFwjvS0JMkYal/NnQaK
7Jm8k9csXF+QQ3omXMd9gpiJrYUAd0a1d99f85pDnVOkiPa9xKMtWIkmFOQobVyI
1j+ruIVvvGj6865loLuumkHagK8hh4trn0PC6Oxd7pL20WPeEk4cxwdCAx0Y3Tmp
5pDZuOsgzFO/dM4LqRgAwPInkjAjmQLuA6hmKs2IJkmXI/geVSQOpaq9hBWSd2Na
YFmTEQ7jYAlL6ThAcugTC6u2FCRAITGI0jSTq1Ce
-----END CERTIFICATE-----
 nil} {OPENSHIFT_DEPLOYMENT_NAME deployment-simple-3 nil} {OPENSHIFT_DEPLOYMENT_NAMESPACE extended-test-cli-deployment-jmxgc-zbvps nil}] {map[] map[]} [{deployer-token-z25j7 true /var/run/secrets/kubernetes.io/serviceaccount  <nil>}] nil nil nil /dev/termination-log File IfNotPresent SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[KILL MKNOD SETGID SETUID],},Privileged:*false,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c38,c32,},RunAsUser:*1001470000,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,} false false false}],RestartPolicy:Never,TerminationGracePeriodSeconds:*10,ActiveDeadlineSeconds:*21600,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{region: infra,},ServiceAccountName:deployer,DeprecatedServiceAccount:deployer,NodeName:ip-172-18-14-174.ec2.internal,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c38,c32,},RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:*1001470000,},ImagePullSecrets:[{deployer-dockercfg-dc9x7}],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,},Status:PodStatus{Phase:Pending,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-12-01 00:48:38 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2017-12-01 00:48:38 +0000 UTC ContainersNotReady containers with unready status: [deployment]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-12-01 00:48:38 +0000 UTC  }],Message:,Reason:,HostIP:172.18.14.174,PodIP:,StartTime:2017-12-01 00:48:38 +0000 UTC,ContainerStatuses:[{deployment {ContainerStateWaiting{Reason:ContainerCreating,Message:,} nil nil} {nil nil nil} false 0 openshift/origin-deployer:bc3cd34  }],QOSClass:BestEffort,InitContainerStatuses:[],},} <*>(0xc420ece700)&Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:deployment-simple-1-deploy,GenerateName:,Namespace:extended-test-cli-deployment-jmxgc-zbvps,SelfLink:/api/v1/namespaces/extended-test-cli-deployment-jmxgc-zbvps/pods/deployment-simple-1-deploy,UID:6961995b-d631-11e7-bfc1-0edcbcf5e722,ResourceVersion:20135,Generation:0,CreationTimestamp:2017-12-01 00:48:56 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{openshift.io/deployer-pod-for.name: deployment-simple-1,},Annotations:map[string]string{openshift.io/deployment-config.name: deployment-simple,openshift.io/deployment.name: deployment-simple-1,openshift.io/scc: restricted,},OwnerReferences:[{v1 ReplicationController deployment-simple-1 52e68614-d631-11e7-bfc1-0edcbcf5e722 <nil> <nil>}],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[{deployer-token-z25j7 {nil nil nil nil nil SecretVolumeSource{SecretName:deployer-token-z25j7,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{deployment openshift/origin-deployer:bc3cd34 [] []  [] [] [{KUBERNETES_MASTER https://ip-172-18-14-174.ec2.internal:8443 nil} {OPENSHIFT_MASTER https://ip-172-18-14-174.ec2.internal:8443 nil} {BEARER_TOKEN_FILE /var/run/secrets/kubernetes.io/serviceaccount/token nil} {OPENSHIFT_CA_DATA -----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE1MTIwODY4ODUwHhcNMTcxMjAxMDAwODA0WhcNMjIxMTMw
MDAwODA1WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTIwODY4ODUw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCqKIu61x0kcVgY1vtdx0bn
DuE9wBZlGn7hNIh6ZlEHVtX0/O2ZbrK/Dl8d1vxPVkwvI59EHay+rp8BgeeQw2Ax
Yl3ofouK3bs02/nIwPM55pf1Bn3gfddVXFvOoazTcIB0a2CQro9A++i115GpZTfn
JvcqlWEWPesAj+B2sXT9gNMu7+w8hhRoHDG00tJQGydVI8rcpYl+xxzNCcLkh7lU
NXodkzOecnTTvhxwSpt7weXnhRVQTFbeyvOrYQQfaN4oT3+zDt4QgZ+sW2xT4QcG
j4u2wVgh8iBmvnhucVyn8RfSclHLfdrGuJxs/4leQJsn9AQNz7PYbJt963agHZi/
AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQB1FHEiXTIMssKdqKrbk6RQoMCtu2CTbqUbgmPd0efgRG9F
oS5DVsJBNr0okzoKZm62RyzDn4/Lv80MxTt3vBYu1uOGlhFwjvS0JMkYal/NnQaK
7Jm8k9csXF+QQ3omXMd9gpiJrYUAd0a1d99f85pDnVOkiPa9xKMtWIkmFOQobVyI
1j+ruIVvvGj6865loLuumkHagK8hh4trn0PC6Oxd7pL20WPeEk4cxwdCAx0Y3Tmp
5pDZuOsgzFO/dM4LqRgAwPInkjAjmQLuA6hmKs2IJkmXI/geVSQOpaq9hBWSd2Na
YFmTEQ7jYAlL6ThAcugTC6u2FCRAITGI0jSTq1Ce
-----END CERTIFICATE-----
 nil} {OPENSHIFT_DEPLOYMENT_NAME deployment-simple-1 nil} {OPENSHIFT_DEPLOYMENT_NAMESPACE extended-test-cli-deployment-jmxgc-zbvps nil}] {map[] map[]} [{deployer-token-z25j7 true /var/run/secrets/kubernetes.io/serviceaccount  <nil>}] nil nil nil /dev/termination-log File IfNotPresent SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[KILL MKNOD SETGID SETUID],},Privileged:*false,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c38,c32,},RunAsUser:*1001470000,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,} false false false}],RestartPolicy:Never,TerminationGracePeriodSeconds:*10,ActiveDeadlineSeconds:*21600,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{region: infra,},ServiceAccountName:deployer,DeprecatedServiceAccount:deployer,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c38,c32,},RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:*1001470000,},ImagePullSecrets:[{deployer-dockercfg-dc9x7}],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,},Status:PodStatus{Phase:Pending,Conditions:[],Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[],QOSClass:BestEffort,InitContainerStatuses:[],},}]
	
Expected
    <int>: 2
to be <=
    <int>: 1

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17429/test_pull_request_origin_extended_conformance_crio/185/

[mfojtik]

/assign tnozicka

[tnozicka]

this looks suspicious

Status:PodStatus{Phase:Pending,Conditions:[],Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[],QOSClass:BestEffort,InitContainerStatuses:[],},}]

and likely kubelet

@bparees could you rebase your PR to pick up #17514
Judging by age of your commits you are probably missing it

[bparees]

PRs are always merged on top of master before being tested. Rebasing my PR wouldn't change anything in terms of testing. Ben Parees | OpenShift

…

On Dec 5, 2017 03:30, "Tomáš Nožička" ***@***.***> wrote: this looks suspicious Status:PodStatus{Phase:Pending,Conditions:[],Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[],QOSClass:BestEffort,InitContainerStatuses:[],},}] and likely kubelet @bparees <https://github.com/bparees> could you rebase your PR to pick up #17514 <#17514> Judging by age of your commits you are probably missing it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17581 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEvl3nmoln4VPr-1FXz27w3lZeKfvQ74ks5s9P86gaJpZM4Q1LAc> .

[tnozicka]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18074/test_pull_request_origin_extended_conformance_install/5715/#featuredeploymentconfig-deploymentconfigs-with-revision-history-limits-conformance-should-never-persist-more-old-deployments-than-acceptable-after-being-observed-by-the-controller-suiteopenshiftconformanceparallel

[tnozicka]

this is bad

I0117 16:42:26.094971   20209 deployer_controller.go:173] Created deployer pod "history-limit-1-deploy" for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-1"
I0117 16:42:31.770294   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-1" from Pending to Running (scale: 842520861548)
I0117 16:42:43.280856   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-1" from Running to Complete (scale: 842473021468)
 I0117 16:42:43.910661   20209 deployer_controller.go:173] Created deployer pod "history-limit-2-deploy" for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-2"
I0117 16:42:43.923330   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-2" from New to Pending (scale: 842473028612)
I0117 16:42:50.117947   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-
deployment-5wb7s-rw7nn/history-limit-2" from Pending to Running (scale: 842489733420)
I0117 16:43:00.131808   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-2" from Running to Complete (scale: 842398424476)
I0117 16:43:00.684431   20209 deployer_controller.go:173] Created deployer pod "history-limit-3-deploy" for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-3"
I0117 16:43:00.712534   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-3" from New to Pending (scale: 842408057172)
I0117 16:43:05.563722   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-3" from Pending to Running (scale: 842411595212)
I0117 16:43:18.999531   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-3" from Running to Complete (scale: 842499659580)
I0117 16:43:19.737828   20209 deployer_controller.go:173] Created deployer pod "history-limit-4-deploy" for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-4"
I0117 16:43:19.801233   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-4" from New to Pending (scale: 842514932788)
I0117 16:43:27.239808   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-4" from Pending to Running (scale: 842523808764)
I0117 16:43:37.633164   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-4" from Running to Complete (scale: 842504264492)
I0117 16:43:38.157818   20209 deployer_controller.go:173] Created deployer pod "history-limit-5-deploy" for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-5"
I0117 16:43:38.183773   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-5" from New to Pending (scale: 842440587268)
I0117 16:43:44.825733   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-5" from Pending to Running (scale: 842461530060)
I0117 16:44:05.841419   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-5" from Running to Complete (scale: 842405207612)
I0117 16:44:06.507722   20209 deployer_controller.go:173] Created deployer pod "history-limit-6-deploy" for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-6"
I0117 16:44:06.525363   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-6" from New to Pending (scale: 842441583876)
I0117 16:44:12.100420   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-6" from Pending to Running (scale: 842514671004)
I0117 16:44:20.597455   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-6" from Running to Complete (scale: 842515421980)
I0117 16:44:21.168556   20209 deployer_controller.go:173] Created deployer pod "history-limit-4-deploy" for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-4"
E0117 16:44:21.177749   20209 deployer_controller.go:222] Failing rollout for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-6" because its deployer pod "history-limit-6-deploy" disappeared
I0117 16:44:21.222782   20209 deployer_controller.go:173] Created deployer pod "history-limit-5-deploy" for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-5"
I0117 16:44:21.452513   20209 deployer_controller.go:173] Created deployer pod "history-limit-7-deploy" for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-7"
I0117 16:44:21.501961   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-7" from New to Pending (scale: 842514106596)
I0117 16:44:27.724227   20209 deployer_controller.go:289] Updated rollout status for "extended-test-cli-deployment-5wb7s-rw7nn/history-limit-7" from Pending to Running (scale: 842506053708)

I'll continue searching for the cause tomorrow but the only place it creates a deployer is when the current status is deployapi.DeploymentStatusNew which after it's done is Complete. I am not sure it's because the informer is not synced - it looks like it has to see other states to transition to Complete in the first place. Maybe something is doing retry update on the RC wrong and overwrites the value...