Skip to content
This repository has been archived by the owner on Oct 21, 2020. It is now read-only.

Digital Ocean external volume test failing #529

Closed
tianhuil opened this issue Dec 26, 2017 · 28 comments
Closed

Digital Ocean external volume test failing #529

tianhuil opened this issue Dec 26, 2017 · 28 comments
Labels
area/digitalocean

Comments

@tianhuil
Copy link
Contributor

tianhuil commented Dec 26, 2017 •
edited
Loading

I built a PV and connected to Digital Ocean as per the README.md. Aside from a small DNS fix, this has worked. However, I am unable to successfully complete the last two steps in the README.md

$ kubectl create -f examples/pvc.yaml
$ kubectl create -f examples/pod-application.yaml

The PVC appears to successfully bind but the pod does not start up:

$ kubectl get pvc
NAME      STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pv1       Pending                                       default        17s

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM         STORAGECLASS   REASON    AGE
pvc-40a76408-ea6e-11e7-901c-a2fa2817e02b   1Gi        RWO            Delete           Bound       default/pv1   default                  1s

$ kubectl get pods
NAME        READY     STATUS              RESTARTS   AGE
busy-pod    0/1       ContainerCreating   0          8m
busy-pod2   1/1       Running             0          13s

and it appears to never want to startup. busy-pod2 is the same pod but without the external mounting

apiVersion: v1
kind: Pod
metadata:
  name: busy-pod2
spec:
  containers:
  - command:
    - /bin/sleep
    - 7d
    image: busybox
    name: busy-container1

FYI: using commit ce571982aeedba29185048b8398fab90116f2d29. Let me know if you have any ideas @klausenbusk.
@tianhuil
Copy link
Contributor Author

tianhuil commented Dec 26, 2017 •
edited
Loading

And here is the message from the pod, PVC, and PVC:

$ kubectl describe pod busy-pod
Name:         busy-pod
Namespace:    default
Node:         node2/192.241.159.35
Start Time:   Tue, 26 Dec 2017 14:05:23 -0500
Labels:       <none>
Annotations:  <none>
Status:       Pending
IP:           
Containers:
  busy-container1:
    Container ID:  
    Image:         busybox
    Image ID:      
    Port:          <none>
    Command:
      /bin/sleep
      7d
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/pv-1 from vol1 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ndh9d (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  vol1:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pv1
    ReadOnly:   false
  default-token-ndh9d:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-ndh9d
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                 From               Message
  ----     ------                 ----                ----               -------
  Normal   Scheduled              41m                 default-scheduler  Successfully assigned busy-pod to node2
  Normal   SuccessfulMountVolume  41m                 kubelet, node2     MountVolume.SetUp succeeded for volume "default-token-ndh9d"
  Warning  FailedMount            33s (x18 over 39m)  kubelet, node2     Unable to mount volumes for pod "busy-pod_default(b9841c6e-ea6f-11e7-901c-a2fa2817e02b)": timeout expired waiting for volumes to attach/mount for pod "default"/"busy-pod". list of unattached/unmounted volumes=[vol1]

$ kubectl describe pvc
Name:          pv1
Namespace:     default
StorageClass:  default
Status:        Bound
Volume:        pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b
Labels:        <none>
Annotations:   control-plane.alpha.kubernetes.io/leader={"holderIdentity":"9e90238f-ea6d-11e7-ba2f-0a580af40121","leaseDurationSeconds":15,"acquireTime":"2017-12-26T19:05:23Z","renewTime":"2017-12-26T19:05:25Z","lea...
               pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
               volume.beta.kubernetes.io/storage-provisioner=external/digitalocean
Capacity:      1Gi
Access Modes:  RWO
Events:
  Type    Reason                 Age                From                                                                                                  Message
  ----    ------                 ----               ----                                                                                                  -------
  Normal  ExternalProvisioning   57m (x3 over 57m)  persistentvolume-controller                                                                           waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
  Normal  Provisioning           57m                external/digitalocean digitalocean-provisioner-6c7bbf4ccc-trlz2 9e90238f-ea6d-11e7-ba2f-0a580af40121  External provisioner is provisioning volume for claim "default/pv1"
  Normal  ProvisioningSucceeded  57m                external/digitalocean digitalocean-provisioner-6c7bbf4ccc-trlz2 9e90238f-ea6d-11e7-ba2f-0a580af40121  Successfully provisioned volume pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b

$ kubectl describe pv pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b
Name:            pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b
Labels:          failure-domain.beta.kubernetes.io/zone=nyc1
Annotations:     Provisioner_Id=
                 digitalocean.external-storage.incubator.kubernetes.io/VolumeID=b9bc7ffb-ea6f-11e7-bef0-0242ac116407
                 kubernetes.io/createdby=digitalocean-provisioner
                 pv.kubernetes.io/provisioned-by=external/digitalocean
StorageClass:    default
Status:          Bound
Claim:           default/pv1
Reclaim Policy:  Delete
Access Modes:    RWO
Capacity:        1Gi
Message:
Source:
    Type:    FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
    Driver:      Options:  %v

    FSType:                                           external-storage/digitalocean
    SecretRef:
    ReadOnly:                                         <nil>
%!(EXTRA bool=false, map[string]string=map[])Events:  <none>

And both the zone for the sc and the region of node are nyc1.

@tianhuil
Copy link
Contributor Author

And the logs on the node are telling me this:

Dec 26 20:19:52 node2 kubelet[5443]: E1226 20:19:52.831789    5443 kubelet.go:1625] Unable to mount volumes for pod "busy-pod_default(b9841c6e-ea6f-11e7-901c-a2fa2817e02b)": timeout expired waiting for volumes to attach/mount for pod "default"/"busy-pod". list of unattached/unmounted volumes=[vol1]; skipping pod
Dec 26 20:19:52 node2 kubelet[5443]: E1226 20:19:52.831910    5443 pod_workers.go:186] Error syncing pod b9841c6e-ea6f-11e7-901c-a2fa2817e02b ("busy-pod_default(b9841c6e-ea6f-11e7-901c-a2fa2817e02b)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"busy-pod". list of unattached/unmounted volumes=[vol1]
Dec 26 20:20:04 node2 kubelet[5443]: W1226 20:20:04.904166    5443 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b
Dec 26 20:20:46 node2 kubelet[5443]: I1226 20:20:46.645430    5443 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b") pod "busy-pod" (UID: "b9841c6e-ea6f-11e7-901c-a2fa2817e02b")
Dec 26 20:20:46 node2 kubelet[5443]: E1226 20:20:46.652259    5443 nestedpendingoperations.go:263] Operation for "\"flexvolume-external-storage/digitalocean/pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b\"" failed. No retries permitted until 2017-12-26 20:22:48.652107441 +0000 UTC m=+757282.446167976 (durationBeforeRetry 2m2s). Error: "Volume not attached according to node status for volume \"pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-b960256a-ea6f-11e7-901c-a2fa2817e02b\") pod \"busy-pod\" (UID: \"b9841c6e-ea6f-11e7-901c-a2fa2817e02b\") "
Dec 26 20:22:07 node2 kubelet[5443]: E1226 20:22:07.834457    5443 kubelet.go:1625] Unable to mount volumes for pod "busy-pod_default(b9841c6e-ea6f-11e7-901c-a2fa2817e02b)": timeout expired waiting for volumes to attach/mount for pod "default"/"busy-pod". list of unattached/unmounted volumes=[vol1]; skipping pod
Dec 26 20:22:07 node2 kubelet[5443]: E1226 20:22:07.834584    5443 pod_workers.go:186] Error syncing pod b9841c6e-ea6f-11e7-901c-a2fa2817e02b ("busy-pod_default(b9841c6e-ea6f-11e7-901c-a2fa2817e02b)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"busy-pod". list of unattached/unmounted volumes=[vol1]

@klausenbusk
Copy link
Contributor

Probably the same DNS issue as in #527, the flexplugin which is run by kube-controller-manager need a working network connection like the provisioner needs.

Please run: kubectl get pods --namespace=kube-system | grep kube-dns and kubectl get services --namespace=kube-system.

@tianhuil
Copy link
Contributor Author

Thanks. Here's the results of those two queries.

$ kubectl get pods --namespace=kube-system | grep kube-dns
kube-dns-6f4fd4bdf-znz9m                    3/3       Running   0          8d
$ kubectl get services --namespace=kube-system
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kube-dns               ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP   8d
kubernetes-dashboard   ClusterIP   10.98.40.240    <none>        443/TCP         8d

@klausenbusk
Copy link
Contributor

Thanks. Here's the results of those two queries.

Ìs all pods in the kube-system namespace running? kubectl get pods --namespace=kube-system

@tianhuil
Copy link
Contributor Author

Hi @klausenbusk: thanks for the quick response. They're all running quite hapily:

$ kubectl get pods --namespace=kube-system
NAME                                        READY     STATUS    RESTARTS   AGE
digitalocean-flexplugin-deploy-6znwf        1/1       Running   0          5m
digitalocean-flexplugin-deploy-bq95x        1/1       Running   0          5m
digitalocean-flexplugin-deploy-mw4kf        1/1       Running   0          5m
digitalocean-provisioner-6c7bbf4ccc-5h7lp   1/1       Running   0          5m
digitalocean-provisioner-6c7bbf4ccc-hrn7r   1/1       Running   0          5m
etcd-master                                 1/1       Running   0          8d
kube-apiserver-master                       1/1       Running   0          8d
kube-controller-manager-master              1/1       Running   0          8d
kube-dns-6f4fd4bdf-znz9m                    3/3       Running   0          8d
kube-flannel-ds-29cb5                       2/2       Running   1          8d
kube-flannel-ds-5jwbv                       2/2       Running   1          8d
kube-flannel-ds-xtmgr                       2/2       Running   2          8d
kube-proxy-k8k9q                            1/1       Running   0          8d
kube-proxy-qvdln                            1/1       Running   0          8d
kube-proxy-tqs4s                            1/1       Running   0          8d
kube-scheduler-master                       1/1       Running   0          8d
kubernetes-dashboard-7b7b5cd79b-6tx8f       1/1       Running   0          8d
tiller-deploy-6657cd6b8d-hh8fb              1/1       Running   0          26m

@tianhuil
Copy link
Contributor Author

And FYI, kube-controller-manager has public nameservers.

$ kubectl exec -n kube-system -it kube-controller-manager-master -- cat /etc/resolv.conf
nameserver 67.207.67.3
nameserver 67.207.67.2

@klausenbusk
Copy link
Contributor

And FYI, kube-controller-manager has public nameservers.

Is networking working inside that pod? Ex: curling api.digitalocean.com

@tianhuil
Copy link
Contributor Author

Yes, if the nameserver's work, I can get curl. Note: even on the digitalocean-flexplugin-deploy and digitalocean-provisioner, ping always worked, but wget would only work if the public nameservers are listed in /etc/resolv.conf:

$ kubectl exec -n kube-system -it kube-controller-manager-master -- sh
/ # wget yahoo.com
Connecting to yahoo.com (98.138.252.38:80)
Connecting to www.yahoo.com (98.139.180.180:443)
index.html           100% |*********************************************************************|   501k  0:00:00 ETA

@klausenbusk
Copy link
Contributor

Yes, if the nameserver's work, I can get curl. Note: even on the digitalocean-flexplugin-deploy and digitalocean-provisioner, ping always worked, but wget would only work if the public nameservers are listed in /etc/resolv.conf:

Hmm.. Please check the log of kube-controller-manager-master, it should show why the mounting is failing.
BTW: What version of k8s are you using?

@tianhuil
Copy link
Contributor Author

Sorry, I got a little lost and it took me a while to find where I was. To answer your questions:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T19:12:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

My client is OSX and the master and nodes are running Ubuntu 16.

Here are the last 30 lines from kubectl logs kube-controller-manager-master -n kube-system:

I1226 21:41:13.246531       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096008", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-f5krw
I1226 21:41:13.266757       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096008", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-hhdn5
I1226 21:41:13.273234       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096008", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-dd8zl
I1226 21:41:24.006003       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"63cd51fc-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096050", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set digitalocean-provisioner-6c7bbf4ccc to 0
I1226 21:41:24.050151       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"63d12ca7-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096052", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-2wzpz
I1226 21:41:24.069316       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"63d12ca7-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096052", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-wdfcd
I1226 21:41:46.164183       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"921fbdee-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096103", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set digitalocean-provisioner-6c7bbf4ccc to 2
I1226 21:41:46.194670       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"92232d91-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096104", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-tmjlz
I1226 21:41:46.217120       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"92232d91-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096104", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-stfns
I1226 21:41:46.392761       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-rhh2d
I1226 21:41:46.424706       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-z5qjt
I1226 21:41:46.427040       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-cbtvm
I1226 21:43:30.772895       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"d07cafd5-ea85-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1096315", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:43:30.958061       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"d07cafd5-ea85-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1096318", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:52:32.919198       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097085", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-cbtvm
I1226 21:52:32.931623       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097085", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-rhh2d
I1226 21:52:32.967663       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097085", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-z5qjt
E1226 21:52:33.036884       1 daemon_controller.go:265] kube-system/digitalocean-flexplugin-deploy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", ResourceVersion:"1097085", Generation:2, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649921306, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc422710a00), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"flexvolume-mount", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc422710a20), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"flex-deploy", Image:"quay.io/external_storage/digitalocean-flexplugin", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"DIGITALOCEAN_ACCESS_TOKEN", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc422710a40)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"flexvolume-mount", ReadOnly:false, MountPath:"/flexmnt", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc422829c90), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"1399756e-ea87-11e7-acd1-9801a7a79b1b":"139975b4-ea87-11e7-acd1-9801a7a79b1b"}, ServiceAccountName:"", DeprecatedServiceAccount:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc4226a0340), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc422f96368)}, MinReadySeconds:0, TemplateGeneration:2, RevisionHistoryLimit:(*int32)(0xc422829cc0)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:3, NumberMisscheduled:0, DesiredNumberScheduled:3, NumberReady:3, ObservedGeneration:1, UpdatedNumberScheduled:3, NumberAvailable:3, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "digitalocean-flexplugin-deploy": the object has been modified; please apply your changes to the latest version and try again
I1226 21:52:39.068118       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"92232d91-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-stfns
I1226 21:52:39.074212       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"921fbdee-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097119", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set digitalocean-provisioner-6c7bbf4ccc to 0
I1226 21:52:39.104308       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"92232d91-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-tmjlz
I1226 21:52:46.424373       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"1bafb70a-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097150", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set digitalocean-provisioner-6c7bbf4ccc to 2
I1226 21:52:46.448051       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"1bb18bc1-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097151", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-qhlwj
I1226 21:52:46.467575       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"1bb18bc1-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097151", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-jfq4h
I1226 21:52:46.660334       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"1bd0f401-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097168", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-q997m
I1226 21:52:46.674481       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"1bd0f401-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097168", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-f25q6
I1226 21:52:46.676432       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"1bd0f401-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097168", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-z8gq2
E1226 21:52:46.740345       1 daemon_controller.go:265] kube-system/digitalocean-flexplugin-deploy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/digitalocean-flexplugin-deploy", UID:"1bd0f401-ea87-11e7-901c-a2fa2817e02b", ResourceVersion:"1097174", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649921966, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc422eb9bc0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"flexvolume-mount", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc422eb9be0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"flex-deploy", Image:"quay.io/external_storage/digitalocean-flexplugin", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"DIGITALOCEAN_ACCESS_TOKEN", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc422eb9c00)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"flexvolume-mount", ReadOnly:false, MountPath:"/flexmnt", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc422eedfc0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", DeprecatedServiceAccount:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc422b5ee40), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc4227b4a78)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc422eedff0)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:1, NumberMisscheduled:0, DesiredNumberScheduled:3, NumberReady:0, ObservedGeneration:1, UpdatedNumberScheduled:1, NumberAvailable:0, NumberUnavailable:3, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "digitalocean-flexplugin-deploy": the object has been modified; please apply your changes to the latest version and try again
I1226 21:54:05.950958       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4b143baa-ea87-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1097359", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:54:06.063174       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4b143baa-ea87-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1097361", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator

@klausenbusk
Copy link
Contributor

Sorry, I got a little lost and it took me a while to find where I was. To answer your questions:

No problem..

Here are the last 30 lines from kubectl logs kube-controller-manager-master -n kube-system:

Not anything useful, the errors is probably more than 50 lines back.

BTW: What version of flannel is you using? flannel v0.9.1 contain a fix, which could explain the DNS issue: flannel-io/flannel#872

It would be a lot easier to help with direct access to the cluster..

@tianhuil
Copy link
Contributor Author

tianhuil commented Dec 26, 2017 •
edited
Loading

I'm running flannel v0.9.0-rc1 but will redeploy the k8s network and try again. If you send me your public key, I'll be happy to add you to k8s cluster's (the underlying DO boxes) and you can copy off the admin.conf and run whatever you need (I'll have to check with our CTO to make sure he's OK with that).

Here are the last 200 lines:

I1226 18:54:55.944867       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"40a76408-ea6e-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1081231", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 18:54:57.266522       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"40a76408-ea6e-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1081237", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 18:54:59.463164       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"40a76408-ea6e-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1081242", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 18:55:01.482781       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"40a76408-ea6e-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1081248", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 18:55:03.555117       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"40a76408-ea6e-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1081252", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 18:55:05.661563       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"40a76408-ea6e-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1081257", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 18:55:07.770487       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"40a76408-ea6e-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1081262", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 18:55:10.108351       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"40a76408-ea6e-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1081268", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 18:55:10.409278       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"40a76408-ea6e-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1081268", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 19:04:34.173241       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"9c3a20a8-ea6f-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1082106", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 19:04:34.186384       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"9c3a20a8-ea6f-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1082106", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 19:04:34.290393       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"9c3a20a8-ea6f-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1082109", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 19:04:36.300924       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"9c3a20a8-ea6f-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1082119", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 19:04:38.354957       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"9c3a20a8-ea6f-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1082124", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 19:04:40.455529       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"9c3a20a8-ea6f-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1082132", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 19:05:23.069518       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"b960256a-ea6f-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1082204", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 19:05:23.073271       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"b960256a-ea6f-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1082204", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 19:05:23.295799       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"b960256a-ea6f-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1082210", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
W1226 19:08:41.282982       1 reflector.go:341] k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of *v1.ConfigMap ended with: too old resource version: 1081103 (1081370)
W1226 19:21:38.320431       1 reflector.go:341] k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of *v1.ConfigMap ended with: too old resource version: 1082519 (1082556)
W1226 19:37:52.346849       1 reflector.go:341] k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of *v1.ConfigMap ended with: too old resource version: 1083634 (1083966)
W1226 19:52:58.370339       1 reflector.go:341] k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of *v1.ConfigMap ended with: too old resource version: 1085039 (1085259)
W1226 20:09:43.399426       1 reflector.go:341] k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of *v1.ConfigMap ended with: too old resource version: 1086374 (1086748)
W1226 20:24:04.434511       1 reflector.go:341] k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of *v1.ConfigMap ended with: too old resource version: 1087841 (1088008)
W1226 20:39:14.476047       1 reflector.go:341] k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of *v1.ConfigMap ended with: too old resource version: 1089077 (1089321)
I1226 20:45:35.729144       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"tiller-deploy", UID:"3f8bd886-e398-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1090940", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set tiller-deploy-6657cd6b8d to 0
I1226 20:45:35.768507       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"tiller-deploy-6657cd6b8d", UID:"3f8ee804-e398-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1090941", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: tiller-deploy-6657cd6b8d-mk4n4
I1226 20:50:22.558807       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"tiller-deploy", UID:"642b0acb-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091336", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set tiller-deploy-6657cd6b8d to 1
I1226 20:50:22.602385       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"tiller-deploy-6657cd6b8d", UID:"642d3730-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091337", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: tiller-deploy-6657cd6b8d-279gl
I1226 20:50:46.400911       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"jupyterhub-test", Name:"hub-deployment", UID:"47642cb2-e398-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091410", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set hub-deployment-66557459f to 0
I1226 20:50:46.421742       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"jupyterhub-test", Name:"hub-deployment-66557459f", UID:"47676b40-e398-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091411", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: hub-deployment-66557459f-nls64
I1226 20:50:49.565839       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"jupyterhub-test", Name:"proxy", UID:"47663e4e-e398-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091427", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set proxy-5b4875b498 to 0
I1226 20:50:49.591635       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"jupyterhub-test", Name:"proxy-5b4875b498", UID:"476b5f6f-e398-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091428", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: proxy-5b4875b498-lh4bj
I1226 20:52:31.345070       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"tiller-deploy", UID:"642b0acb-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091585", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set tiller-deploy-6657cd6b8d to 0
I1226 20:52:31.374300       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"tiller-deploy-6657cd6b8d", UID:"642d3730-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091586", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: tiller-deploy-6657cd6b8d-279gl
I1226 20:53:01.817206       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"tiller-deploy", UID:"c315e995-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091648", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set tiller-deploy-6657cd6b8d to 1
I1226 20:53:01.868854       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"tiller-deploy-6657cd6b8d", UID:"c3196faf-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091649", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: tiller-deploy-6657cd6b8d-hh8fb
I1226 20:53:31.795955       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"jupyterhub-test", Name:"hub-deployment", UID:"d4f43054-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091732", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set hub-deployment-f4cb5d7bd to 1
I1226 20:53:31.819018       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"jupyterhub-test", Name:"proxy", UID:"d4f77966-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091734", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set proxy-85b745ccf9 to 1
I1226 20:53:31.877143       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"jupyterhub-test", Name:"hub-deployment-f4cb5d7bd", UID:"d4f70961-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091733", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: hub-deployment-f4cb5d7bd-zlhb7
I1226 20:53:31.877388       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"jupyterhub-test", Name:"proxy-85b745ccf9", UID:"d4fa6a9c-ea7e-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1091735", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: proxy-85b745ccf9-s7t8t
I1226 21:12:11.494345       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"a18bbab9-ea6d-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093366", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-lrv44
I1226 21:12:11.498965       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"a18bbab9-ea6d-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093366", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-pglgd
I1226 21:12:11.501751       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"a18bbab9-ea6d-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093366", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-v8sng
E1226 21:12:11.531590       1 daemon_controller.go:265] kube-system/digitalocean-flexplugin-deploy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/digitalocean-flexplugin-deploy", UID:"a18bbab9-ea6d-11e7-901c-a2fa2817e02b", ResourceVersion:"1093366", Generation:2, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649911024, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc4214a6300), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"flexvolume-mount", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc4214a6320), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"flex-deploy", Image:"quay.io/external_storage/digitalocean-flexplugin", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"DIGITALOCEAN_ACCESS_TOKEN", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc4214a6360)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"flexvolume-mount", ReadOnly:false, MountPath:"/flexmnt", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc4223cc090), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"7054223c-ea81-11e7-8044-9801a7a79b1b":"70542282-ea81-11e7-8044-9801a7a79b1b"}, ServiceAccountName:"", DeprecatedServiceAccount:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc4223d1580), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc42000dfc8)}, MinReadySeconds:0, TemplateGeneration:2, RevisionHistoryLimit:(*int32)(0xc4223cc0c0)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:3, NumberMisscheduled:0, DesiredNumberScheduled:3, NumberReady:3, ObservedGeneration:1, UpdatedNumberScheduled:3, NumberAvailable:3, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "digitalocean-flexplugin-deploy": the object has been modified; please apply your changes to the latest version and try again
I1226 21:12:23.388921       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"9a85bdab-ea6d-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093408", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set digitalocean-provisioner-6c7bbf4ccc to 0
I1226 21:12:23.436361       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"9a8b5d2b-ea6d-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093410", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-trlz2
I1226 21:12:23.465267       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"9a8b5d2b-ea6d-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093410", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-4g256
I1226 21:13:54.720159       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"addee2cf-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093559", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set digitalocean-provisioner-6c7bbf4ccc to 2
I1226 21:13:54.777457       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"ade38ef5-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093560", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-5h7lp
I1226 21:13:54.797413       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"ade38ef5-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093560", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-hrn7r
I1226 21:13:54.954636       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"ae028038-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093577", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-6znwf
I1226 21:13:54.993348       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"ae028038-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093577", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-mw4kf
I1226 21:13:55.004885       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"ae028038-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1093577", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-bq95x
I1226 21:14:30.987623       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:14:41.335766       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:14:56.336504       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:15:11.339656       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:15:26.336836       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:15:41.339260       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:15:56.343175       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:16:11.339806       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:16:26.340696       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:16:41.338993       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:16:56.339668       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:17:11.341330       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:17:26.340907       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:17:41.341116       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:17:56.347451       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:18:11.344268       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:18:26.346894       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:18:41.346770       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:18:56.346574       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:19:11.345703       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:19:26.350915       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:19:41.347582       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:19:56.347176       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:20:11.348404       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:20:26.348170       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:20:41.350348       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:20:56.348575       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:21:11.349641       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:21:26.350009       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:21:41.350706       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:21:56.350028       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:22:11.349703       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:22:26.351778       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:22:41.351329       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:22:56.352249       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:23:11.352248       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:23:26.352270       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:23:41.353949       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:23:56.352174       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:24:11.353630       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:24:26.352990       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:24:41.355541       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:24:56.356117       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:25:11.356127       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:25:26.356320       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:25:41.357451       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:25:56.356949       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:26:11.357560       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:26:26.359628       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:26:41.362753       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:26:56.361346       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:27:11.361754       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:27:26.362775       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:27:41.364156       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:27:56.364244       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:28:11.364468       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:28:26.363923       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:28:41.366260       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:28:56.367549       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:29:11.367546       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:29:26.368126       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:29:41.372508       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:29:56.373041       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:30:11.368396       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:30:26.370703       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:30:41.371427       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:30:56.376356       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:31:11.370626       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:31:26.373113       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:31:41.372174       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:31:56.373658       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:32:11.372875       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:32:26.376269       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:32:41.382857       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:32:56.375187       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:33:11.374312       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:33:26.374296       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:33:41.377529       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:33:56.375850       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:34:11.380975       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:34:26.379446       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:34:41.376581       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:34:56.378569       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:35:11.378282       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:35:26.379226       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:35:41.379149       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:35:56.380458       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:36:11.381640       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:36:26.383629       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:36:41.384173       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:36:56.384178       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:37:11.384371       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:37:26.384712       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:37:41.384755       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:37:56.385281       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:38:11.387196       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:38:26.389819       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:38:41.387902       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:38:56.388074       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:39:11.388776       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:39:26.390207       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:39:41.392629       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"c37f6e75-ea81-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1093681", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:39:54.306968       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"ae028038-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095767", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-bq95x
I1226 21:39:54.310032       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"ae028038-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095767", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-mw4kf
I1226 21:39:54.327606       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"ae028038-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095767", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-6znwf
E1226 21:39:54.360429       1 daemon_controller.go:265] kube-system/digitalocean-flexplugin-deploy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/digitalocean-flexplugin-deploy", UID:"ae028038-ea81-11e7-901c-a2fa2817e02b", ResourceVersion:"1095767", Generation:2, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649919634, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc4225b3a40), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"flexvolume-mount", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc4225b3a60), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"flex-deploy", Image:"quay.io/external_storage/digitalocean-flexplugin", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"DIGITALOCEAN_ACCESS_TOKEN", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc4225b3aa0)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"flexvolume-mount", ReadOnly:false, MountPath:"/flexmnt", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc422678510), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"4f6fd968-ea85-11e7-9da2-9801a7a79b1b":"4f6fd9ae-ea85-11e7-9da2-9801a7a79b1b"}, ServiceAccountName:"", DeprecatedServiceAccount:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc4227731c0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc42000db18)}, MinReadySeconds:0, TemplateGeneration:2, RevisionHistoryLimit:(*int32)(0xc422678550)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:3, NumberMisscheduled:0, DesiredNumberScheduled:3, NumberReady:3, ObservedGeneration:1, UpdatedNumberScheduled:3, NumberAvailable:3, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "digitalocean-flexplugin-deploy": the object has been modified; please apply your changes to the latest version and try again
I1226 21:40:08.095894       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"addee2cf-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095819", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set digitalocean-provisioner-6c7bbf4ccc to 0
I1226 21:40:08.125523       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"ade38ef5-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095821", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-5h7lp
I1226 21:40:08.147494       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"ade38ef5-ea81-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095821", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-hrn7r
I1226 21:40:28.442064       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"63cd51fc-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095872", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set digitalocean-provisioner-6c7bbf4ccc to 2
I1226 21:40:28.505714       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"63d12ca7-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095873", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-2wzpz
I1226 21:40:28.539597       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"63d12ca7-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095873", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-wdfcd
I1226 21:40:28.852206       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095888", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-f5krw
I1226 21:40:28.897692       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095888", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-dd8zl
I1226 21:40:28.910556       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1095888", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-hhdn5
E1226 21:40:28.960624       1 daemon_controller.go:265] kube-system/digitalocean-flexplugin-deploy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", ResourceVersion:"1095888", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649921228, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc4232406a0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"flexvolume-mount", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc4232406c0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"flex-deploy", Image:"quay.io/external_storage/digitalocean-flexplugin", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"DIGITALOCEAN_ACCESS_TOKEN", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc4232406e0)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"flexvolume-mount", ReadOnly:false, MountPath:"/flexmnt", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc4231f7c10), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", DeprecatedServiceAccount:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc4233dec00), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc4227b4358)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc4231f7c40)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:0, NumberMisscheduled:0, DesiredNumberScheduled:0, NumberReady:0, ObservedGeneration:0, UpdatedNumberScheduled:0, NumberAvailable:0, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "digitalocean-flexplugin-deploy": the object has been modified; please apply your changes to the latest version and try again
I1226 21:41:13.246531       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096008", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-f5krw
I1226 21:41:13.266757       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096008", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-hhdn5
I1226 21:41:13.273234       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"63f7d08a-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096008", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-dd8zl
I1226 21:41:24.006003       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"63cd51fc-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096050", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set digitalocean-provisioner-6c7bbf4ccc to 0
I1226 21:41:24.050151       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"63d12ca7-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096052", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-2wzpz
I1226 21:41:24.069316       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"63d12ca7-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096052", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-wdfcd
I1226 21:41:46.164183       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"921fbdee-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096103", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set digitalocean-provisioner-6c7bbf4ccc to 2
I1226 21:41:46.194670       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"92232d91-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096104", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-tmjlz
I1226 21:41:46.217120       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"92232d91-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096104", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-stfns
I1226 21:41:46.392761       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-rhh2d
I1226 21:41:46.424706       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-z5qjt
I1226 21:41:46.427040       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1096121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-cbtvm
I1226 21:43:30.772895       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"d07cafd5-ea85-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1096315", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:43:30.958061       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"d07cafd5-ea85-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1096318", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:52:32.919198       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097085", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-cbtvm
I1226 21:52:32.931623       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097085", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-rhh2d
I1226 21:52:32.967663       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097085", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-flexplugin-deploy-z5qjt
E1226 21:52:33.036884       1 daemon_controller.go:265] kube-system/digitalocean-flexplugin-deploy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/digitalocean-flexplugin-deploy", UID:"9240feb8-ea85-11e7-901c-a2fa2817e02b", ResourceVersion:"1097085", Generation:2, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649921306, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc422710a00), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"flexvolume-mount", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc422710a20), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"flex-deploy", Image:"quay.io/external_storage/digitalocean-flexplugin", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"DIGITALOCEAN_ACCESS_TOKEN", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc422710a40)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"flexvolume-mount", ReadOnly:false, MountPath:"/flexmnt", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc422829c90), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"1399756e-ea87-11e7-acd1-9801a7a79b1b":"139975b4-ea87-11e7-acd1-9801a7a79b1b"}, ServiceAccountName:"", DeprecatedServiceAccount:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc4226a0340), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc422f96368)}, MinReadySeconds:0, TemplateGeneration:2, RevisionHistoryLimit:(*int32)(0xc422829cc0)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:3, NumberMisscheduled:0, DesiredNumberScheduled:3, NumberReady:3, ObservedGeneration:1, UpdatedNumberScheduled:3, NumberAvailable:3, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "digitalocean-flexplugin-deploy": the object has been modified; please apply your changes to the latest version and try again
I1226 21:52:39.068118       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"92232d91-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-stfns
I1226 21:52:39.074212       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"921fbdee-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097119", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set digitalocean-provisioner-6c7bbf4ccc to 0
I1226 21:52:39.104308       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"92232d91-ea85-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097121", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: digitalocean-provisioner-6c7bbf4ccc-tmjlz
I1226 21:52:46.424373       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"1bafb70a-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097150", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set digitalocean-provisioner-6c7bbf4ccc to 2
I1226 21:52:46.448051       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"1bb18bc1-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097151", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-qhlwj
I1226 21:52:46.467575       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"1bb18bc1-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097151", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-jfq4h
I1226 21:52:46.660334       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"1bd0f401-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097168", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-q997m
I1226 21:52:46.674481       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"1bd0f401-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097168", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-f25q6
I1226 21:52:46.676432       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"1bd0f401-ea87-11e7-901c-a2fa2817e02b", APIVersion:"extensions", ResourceVersion:"1097168", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-z8gq2
E1226 21:52:46.740345       1 daemon_controller.go:265] kube-system/digitalocean-flexplugin-deploy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/digitalocean-flexplugin-deploy", UID:"1bd0f401-ea87-11e7-901c-a2fa2817e02b", ResourceVersion:"1097174", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649921966, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc422eb9bc0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"flexvolume-mount", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc422eb9be0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"flex-deploy", Image:"quay.io/external_storage/digitalocean-flexplugin", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"DIGITALOCEAN_ACCESS_TOKEN", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc422eb9c00)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"flexvolume-mount", ReadOnly:false, MountPath:"/flexmnt", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc422eedfc0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", DeprecatedServiceAccount:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc422b5ee40), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc4227b4a78)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc422eedff0)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:1, NumberMisscheduled:0, DesiredNumberScheduled:3, NumberReady:0, ObservedGeneration:1, UpdatedNumberScheduled:1, NumberAvailable:0, NumberUnavailable:3, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "digitalocean-flexplugin-deploy": the object has been modified; please apply your changes to the latest version and try again
I1226 21:54:05.950958       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4b143baa-ea87-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1097359", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1226 21:54:06.063174       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4b143baa-ea87-11e7-901c-a2fa2817e02b", APIVersion:"v1", ResourceVersion:"1097361", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator

@klausenbusk
Copy link
Contributor

klausenbusk commented Dec 26, 2017 •
edited
Loading

Here are the last 200 lines:

Still not anything interesting, weird.. hmm

@tianhuil
Copy link
Contributor Author

tianhuil commented Dec 27, 2017 •
edited
Loading

@klausenbusk: I think I have the interesting error you've been looking for. Upgraded the server to use flannel v0.9.1, got rid of the /etc/resolv.conf hack. Here are the results of the test

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM         STORAGECLASS   REASON    AGE
hub-db-dir-volume                          2Gi        RWO            Retain           Available                                          1h
pvc-4163f7dd-eaae-11e7-87b4-e24528ed5ba2   1Gi        RWO            Delete           Bound       default/pv1   default                  1h
$ kubectl get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pv1       Bound     pvc-4163f7dd-eaae-11e7-87b4-e24528ed5ba2   1Gi        RWO            default        1h
$ kubectl get pod
NAME       READY     STATUS              RESTARTS   AGE
busy-pod   0/1       ContainerCreating   0          1h

showing that the pod is still stuck. And here are the results of

kubectl logs -n kube-system kube-controller-manager-master

(or at least the last 100 lines thereof):

I1227 02:09:27.031099       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:09:32.033467       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:09:37.037220       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:09:42.053001       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:09:47.056990       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:09:52.059399       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:09:57.067166       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:02.070638       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:07.071361       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:12.071885       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:17.076077       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:22.077148       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:27.081043       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:32.083687       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:37.085134       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:42.087834       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:47.088344       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:52.089175       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:10:57.090509       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:02.092525       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:07.097692       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:12.098666       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:17.099193       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:22.102143       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:27.102773       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:32.103705       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:37.104277       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:42.104875       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:47.106363       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:52.108492       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:11:57.109228       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:02.113185       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:07.114725       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:12.115153       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:16.888786       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"tiller-deploy", UID:"5c60bd15-eaab-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"831", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set tiller-deploy-6657cd6b8d to 1
I1227 02:12:16.943729       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"tiller-deploy-6657cd6b8d", UID:"5c68fdd4-eaab-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"832", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: tiller-deploy-6657cd6b8d-xzh6r
I1227 02:12:17.117115       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:22.119871       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:27.132683       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:32.133149       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:37.134132       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:42.135167       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:47.136041       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:52.136495       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:12:57.137136       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:02.138109       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:07.141140       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:12.143932       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:17.144509       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:22.146312       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:27.150757       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:32.152591       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:37.157072       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:42.159867       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:47.160440       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:52.161343       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:13:57.162537       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:14:02.165096       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:14:07.167524       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:14:12.168022       1 node_controller.go:611] Initializing eviction metric for zone: 
I1227 02:14:12.268593       1 logs.go:41] [INFO] signed certificate with serial number 227435845352100554766971070084664868867868949586
E1227 02:14:12.676117       1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="node1"  does not exist
I1227 02:14:12.734276       1 range_allocator.go:307] Set node node1 PodCIDR to 10.244.1.0/24
I1227 02:14:12.873679       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-flannel-ds", UID:"9a78912e-eaaa-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"501", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-flannel-ds-28nw6
I1227 02:14:12.916069       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"97cb64f5-eaaa-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"435", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-9xfft
E1227 02:14:12.981260       1 daemon_controller.go:265] kube-system/kube-flannel-ds failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-flannel-ds", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/kube-flannel-ds", UID:"9a78912e-eaaa-11e7-87b4-e24528ed5ba2", ResourceVersion:"996", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649937211, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"flannel", "tier":"node"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc42264fee0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"flannel", "tier":"node"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"run", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc42264ff00), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"cni", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc42264ff20), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"flannel-cfg", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc4227fd200), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container{v1.Container{Name:"install-cni", Image:"quay.io/coreos/flannel:v0.9.1-amd64", Command:[]string{"cp"}, Args:[]string{"-f", "/etc/kube-flannel/cni-conf.json", "/etc/cni/net.d/10-flannel.conf"}, WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar(nil), Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"cni", ReadOnly:false, MountPath:"/etc/cni/net.d", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"flannel-cfg", ReadOnly:false, MountPath:"/etc/kube-flannel/", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, Containers:[]v1.Container{v1.Container{Name:"kube-flannel", Image:"quay.io/coreos/flannel:v0.9.1-amd64", Command:[]string{"/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"POD_NAME", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc42264ff60)}, v1.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc42264ffa0)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"run", ReadOnly:false, MountPath:"/run", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"flannel-cfg", ReadOnly:false, MountPath:"/etc/kube-flannel/", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc4227fd380), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc4225d35e0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"beta.kubernetes.io/arch":"amd64"}, ServiceAccountName:"flannel", DeprecatedServiceAccount:"flannel", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc4227fd3c0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node-role.kubernetes.io/master", Operator:"Exists", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"OnDelete", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(nil)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc4225d3688)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:1, NumberMisscheduled:0, DesiredNumberScheduled:2, NumberReady:1, ObservedGeneration:1, UpdatedNumberScheduled:1, NumberAvailable:1, NumberUnavailable:1, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "kube-flannel-ds": the object has been modified; please apply your changes to the latest version and try again
E1227 02:14:12.992985       1 daemon_controller.go:265] kube-system/kube-proxy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/kube-proxy", UID:"97cb64f5-eaaa-11e7-87b4-e24528ed5ba2", ResourceVersion:"435", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649937207, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc422a04fc0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"kube-proxy", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc4229bb980), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc422a04fe0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc422a05000), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kube-proxy", Image:"gcr.io/google_containers/kube-proxy-amd64:v1.9.0", Command:[]string{"/usr/local/bin/kube-proxy", "--config=/var/lib/kube-proxy/config.conf"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar(nil), Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"kube-proxy", ReadOnly:false, MountPath:"/var/lib/kube-proxy", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc4229bba00), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc422a00928), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"kube-proxy", DeprecatedServiceAccount:"kube-proxy", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc4229bba40), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node-role.kubernetes.io/master", Operator:"", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"node.cloudprovider.kubernetes.io/uninitialized", Operator:"", Value:"true", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc4225fec20)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc422a0097c)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:1, NumberMisscheduled:0, DesiredNumberScheduled:1, NumberReady:1, ObservedGeneration:1, UpdatedNumberScheduled:1, NumberAvailable:1, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "kube-proxy": the object has been modified; please apply your changes to the latest version and try again
I1227 02:14:17.168535       1 node_controller.go:611] Initializing eviction metric for zone: 
W1227 02:14:17.169025       1 node_controller.go:964] Missing timestamp for Node node1. Assuming now as a timestamp.
I1227 02:14:17.169849       1 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"node1", UID:"a16b3400-eaab-11e7-87b4-e24528ed5ba2", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node node1 event: Registered Node node1 in Controller
I1227 02:14:17.169910       1 node_controller.go:830] Controller detected that all Nodes are not-Ready. Entering master disruption mode.
I1227 02:14:37.172123       1 node_controller.go:857] Controller detected that some Nodes are Ready. Exiting master disruption mode.
I1227 02:15:02.626736       1 logs.go:41] [INFO] signed certificate with serial number 235313528485840242928445915715544159380187352583
E1227 02:15:03.258245       1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="node2"  does not exist
I1227 02:15:03.400452       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-flannel-ds", UID:"9a78912e-eaaa-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"1123", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-flannel-ds-2qx8r
I1227 02:15:03.400507       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"97cb64f5-eaaa-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"1056", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-lx6dk
I1227 02:15:03.431848       1 range_allocator.go:307] Set node node2 PodCIDR to 10.244.2.0/24
E1227 02:15:03.564716       1 daemon_controller.go:265] kube-system/kube-proxy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/kube-proxy", UID:"97cb64f5-eaaa-11e7-87b4-e24528ed5ba2", ResourceVersion:"1056", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649937207, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc420fe2a80), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"kube-proxy", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc42260d500), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc420fe2aa0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc420fe2ac0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kube-proxy", Image:"gcr.io/google_containers/kube-proxy-amd64:v1.9.0", Command:[]string{"/usr/local/bin/kube-proxy", "--config=/var/lib/kube-proxy/config.conf"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar(nil), Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"kube-proxy", ReadOnly:false, MountPath:"/var/lib/kube-proxy", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc42260d580), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc421831e98), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"kube-proxy", DeprecatedServiceAccount:"kube-proxy", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc42260d5c0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node-role.kubernetes.io/master", Operator:"", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"node.cloudprovider.kubernetes.io/uninitialized", Operator:"", Value:"true", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc420f6ca70)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc421831fec)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:2, NumberMisscheduled:0, DesiredNumberScheduled:2, NumberReady:2, ObservedGeneration:1, UpdatedNumberScheduled:2, NumberAvailable:2, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "kube-proxy": the object has been modified; please apply your changes to the latest version and try again
W1227 02:15:07.188559       1 node_controller.go:964] Missing timestamp for Node node2. Assuming now as a timestamp.
I1227 02:15:07.196334       1 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"node2", UID:"bf8ece12-eaab-11e7-87b4-e24528ed5ba2", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node node2 event: Registered Node node2 in Controller
I1227 02:21:42.805618       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"jupyterhub-test", Name:"hub-deployment", UID:"ada60daa-eaac-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"1730", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set hub-deployment-5fdd998d89 to 1
I1227 02:21:42.860823       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"jupyterhub-test", Name:"proxy", UID:"adaba446-eaac-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"1732", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set proxy-86cc5947d to 1
I1227 02:21:42.941513       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"jupyterhub-test", Name:"hub-deployment-5fdd998d89", UID:"adb7852f-eaac-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"1733", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: hub-deployment-5fdd998d89-jmjbh
I1227 02:21:42.975215       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"jupyterhub-test", Name:"proxy-86cc5947d", UID:"adb8e6bd-eaac-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"1736", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: proxy-86cc5947d-cftcx
I1227 02:30:12.003117       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"dd340bfe-eaad-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"2510", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set digitalocean-provisioner-6c7bbf4ccc to 2
I1227 02:30:12.132733       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"dd37cef9-eaad-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"2511", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-j9ctc
I1227 02:30:12.208122       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-6c7bbf4ccc", UID:"dd37cef9-eaad-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"2511", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-6c7bbf4ccc-dszxb
I1227 02:30:12.515452       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"dd665754-eaad-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"2522", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-qpthp
I1227 02:30:12.715661       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"dd665754-eaad-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"2522", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-kl7db
I1227 02:30:12.788302       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"dd665754-eaad-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"2522", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-qzl6g
E1227 02:30:20.674204       1 daemon_controller.go:265] kube-system/digitalocean-flexplugin-deploy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/digitalocean-flexplugin-deploy", UID:"dd665754-eaad-11e7-87b4-e24528ed5ba2", ResourceVersion:"2573", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649938612, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc4214509c0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"flexvolume-mount", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc4214509e0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"flex-deploy", Image:"quay.io/external_storage/digitalocean-flexplugin", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"DIGITALOCEAN_ACCESS_TOKEN", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc421450a00)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"flexvolume-mount", ReadOnly:false, MountPath:"/flexmnt", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc4202bd1a0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", DeprecatedServiceAccount:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc420ee8bc0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc420f6c830)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc4202bd230)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:3, NumberMisscheduled:0, DesiredNumberScheduled:3, NumberReady:1, ObservedGeneration:1, UpdatedNumberScheduled:3, NumberAvailable:1, NumberUnavailable:2, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "digitalocean-flexplugin-deploy": the object has been modified; please apply your changes to the latest version and try again
I1227 02:33:00.077043       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4163f7dd-eaae-11e7-87b4-e24528ed5ba2", APIVersion:"v1", ResourceVersion:"2826", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1227 02:33:00.349230       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4163f7dd-eaae-11e7-87b4-e24528ed5ba2", APIVersion:"v1", ResourceVersion:"2828", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1227 02:33:01.853281       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4163f7dd-eaae-11e7-87b4-e24528ed5ba2", APIVersion:"v1", ResourceVersion:"2828", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1227 02:33:02.543469       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4163f7dd-eaae-11e7-87b4-e24528ed5ba2", APIVersion:"v1", ResourceVersion:"2840", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1227 02:33:04.810003       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4163f7dd-eaae-11e7-87b4-e24528ed5ba2", APIVersion:"v1", ResourceVersion:"2844", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1227 02:33:07.342639       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4163f7dd-eaae-11e7-87b4-e24528ed5ba2", APIVersion:"v1", ResourceVersion:"2849", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1227 02:33:09.840931       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4163f7dd-eaae-11e7-87b4-e24528ed5ba2", APIVersion:"v1", ResourceVersion:"2857", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I1227 02:33:12.244857       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"4163f7dd-eaae-11e7-87b4-e24528ed5ba2", APIVersion:"v1", ResourceVersion:"2864", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
W1227 02:37:27.694868       1 reflector.go:341] k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of *v1.ConfigMap ended with: too old resource version: 235 (2065)

@klausenbusk
Copy link
Contributor

Could you run kubectl logs -n kube-system kube-controller-manager-master | grep -i flex?

@tianhuil
Copy link
Contributor Author

tianhuil commented Dec 27, 2017 •
edited
Loading

Here it is:

$ kubectl logs -n kube-system kube-controller-manager-master | grep -i flex
W1227 02:06:45.669273       1 probe.go:215] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
E1227 02:07:02.538167       1 daemon_controller.go:265] kube-system/kube-proxy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/kube-proxy", UID:"97cb64f5-eaaa-11e7-87b4-e24528ed5ba2", ResourceVersion:"183", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649937207, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc420b30820), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"kube-proxy", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc421150240), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc420b30860), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc420b308e0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kube-proxy", Image:"gcr.io/google_containers/kube-proxy-amd64:v1.9.0", Command:[]string{"/usr/local/bin/kube-proxy", "--config=/var/lib/kube-proxy/config.conf"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar(nil), Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"kube-proxy", ReadOnly:false, MountPath:"/var/lib/kube-proxy", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc4211502c0), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc4201c13b8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"kube-proxy", DeprecatedServiceAccount:"kube-proxy", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc421150300), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node-role.kubernetes.io/master", Operator:"", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"node.cloudprovider.kubernetes.io/uninitialized", Operator:"", Value:"true", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc420074b10)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc4201c158c)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:0, NumberMisscheduled:0, DesiredNumberScheduled:0, NumberReady:0, ObservedGeneration:0, UpdatedNumberScheduled:0, NumberAvailable:0, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "kube-proxy": the object has been modified; please apply your changes to the latest version and try again
E1227 02:14:12.981260       1 daemon_controller.go:265] kube-system/kube-flannel-ds failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-flannel-ds", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/kube-flannel-ds", UID:"9a78912e-eaaa-11e7-87b4-e24528ed5ba2", ResourceVersion:"996", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649937211, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"flannel", "tier":"node"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc42264fee0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"flannel", "tier":"node"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"run", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc42264ff00), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"cni", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc42264ff20), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"flannel-cfg", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc4227fd200), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container{v1.Container{Name:"install-cni", Image:"quay.io/coreos/flannel:v0.9.1-amd64", Command:[]string{"cp"}, Args:[]string{"-f", "/etc/kube-flannel/cni-conf.json", "/etc/cni/net.d/10-flannel.conf"}, WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar(nil), Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"cni", ReadOnly:false, MountPath:"/etc/cni/net.d", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"flannel-cfg", ReadOnly:false, MountPath:"/etc/kube-flannel/", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, Containers:[]v1.Container{v1.Container{Name:"kube-flannel", Image:"quay.io/coreos/flannel:v0.9.1-amd64", Command:[]string{"/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"POD_NAME", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc42264ff60)}, v1.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc42264ffa0)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"run", ReadOnly:false, MountPath:"/run", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"flannel-cfg", ReadOnly:false, MountPath:"/etc/kube-flannel/", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc4227fd380), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc4225d35e0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"beta.kubernetes.io/arch":"amd64"}, ServiceAccountName:"flannel", DeprecatedServiceAccount:"flannel", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc4227fd3c0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node-role.kubernetes.io/master", Operator:"Exists", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"OnDelete", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(nil)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc4225d3688)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:1, NumberMisscheduled:0, DesiredNumberScheduled:2, NumberReady:1, ObservedGeneration:1, UpdatedNumberScheduled:1, NumberAvailable:1, NumberUnavailable:1, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "kube-flannel-ds": the object has been modified; please apply your changes to the latest version and try again
E1227 02:14:12.992985       1 daemon_controller.go:265] kube-system/kube-proxy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/kube-proxy", UID:"97cb64f5-eaaa-11e7-87b4-e24528ed5ba2", ResourceVersion:"435", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649937207, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc422a04fc0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"kube-proxy", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc4229bb980), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc422a04fe0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc422a05000), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kube-proxy", Image:"gcr.io/google_containers/kube-proxy-amd64:v1.9.0", Command:[]string{"/usr/local/bin/kube-proxy", "--config=/var/lib/kube-proxy/config.conf"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar(nil), Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"kube-proxy", ReadOnly:false, MountPath:"/var/lib/kube-proxy", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc4229bba00), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc422a00928), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"kube-proxy", DeprecatedServiceAccount:"kube-proxy", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc4229bba40), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node-role.kubernetes.io/master", Operator:"", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"node.cloudprovider.kubernetes.io/uninitialized", Operator:"", Value:"true", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc4225fec20)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc422a0097c)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:1, NumberMisscheduled:0, DesiredNumberScheduled:1, NumberReady:1, ObservedGeneration:1, UpdatedNumberScheduled:1, NumberAvailable:1, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "kube-proxy": the object has been modified; please apply your changes to the latest version and try again
E1227 02:15:03.564716       1 daemon_controller.go:265] kube-system/kube-proxy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/kube-proxy", UID:"97cb64f5-eaaa-11e7-87b4-e24528ed5ba2", ResourceVersion:"1056", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649937207, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc420fe2a80), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"kube-proxy", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc42260d500), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc420fe2aa0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc420fe2ac0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kube-proxy", Image:"gcr.io/google_containers/kube-proxy-amd64:v1.9.0", Command:[]string{"/usr/local/bin/kube-proxy", "--config=/var/lib/kube-proxy/config.conf"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar(nil), Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"kube-proxy", ReadOnly:false, MountPath:"/var/lib/kube-proxy", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc42260d580), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc421831e98), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"kube-proxy", DeprecatedServiceAccount:"kube-proxy", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc42260d5c0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node-role.kubernetes.io/master", Operator:"", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"node.cloudprovider.kubernetes.io/uninitialized", Operator:"", Value:"true", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc420f6ca70)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc421831fec)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:2, NumberMisscheduled:0, DesiredNumberScheduled:2, NumberReady:2, ObservedGeneration:1, UpdatedNumberScheduled:2, NumberAvailable:2, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "kube-proxy": the object has been modified; please apply your changes to the latest version and try again
I1227 02:30:12.515452       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"dd665754-eaad-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"2522", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-qpthp
I1227 02:30:12.715661       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"dd665754-eaad-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"2522", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-kl7db
I1227 02:30:12.788302       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"dd665754-eaad-11e7-87b4-e24528ed5ba2", APIVersion:"extensions", ResourceVersion:"2522", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-qzl6g
E1227 02:30:20.674204       1 daemon_controller.go:265] kube-system/digitalocean-flexplugin-deploy failed with : error storing status for daemon set &v1beta1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/extensions/v1beta1/namespaces/kube-system/daemonsets/digitalocean-flexplugin-deploy", UID:"dd665754-eaad-11e7-87b4-e24528ed5ba2", ResourceVersion:"2573", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63649938612, loc:(*time.Location)(0x5039040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc4214509c0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"digitalocean-flexplugin-deploy", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"digitalocean-flexplugin-deploy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"flexvolume-mount", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc4214509e0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"flex-deploy", Image:"quay.io/external_storage/digitalocean-flexplugin", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"DIGITALOCEAN_ACCESS_TOKEN", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc421450a00)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:25, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"25m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:33554432, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"flexvolume-mount", ReadOnly:false, MountPath:"/flexmnt", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil)}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc4202bd1a0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", DeprecatedServiceAccount:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, SecurityContext:(*v1.PodSecurityContext)(0xc420ee8bc0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil)}}, UpdateStrategy:v1beta1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1beta1.RollingUpdateDaemonSet)(0xc420f6c830)}, MinReadySeconds:0, TemplateGeneration:1, RevisionHistoryLimit:(*int32)(0xc4202bd230)}, Status:v1beta1.DaemonSetStatus{CurrentNumberScheduled:3, NumberMisscheduled:0, DesiredNumberScheduled:3, NumberReady:1, ObservedGeneration:1, UpdatedNumberScheduled:3, NumberAvailable:1, NumberUnavailable:2, CollisionCount:(*int32)(nil), Conditions:[]v1beta1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.extensions "digitalocean-flexplugin-deploy": the object has been modified; please apply your changes to the latest version and try again

@klausenbusk
Copy link
Contributor

Could you run find /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ on the master node and a regular node?

@tianhuil
Copy link
Contributor Author

Sure! The results appear to be the same:

root@master:~#  find /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/external-storage~digitalocean
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/external-storage~digitalocean/do_token
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/external-storage~digitalocean/digitalocean

root@node2:~#
root@node2:~# find /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/external-storage~digitalocean
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/external-storage~digitalocean/do_token
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/external-storage~digitalocean/digitalocean

root@node1:~# find /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/external-storage~digitalocean
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/external-storage~digitalocean/do_token
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/external-storage~digitalocean/digitalocean

@klausenbusk
Copy link
Contributor

Could you run the same command inside the kube-controller-manager-master pod?

@tianhuil tianhuil reopened this Dec 27, 2017
@tianhuil
Copy link
Contributor Author

And in case you're curious:

# ll /usr/libexec/kubernetes/kubelet-plugins/volume/exec/*
total 6544
drwxr-xr-x 2 root root    4096 Dec 27 02:30 ./
drwxr-xr-x 3 root root    4096 Dec 27 02:30 ../
-rwxr-xr-x 1 root root 6685080 Dec 27 02:30 digitalocean*
-rw------- 1 root root      65 Dec 27 02:30 do_token

@tianhuil
Copy link
Contributor Author

tianhuil commented Dec 27, 2017 •
edited
Loading

And for your previous question:

$ kubectl exec -it -n kube-system kube-controller-manager-master -- find /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/

There's just that one file.

@klausenbusk
Copy link
Contributor

klausenbusk commented Dec 27, 2017 •
edited
Loading

Aha, that explain it.
Without access to the flexplugin, kube-controller-manager can't attach/detach the volume. Try adding a hostPath mount to the manifest file (/etc/kubernetes/manifests/kube-controller-manager.yaml), look at digitalocean-flexdeploy.yaml for "inspiration".

@tianhuil
Copy link
Contributor Author

tianhuil commented Dec 27, 2017 •
edited
Loading

That works!

$ kubectl describe pod kube-controller-manager-master -n kube-system
...
    Mounts:
      /etc/kubernetes/controller-manager.conf from kubeconfig (ro)
      /etc/kubernetes/pki from k8s-certs (ro)
      /etc/ssl/certs from ca-certs (ro)
      /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ from flexvolume-mount (rw)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  kubeconfig:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/controller-manager.conf
    HostPathType:  FileOrCreate
  k8s-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki
    HostPathType:  DirectoryOrCreate
  ca-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  flexvolume-mount:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
    HostPathType:

And the result of spinning up a pod with a PVC:


$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM         STORAGECLASS   REASON    AGE
pvc-cb5ff9e6-eb1e-11e7-87b4-e24528ed5ba2   1Gi        RWO            Delete           Bound       default/pv1   default                  2m
$ kubectl get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pv1       Bound     pvc-cb5ff9e6-eb1e-11e7-87b4-e24528ed5ba2   1Gi        RWO            default        2m

$ kubectl get pods
NAME       READY     STATUS    RESTARTS   AGE
busy-pod   1/1       Running   0          41s

Thanks so much for the tireless debugging @klausenbusk! Should I add this to the README.md? Curious why you never had to add the mount to kube-controller-manager-master.

@klausenbusk
Copy link
Contributor

Thanks so much for the tireless debugging @klausenbusk!

No problem.

Should I add this to the README.md?

It should definitely be documented to help users. Maybe as a Note under point 3: Please ensure the flexvolume plugin dir path is accessible for kube-controller-manager or something like that, feel free to open a PR.

Curious why you never had to add the mount to kube-controller-manager-master.

I use bootkube (on Container Linux) which work slightly different than kubeadm, but I also added the hostPath mount to the kube-controller-manager manifest file (I even changed --flex-volume-plugin-dir as /usr is read-only on Container Linux). I just thought it wasn't need for most setup.

Regard kubeadm it already add the hostPath mount, but it was later changed to only add the mount if flexvolumeDir (default: /usr/libexec/kubernetes/kubelet-plugins/volume/exec) exists. The check is done when running kubeadm init afaik.

@klausenbusk klausenbusk mentioned this issue Dec 27, 2017
Closed
@klausenbusk
Copy link
Contributor

It should definitely be documented to help users.

I have opened a issue to keep track of it: #531 , as said before: feel free to open a PR.

@tianhuil tianhuil mentioned this issue Jan 1, 2018
Closed
@tianhuil
Copy link
Contributor Author

tianhuil commented Jan 1, 2018

Done!

@klausenbusk klausenbusk mentioned this issue Jan 2, 2018
Closed
@lloeki lloeki mentioned this issue Jan 24, 2018
Closed
@klausenbusk
Copy link
Contributor

/area digitalocean

Just for the record.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area/digitalocean
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@klausenbusk @tianhuil @k8s-ci-robot