limit provision synchronous operation numbers

[wenjun93]

if there is a huge number of pvc request at the same time, for each pvc, controller will create a leaderElector and spawn a go routine, this will lead to huge map of goroutines in the controller, and cause the provisioner not working.
per my test, the provisioner was stuck when creating 100 PVCs .

[rootfs]

/assign @jsafrane

[msau42]

What is it stuck on? Theoretically, go supports 10k goroutines by default.

[wenjun93]

@msau42

[wenjun93]

here is the log when create a number of pvc at the same time, and most pvc stuck in Pending status

I0209 13:17:26.656069 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-79[2f771cf8-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656082 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-8[2fa18feb-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656098 7 controller.go:1076] scheduleOperation[provision-default/fail-pvc-105[1ee3acf9-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656115 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-36[26668eb1-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656128 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-51[2a24fe63-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656156 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-54[2aa5dd80-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656170 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-93[325e7d33-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656182 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-31[25801803-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656195 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-39[27944e75-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656209 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-4[27be610b-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656222 7 controller.go:1076] scheduleROperation[lock-provision-default/fail-pvc-96[32e0b818-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656238 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-118[21559e41-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656253 7 controller.go:1076] scheduleOperation[provision-default/fail-pvc-20[2361154e-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656273 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-82[30283674-0d9b-11e8-86aa-90e2baee0748]]
I0209 13:17:26.656302 7 controller.go:1076] scheduleOperation[lock-provision-default/fail-pvc-7[2da193ba-0d9b-11e8-86aa-90e2baee0748]]

[msau42]

Yes but why aren't they making any progress? Why does reducing the number of parallel operations help?

[wongma7]

We will be getting rid of the leader election entirely I think, let's wait for that kubernetes-csi/external-provisioner#68 (comment)