Postpone Deletion of a Persistent Volume Claim in case It Is Used by a Pod #1174

pospispa · 2017-10-11T15:12:24Z

Proposal for postponing deletion of Persistent Volume Claim in case it's used by a pod.

Implementation will fix issue kubernetes/kubernetes#45143

/sig storage
/cc @kubernetes/sig-storage-proposals

msau42 · 2017-10-11T18:11:55Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+- In case the PVC is used by a pod: sets condition Terminating to true for the PVC.
+
+### Scheduler
+Scheduler will check if a pod uses a PVC and if any of the PVCs has condition Terminating set to true. In case this is true the scheduling of the pod will end up with an error: "PVC (%pvcName) is in Terminating state".


Can you just check if the PVC has the deletion timestamp set?

anything that reacts to absence of a PV or PVC (scheduler, kubelet) should also react to the presence of a deletionTimestamp on a PV or PVC

Let me firstly think loudly about the deletionTimestamp. The description of the deletionTimestamp states: "The resource will be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field." So in case the deletionTimestamp is set for a PVC the PVC must be deleted after the time in the deletionTimestamp. However, in case the PVC is used by a pod it's not guaranteed that the pod won't be using the PVC after the time in the deletionTimestamp. That's why I prefer not to set the deletionTimestamp in PVCs, but use the condition Terminating for storing the information that the PVC is going to be deleted.

On the other hand, scheduler must be able to decide whether PVC is going to be deleted or not in order to fail scheduling of a pod that uses the PVC that is going to be deleted.
So it will be better if the condition that PVC is going to be deleted will be: deletionTimestampIsSet(PVC) || terminatingConditionIsTrue(PVC).

An object with a deletionTimestamp and non-empty finalizers blocks until finalizers are removed. The description of deletionTimestamp should be updated. The scheduler and kubelet should not schedule or mount based on PVs or PVCs with deletionTimestamps set

I assume that after kubectl delete pvc pvc-name the PVC may or may not have the deletionTimestamp set. So it's not possible to rely solely on deletionTimestamp for finding out whether a PVC is being deleted or not. Am I right?

I assume that after kubectl delete pvc pvc-name the PVC may or may not have the deletionTimestamp set

Not sure what you mean. An effective DELETE API call either sets the deletionTimestamp or completely removes the object from etcd.

So in case of a PVC with a finalizer set the DELETE API call will always set the deletionTimestamp as the deletion has to be postponed because of the finalizer.
Thus, there can't be a PVC that is being deleted with deletionTimestamp == nil.
That said, there is no need for using condition Terminating for marking a PVC that is being deleted.

msau42 · 2017-10-11T18:21:03Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+When a PVC is created finalizer is added into its metadata.
+
+### Finalizer
+Finalizer can list pods and watch for pods. It caches status of pods that use a PVC. It also watches for PVC deletes.


Is this a new controller to handle PVC finalizer?

Yes, it's a new internal controller.

msau42 · 2017-10-11T18:29:20Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+Finalizer can list pods and watch for pods. It caches status of pods that use a PVC. It also watches for PVC deletes.
+
+Finalizer reacts to PVC delete in one of the below ways:
+- In case the PVC is not used by a pod: removes the finalizer information from the PVC metadata.


Could you have a race condition between adding and deleting pods?

Pod A is currently using PVC.

Pod A is deleted.

Soon after, Pod B is created and scheduled that uses PVC.

PVC is deleted.

Informer gets Pod delete for Pod A, sees no other Pod is using the PVC in its cache, and removes finalizer.

Informer gets Pod add for Pod B, but finalizer is already deleted.

Presence of a PV or PVC deletion timestamp should fail setting up a PVC volume for a pod. That would prevent new pods from starting once the PV or PVC had started deletion

What if Pod B got setup with the volume before the informer receives the updates?

I agree with @msau42 that the informer may get PVC delete before Pod B setup so in case the finalizer is removed immediately after PVC delete there is a chance for a race condition.

Currently, I can't think of other solution for the race condition than to postpone the finalizer removal from PVC for at least deletionPostponePeriod.

That's not really safe either. The controller could be down.

Isn't this best effort? How strong of a guarantee is required? The description above implies a strong guarantee isn't required, and is vague about what the consequences are.

That's not really safe either.

I agree.

The controller could be down.

In case the controller is done the finalizer setting is not being removed from PVCs so PVCs are not being deleted and PVCs marked for deletion are piling up.

In addition, the controller is not setting finalizer for newly created PVCs.

Isn't this best effort? How strong of a guarantee is required? The description above implies a strong guarantee isn't required, and is vague about what the consequences are.

The worst possible consequences are data lost in some of the possible scenarios that always depend on the storage type being used:

in case the dynamic provisioning is used and reclaim policy is Delete the PVC deletion triggers deletion of the associated storage asset and PV.

the same as above applies for the static provisioning and Delete reclaim policy.

So, IMHO, it makes sense to try to find a solution for the above race condition.

…finalizer-info Automatic merge from submit-queue. Update deletionTimestamp with information about finalizer effect. Updating description of `deletionTimestamp` as adviced [here](#1174 (comment)).

pospispa · 2017-10-13T13:01:37Z

I addressed some of the comments in the update.
I added an alternative solution: "Scheduler In the Role of PVC Deletion Controller". I'd like to get feedback if the alternative has less disadvantages than the proposed implementation.

msau42 · 2017-10-16T22:22:30Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+
+In case a pod is being scheduled and is using PVCs that do not have condition PVCUsedByPod set it will set this condition for these PVCs.
+
+In case a pod is terminated and was using PVCs the scheduler will update PVCUsedByPod condition for these PVCs accordingly.


The scheduler does not handle pod termination today.

So it will mean extending scheduler to handle pod termination. I assume it means that it is a no-go alternative.

msau42 · 2017-10-16T22:25:46Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+
+Note: scheduler is the source of truth of pods that are being started. The information on active pods may be a little bit outdated that causes that deletion of a PVC may be postponed (pod status in schedular is active while the pod is terminated in API server), but this does not cause any harm.
+
+The disadvantage is that scheduler will become responsible for PVC deletion postponing that will make scheduler bigger.


Yeah I don't think it's in the scheduler's scope to handle PVC deletion.

I agree. So this is a no-go alternative.

msau42 · 2017-10-16T22:31:43Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+2. Pod A is deleted.
+3. Soon after, Pod B is created and scheduled that uses PVC.
+4. PVC is deleted.
+5. PVC deletion controller gets Pod delete for Pod A, sees no other Pod is using the PVC in its cache, and removes finalizer.


Could the race be solved if instead of removing the finalizer immediately, you resync all the Pods first?

Having PVC deletion controller resyncs all the pods before removing the finalizer is not enough. There must be a PVC resync done in scheduler or in kubelet. IMHO, these two resyncs will solve the race. I'll update the proposal accordingly.

There must be a PVC resync done in scheduler or in kubelet. IMHO, these two resyncs will solve the race. I'll update the proposal accordingly.

Not sure what you mean by resync in the kubelet... the kubelet does a live lookup of the PVC when setting up the pod's volume. If the PVC has a deletionTimestamp, the kubelet should not continue setting up the volume.

Not sure what you mean by resync in the kubelet... the kubelet does a live lookup of the PVC when setting up the pod's volume. If the PVC has a deletionTimestamp, the kubelet should not continue setting up the volume.

I didn't know that kubelet does a live lookup of the PVC(s) used by the pod so I wanted to add it.

If the PVC has a deletionTimestamp kubelet must not start the pod.

taherv · 2017-10-19T07:09:47Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+
+## Implementation
+
+### API Sever, PVC Admission Controller, PVC Create


Typo : Sever -> Server

Thanks for spotting it.

pospispa · 2017-10-20T16:57:56Z

@childsb @msau42 I believe I've addressed all essential comments including the race. Please, would you review the design proposal once again and let me know what has to be done to have the design proposal merged?

ianchakeres · 2017-10-22T14:52:08Z

Should we consider using the same process for postponing deletion of PVs that are actively used by pods?

ianchakeres · 2017-10-22T15:09:32Z

Should we consider having the PVC Deletion Controller add a finalizer to pods that have PV/PVCs? These finalizers would be used to indicate to the controller that the related PV/PVCs are in use by the related pod. The controller would then be called on pod deletion, and the finalizer would be removed.

liggitt · 2017-10-22T15:09:42Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+### Kubelet
+Kubelet does currently live lookup of PVC(s) that are used by a pod.
+
+In case any of the PVC(s) used by the pod has the `deletionTimestamp` set kubelet won't start the pod but will report and error: "can't start pod (%pod) because it's using PVC (%pvcName) that is being deleted"


Can log a specific message, but should follow the same code path as if the PVC was not found (fails in volume mount creation stage, retries N times, etc)

liggitt · 2017-10-22T15:11:35Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+
+In case any of the PVC(s) used by the pod has the `deletionTimestamp` set kubelet won't start the pod but will report and error: "can't start pod (%pod) because it's using PVC (%pvcName) that is being deleted"
+
+Note: as soon as all the PVC(s) with `deletionTimestamp` set are deleted (in case none of the PVC(s) don't have finalizer information set) the pod that now uses non-existent PVC(s) will be started and will be in pending state (waiting for the non-existent PVC(s) to be created). Otherwise, in case at least one PVC has finalizer information set, i.e. PVC deletion controller is waiting for the scheduled pod to be terminated, there will be a deadlock: kubelet waiting for PVC deletion while PVC deletion controller waiting for pod termination. The deadlock can be resolved only by deleting the scheduled pod.


Kubelets do not retry volume mounts forever. When it fails, the pod should no longer block deletion of the PVC

liggitt · 2017-10-22T15:33:17Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+### PVC Deletion Controller
+PVC deletion controller is a new internal controller.
+
+PVC deletion controller can list pods and watch for pods. It caches status of pods that use a PVC. It also watches for PVC deletes.


Typically, the pattern used here is informers on both resources feeding a single workqueue. In this case, the controller should watch both pods and PVCs:

For PVC add/update/delete events:

if deletionTimestamp is nil and finalizer is missing, add to queue

if deletionTimestamp is non-nil and finalizer is present and first, add to queue

For pod add events:

if pod is terminated, add all referenced PVCs to queue
For pod update events:

if pod is changing from non-terminated to terminated, add all referenced PVCs to queue
For pod delete events:

add all referenced PVCs to queue

When processing the PVC queue:

If PVC is not found in cache, skip

If PVC is in cache with nil deletionTimestamp and missing finalizer, attempt to add finalizer, requeue on error

If PVC is in cache with non-nil deletionTimestamp and present finalizer at first position, do live pod list in namespace, and if all pods referencing the PVC are not yet bound to a node or are terminated, attempt to remove the finalizer, requeue on error

That removes the need to track PVCs in use, removes the need for special logic at startup, unifies retry cases in case of conflicts on PVC updates (or other errors)

See https://github.com/kubernetes/community/blob/master/contributors/devel/controllers.md for more details about the queuing approach, or https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/serviceaccount/serviceaccounts_controller.go for an example of a controller feeding a single workqueue from informers on two separate resources

liggitt · 2017-10-22T15:44:06Z

Should we consider having the PVC Deletion Controller add a finalizer to pods that have PV/PVCs?

The PVC is the object we want to prevent getting deleted prematurely, so that is where the finalizer should go, not on the pod.

liggitt · 2017-10-22T15:53:01Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+A new plugin for PVC admission controller will be created. The plugin will automatically add finalizer information into newly created PVC's metadata.
+
+### Scheduler
+Scheduler will check if a pod uses a PVC and if any of the PVCs has `deletionTimestamp` set. In case this is true the scheduling of the pod will end up with an error: "PVC (%pvcName) is in scheduled for deletion state".


this should just involve an additional check for a non-nil deletionTimestamp on the PVC, and following the same code path as if the PVC was not found

ianchakeres · 2017-10-22T16:02:32Z

@liggitt - Just clarifying... My comment about adding a finalizer on a pod would be in addition to a finalizer on the PVC.

liggitt · 2017-10-22T16:06:52Z

@liggitt - Just clarifying... My comment about adding a finalizer on a pod would be in addition to a finalizer on the PVC.

The additional finalizer doesn't get you anything. We don't actually care about preventing deletion of pods using PVCs, so there's no reason to block it with an additional finalizer on the Pod object.

ianchakeres · 2017-10-22T16:49:05Z

A pod PVC delete finalizer could be used to handle a pod with a PVC/PV being deleted, instead of watching all pods for deletion. This finalizer could reduce the number of events that need to be handled by the controller, to only those pod events that could be related to PVC deletion.

liggitt · 2017-10-22T17:00:32Z

A pod PVC delete finalizer could be used to handle a pod with a PVC/PV being deleted, instead of watching all pods for deletion.

A finalizing controller for pods referencing PVCs would still have to watch all pods (there's no filter to say "just watch pods containing finalizer X")

This finalizer could reduce the number of events that need to be handled by the controller, to only those pod events that could be related to PVC deletion.

As described above, that can be determined by inspection of the pod spec on deletion/termination, without blocking the pod delete, requiring an additional update to remove the finalizer from the pod, and increasing latency of pod cleanup.

The finalizer on the PVC is what we want.

pospispa · 2017-10-24T12:55:31Z

@ianchakeres

Should we consider using the same process for postponing deletion of PVs that are actively used by pods?

I prefer to approach the PVC and PV deletion problems per partes
I prefer to firstly design and implement solution only for PVC deletion because I consider the PVC deletion a security bug and because it affects users. The PV deletion bug affects only admins and admins should avoid deleting bound PVs.

However, I expect that the solution for PV deletion problem will capitalize on experience with solution of the PVC deletion bug and will be probably similar to PVC deletion bug solution.

Should we consider having the PVC Deletion Controller add a finalizer to pods that have PV/PVCs?

In case a finalizer is in an object it means the the object deletion is postponed until the finalizer is removed from the object by a controller.

I agree with @liggitt that postponing pod deletion (adding a finalizer to a pod that have PV/PVC) doesn't bring any benefit as the pod can be deleted immediately and the PVC deletion controller will be informed about the pod deletion anyway.

pospispa · 2017-10-25T14:45:56Z

@liggitt thank you very much for your comments and proposals. I basically updated the design according to your proposals.

jsafrane · 2017-10-25T15:03:18Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+PVC deletion controller watches for both PVC and pod events that are processed as described below:
+1. PVC add/update/delete events:
+  - If `deletionTimestamp` is `nil` and finalizer is missing, the PVC is added to PVC queue.
+  - If `deletionTimestamp` is `non-nil` and finalizer is present and first, the PVC is added to PVC queue.


Why the finalizer position matters?

you're right, I was thinking finalizers were handled in order, but that is not the case. @pospispa you can strike " and first"

liggitt · 2017-10-25T19:42:21Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+In case any of the PVC(s) used by the pod has the `deletionTimestamp` set kubelet won't start the pod but will report and error: "can't start pod (%pod) because it's using PVC (%pvcName) that is being deleted". Kubelet will follow the same code path as if PVC(s) do not exist.
+
+### PVC Deletion Controller
+PVC deletion controller is a new internal controller.


s/deletion/finalizing/ might make the intent clearer

liggitt · 2017-10-25T20:03:06Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+4. Pod delete events:
+  - All referenced PVCs are added to PVC queue.
+
+PVC and pod information are kept in a cache.


this is done inherently when using an informer (the cache should be the one feeding the informer)

liggitt · 2017-10-25T20:03:58Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+The PVC queue holds PVCs that need to be processed according to the below rules:
+- If PVC is not found in cache, the PVC is skipped.
+- If PVC is in cache with `nil` `deletionTimestamp` and missing finalizer, finalizer is added to the PVC. In case the adding finalizer operation fails, the PVC is re-queued into the PVC queue.
+- If PVC is in cache with `non-nil` `deletionTimestamp` and present finalizer at first position, live pod list is done for the PVC namespace. If all pods referencing the PVC are not yet bound to a node or are terminated, the finalizer removal is attempted. In case the finalizer removal operation fails the PVC is re-queued.


strike " at first position", finalizers are processed in any order

pospispa · 2017-10-26T09:46:53Z

@liggitt and @jsafrane thank you for your comments. I updated the design proposal accordingly.

jsafrane · 2017-10-26T10:32:52Z

Looks good enough to me. @msau42 @childsb, WDYT?

msau42

lgtm with a nit

msau42 · 2017-10-26T14:59:48Z

contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md

+PVC finalizing controller won't watch pods because the information whether a PVC is used by a pod or not is now maintained by the scheduler.
+PVC finalizing controller will remove the finalizer infromation from a PVC in case the PVC has `deletionTimestamp` set and does not have the PVCUsedByPod condition set to true.
+
+In this case the known race condition discussed in the implementation section won't happen on two different objects (PVC and pod), but on the same object, i.e. PVC. Therefore, the probability of the race condition will be smaller.


This race is no longer discussed :-)

Removed the information about the race.

pospispa · 2017-10-26T18:07:00Z

Should I already squash all the commits?

msau42 · 2017-10-26T18:09:26Z

yes thanks

…a Pod Proposal for postponing deletion of Persistent Volume Claim in case it's used by a pod. It will fix issue kubernetes/kubernetes#45143

pospispa · 2017-10-26T18:13:07Z

Done.

msau42 · 2017-10-26T18:48:38Z

/lgtm

k8s-github-robot · 2017-10-26T19:30:02Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-10-26T19:33:08Z

Automatic merge from submit-queue.

k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. kind/design Categorizes issue or PR as related to design. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 11, 2017

k8s-github-robot assigned childsb and msau42 Oct 11, 2017

pospispa mentioned this pull request Oct 11, 2017

should prevent the deletion of a PVC that is referenced by an active pod kubernetes/kubernetes#45143

Closed

msau42 reviewed Oct 11, 2017

View reviewed changes

pospispa mentioned this pull request Oct 12, 2017

Update deletionTimestamp with information about finalizer effect. #1177

Merged

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 13, 2017

msau42 reviewed Oct 16, 2017

View reviewed changes

taherv reviewed Oct 19, 2017

View reviewed changes

liggitt reviewed Oct 22, 2017

View reviewed changes

jsafrane reviewed Oct 25, 2017

View reviewed changes

liggitt reviewed Oct 25, 2017

View reviewed changes

msau42 reviewed Oct 26, 2017

View reviewed changes

pospispa mentioned this pull request Oct 26, 2017

StorageObjectInUseProtection (was Persistent Volume Claim Protection) kubernetes/enhancements#498

Closed

Postpone Deletion of a Persistent Volume Claim in case It Is Used by …

b378919

…a Pod Proposal for postponing deletion of Persistent Volume Claim in case it's used by a pod. It will fix issue kubernetes/kubernetes#45143

pospispa force-pushed the postpone-pvc-deletion-if-used-in-a-pod branch from 76d182c to b378919 Compare October 26, 2017 18:12

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 26, 2017

k8s-github-robot merged commit 6c96f9a into kubernetes:master Oct 26, 2017

saad-ali mentioned this pull request Jan 23, 2018

After device is attached, devicepath of FC and iSCSI plugin are empty kubernetes/kubernetes#54108

Closed

msau42 mentioned this pull request Sep 16, 2020

Fix race condition between CSI NodeUnstage and DeleteVolume calls kubernetes/kubernetes#94737

Closed

mkimuram mentioned this pull request May 13, 2021

Add KEP for Secret/ConfigMap protection kubernetes/enhancements#2640

Closed


		In case a pod is being scheduled and is using PVCs that do not have condition PVCUsedByPod set it will set this condition for these PVCs.

		In case a pod is terminated and was using PVCs the scheduler will update PVCUsedByPod condition for these PVCs accordingly.


		Note: scheduler is the source of truth of pods that are being started. The information on active pods may be a little bit outdated that causes that deletion of a PVC may be postponed (pod status in schedular is active while the pod is terminated in API server), but this does not cause any harm.

		The disadvantage is that scheduler will become responsible for PVC deletion postponing that will make scheduler bigger.


		## Implementation

		### API Sever, PVC Admission Controller, PVC Create


		In case any of the PVC(s) used by the pod has the `deletionTimestamp` set kubelet won't start the pod but will report and error: "can't start pod (%pod) because it's using PVC (%pvcName) that is being deleted"

		Note: as soon as all the PVC(s) with `deletionTimestamp` set are deleted (in case none of the PVC(s) don't have finalizer information set) the pod that now uses non-existent PVC(s) will be started and will be in pending state (waiting for the non-existent PVC(s) to be created). Otherwise, in case at least one PVC has finalizer information set, i.e. PVC deletion controller is waiting for the scheduled pod to be terminated, there will be a deadlock: kubelet waiting for PVC deletion while PVC deletion controller waiting for pod termination. The deadlock can be resolved only by deleting the scheduled pod.

Postpone Deletion of a Persistent Volume Claim in case It Is Used by a Pod #1174

Postpone Deletion of a Persistent Volume Claim in case It Is Used by a Pod #1174

Conversation

pospispa commented Oct 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Oct 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Oct 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pospispa commented Oct 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pospispa Oct 17, 2017 • edited Loading

Choose a reason for hiding this comment

liggitt Oct 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pospispa commented Oct 20, 2017

ianchakeres commented Oct 22, 2017

ianchakeres commented Oct 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Oct 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented Oct 22, 2017

Choose a reason for hiding this comment

ianchakeres commented Oct 22, 2017

liggitt commented Oct 22, 2017

ianchakeres commented Oct 22, 2017

liggitt commented Oct 22, 2017

pospispa commented Oct 24, 2017

pospispa commented Oct 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pospispa commented Oct 26, 2017

jsafrane commented Oct 26, 2017

msau42 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pospispa commented Oct 26, 2017

msau42 commented Oct 26, 2017

pospispa commented Oct 26, 2017

msau42 commented Oct 26, 2017

k8s-github-robot commented Oct 26, 2017

k8s-github-robot commented Oct 26, 2017

liggitt Oct 12, 2017 •

edited

Loading

liggitt Oct 11, 2017 •

edited

Loading

pospispa Oct 17, 2017 •

edited

Loading

liggitt Oct 17, 2017 •

edited

Loading

liggitt Oct 22, 2017 •

edited

Loading