Fix flaky unit tests

[johnugeorge]

/kind bug

Currently, unit tests are flaky with frequent errors during its run.

--- FAIL: TestReconcileExperiment (6.29s)
   experiment_controller_suite_test.go:76:
       Expected error:
           <*errors.errorString | 0xc00053a850>: {
               s: "an empty namespace may not be set during creation",
           }
           an empty namespace may not be set during creation
       not to have occurred
   experiment_controller_test.go:268:
       Timed out after 5.000s.
       Expected
           <bool>: false
       to be true

[gaocegege]

Can you reproduce it? I think we have CI to run tests for every PR. Thus unit test should be passed.

I tried locally and does not meet any error.

[johnugeorge]

This happens much more frequently now

`2019-06-16T11:32:15.530+0530 ERROR kubebuilder.controller controller/controller.go:144 Could not wait for Cache to sync {"Controller": "experiment-controller", "error": "failed to wait for experiment-controller caches to sync"}
github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:144
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).start.func2
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:206
2019-06-16T11:32:15.543+0530 INFO kubebuilder.controller controller/controller.go:120 Starting EventSource {"Controller": "experiment-controller", "Source": "kind source: /, Kind="}
2019-06-16T11:32:15.543+0530 INFO kubebuilder.controller controller/controller.go:120 Starting EventSource {"Controller": "experiment-controller", "Source": "kind source: /, Kind="}
2019-06-16T11:32:15.544+0530 INFO experiment-controller experiment/experiment_controller.go:131 Experiment controller created
2019-06-16T11:32:15.644+0530 INFO kubebuilder.webhook webhook/server.go:200 installing webhook configuration in cluster
2019-06-16T11:32:15.644+0530 INFO kubebuilder.controller controller/controller.go:134 Starting Controller {"Controller": "experiment-controller"}
2019-06-16T11:32:15.745+0530 INFO kubebuilder.controller controller/controller.go:153 Starting workers {"Controller": "experiment-controller", "WorkerCount": 1}
2019-06-16T11:32:15.755+0530 DEBUG kubebuilder.controller controller/controller.go:225 Successfully Reconciled {"Controller": "experiment-controller", "Request": "default/tn"}
2019-06-16T11:32:15.760+0530 ERROR experiment-controller experiment/experiment_controller.go:266 Update experiment instance status error {"Experiment": "default/tn", "error": "Operation cannot be fulfilled on experiments.kubeflow.org "tn": the object has been modified; please apply your changes to the latest version and try again"}
github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128
github.com/kubeflow/katib/pkg/controller/v1alpha2/experiment.(*ReconcileExperiment).Reconcile
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/pkg/controller/v1alpha2/experiment/experiment_controller.go:266
github.com/kubeflow/katib/pkg/controller/v1alpha2/experiment.SetupTestReconcile.func1
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/pkg/controller/v1alpha2/experiment/experiment_controller_suite_test.go:67
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/reconcile/reconcile.go:92
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157
github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.Until
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
2019-06-16T11:32:15.760+0530 ERROR kubebuilder.controller controller/controller.go:209 Reconciler error {"Controller": "experiment-controller", "Request": "default/tn", "error": "Operation cannot be fulfilled on experiments.kubeflow.org "tn": the object has been modified; please apply your changes to the latest version and try again"}
github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157
github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.Until
/Users/johnugeo/GO-ROOT/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
2019-06-16T11:32:16.762+0530 INFO experiment-controller experiment/experiment_controller.go:338 CreateTrials {"Experiment": "default/tn", "addCount": 1}
2019-06-16T11:32:16.765+0530 DEBUG kubebuilder.controller controller/controller.go:225 Successfully Reconciled {"Controller": "experiment-controller", "Request": "default/tn"}
2019-06-16T11:32:16.766+0530 INFO experiment-status-util util/status_util.go:72 Objective metric name not found {"trial": "tn-42ccsftc"}
2019-06-16T11:32:16.774+0530 DEBUG kubebuilder.controller controller/controller.go:225 Successfully Reconciled {"Controller": "experiment-controller", "Request": "default/tn"}
2019-06-16T11:32:16.774+0530 INFO experiment-status-util util/status_util.go:72 Objective metric name not found {"trial": "tn-42ccsftc"}
2019-06-16T11:32:16.774+0530 DEBUG kubebuilder.controller controller/controller.go:225 Successfully Reconciled {"Controller": "experiment-controller", "Request": "default/tn"}
2019-06-16T11:32:16.785+0530 DEBUG kubebuilder.controller controller/controller.go:225 Successfully Reconciled {"Controller": "experiment-controller", "Request": "default/tn"}
2019-06-16T11:32:16.790+0530 DEBUG kubebuilder.controller controller/controller.go:225 Successfully Reconciled {"Controller": "experiment-controller", "Request": "default/tn"}

--- FAIL: TestReconcileExperiment (1.26s)
experiment_controller_suite_test.go:80:
Expected error:
<*errors.errorString | 0xc4203d4890>: {
s: "an empty namespace may not be set during creation",
}
an empty namespace may not be set during creation
not to have occurred
FAIL
`

@gaocegege

[johnugeorge]

Related: #652

[gaocegege]

/assign

[gaocegege]

/close

It was fixed.

[k8s-ci-robot]

@gaocegege: Closing this issue.

In response to this:

/close

It was fixed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.