Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wait for webhook key to be present in filesystem #2312

Merged
merged 2 commits into from
Dec 20, 2019

Conversation

anyasabo
Copy link
Contributor

@thbkrkr brought up out of band that he intermittently received this error starting the operator

{"level":"error","@timestamp":"2019-12-19T15:35:28.548Z","logger":"manager","message":"unable to run the manager","ver":"1.0.0-rc3-ae5f8e81","error":"open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/elastic/cloud-on-k8s/cmd/manager.execute
	/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:331
github.com/elastic/cloud-on-k8s/cmd/manager.glob..func1
	/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:81
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
	/go/src/github.com/elastic/cloud-on-k8s/cmd/main.go:27
runtime.main
	/usr/local/go/src/runtime/proc.go:203"}

I suspect that this is because we return setupWebhook() as soon as we update the secret with the generated key, but it may take more time for the kubelet to update the container's file system with the new contents of the secret. I'm not sure there's a simple way to write a test for this, but it worked in my testing. This is more of a proof of concept of checking for it, definitely open to alternate approaches if this is something we want to do.

@anyasabo anyasabo added the >bug Something isn't working label Dec 20, 2019
cmd/manager/main.go Outdated Show resolved Hide resolved
cmd/manager/main.go Show resolved Hide resolved
Copy link
Contributor

@thbkrkr thbkrkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

We have been dragging this issue since v0.8. Nice to see it disappear now.

@anyasabo anyasabo merged commit 2f3f0a0 into elastic:master Dec 20, 2019
@anyasabo anyasabo deleted the waitforkey branch December 20, 2019 23:52
@barkbay barkbay added the v1.0.0 label Dec 31, 2019
@anyasabo anyasabo changed the title Wait for webhook key to be present in fs Wait for webhook key to be present in filesystem Jan 9, 2020
mjmbischoff pushed a commit to mjmbischoff/cloud-on-k8s that referenced this pull request Jan 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug Something isn't working v1.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants