helm operator panics whilst syncing charts

[subnova]

Kubernetes: 1.10.3 (EKS)
Image: quay.io/weaveworks/helm-operator:0.2.1 (also in 0.2.0)

Stack trace:

ts=2018-09-20T11:13:07.925842639Z caller=chartsync.go:106 component=chartsync info="Starting charts sync loop"
ts=2018-09-20T11:13:07.925976309Z caller=operator.go:103 component=operator info="Setting up event handlers"
ts=2018-09-20T11:13:07.925995895Z caller=operator.go:125 component=operator info="Event handlers set up"
ts=2018-09-20T11:13:07.92601648Z caller=operator.go:138 component=operator info="Starting operator"
ts=2018-09-20T11:13:07.926026298Z caller=operator.go:140 component=operator info="Waiting for informer caches to sync"
ts=2018-09-20T11:13:07.932548261Z caller=operator.go:108 component=operator info="CREATING release"
ts=2018-09-20T11:13:07.932614178Z caller=operator.go:109 component=operator info="Custom Resource driven release install"
ts=2018-09-20T11:13:07.932690219Z caller=operator.go:108 component=operator info="CREATING release"
ts=2018-09-20T11:13:07.932706609Z caller=operator.go:109 component=operator info="Custom Resource driven release install"
ts=2018-09-20T11:13:08.026136395Z caller=operator.go:145 component=operator info="Informer caches synced"
ts=2018-09-20T11:13:08.026181139Z caller=operator.go:147 component=operator info="Starting workers"
ts=2018-09-20T11:13:08.026226569Z caller=operator.go:173 component=operator debug="Processing next work queue job ..."
ts=2018-09-20T11:13:08.026263318Z caller=operator.go:176 component=operator debug="PROCESSING item [\"test2/test-chart\"]"
ts=2018-09-20T11:13:08.026294224Z caller=operator.go:233 component=operator debug="Starting to sync cache key test2/test-chart"
ts=2018-09-20T11:13:08.026459058Z caller=operator.go:173 component=operator debug="Processing next work queue job ..."
ts=2018-09-20T11:13:08.026479186Z caller=operator.go:176 component=operator debug="PROCESSING item [\"demo/demo-gocd\"]"
ts=2018-09-20T11:13:08.026490212Z caller=operator.go:233 component=operator debug="Starting to sync cache key demo/demo-gocd"
E0920 11:13:08.051620       7 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:388
/go/src/github.com/weaveworks/flux/git/export.go:13
/go/src/github.com/weaveworks/flux/integrations/helm/chartsync/chartsync.go:190
/go/src/github.com/weaveworks/flux/integrations/helm/chartsync/chartsync.go:190
/go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:255
/go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:211
/go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:221
/go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:166
/go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:150
/go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xfd71f6]

goroutine 81 [running]:
github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x1128ee0, 0x1b56ec0)
        /usr/local/go/src/runtime/panic.go:502 +0x229
github.com/weaveworks/flux/git.(*Export).Dir(...)
        /go/src/github.com/weaveworks/flux/git/export.go:13
github.com/weaveworks/flux/integrations/helm/chartsync.(*ChartChangeSync).reconcileReleaseDef(0xc420325380, 0xc4203f14f0, 0xf, 0xc421579050, 0x2b, 0xc4203f1550, 0xa, 0x0, 0x0, 0xc4203f1569, ...)
        /go/src/github.com/weaveworks/flux/integrations/helm/chartsync/chartsync.go:253 +0x716
github.com/weaveworks/flux/integrations/helm/chartsync.(*ChartChangeSync).ReconcileReleaseDef(0xc420325380, 0xc4203f14f0, 0xf, 0xc421579050, 0x2b, 0xc4203f1550, 0xa, 0x0, 0x0, 0xc4203f1569, ...)
        /go/src/github.com/weaveworks/flux/integrations/helm/chartsync/chartsync.go:190 +0x5f
github.com/weaveworks/flux/integrations/helm/operator.(*Controller).syncHandler(0xc420328400, 0xc4203f1840, 0x10, 0x10c0320, 0xc42158cce0)
        /go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:255 +0x861
github.com/weaveworks/flux/integrations/helm/operator.(*Controller).processNextWorkItem.func1(0xc420328400, 0x10c0320, 0xc42158cce0, 0x0, 0x0)
        /go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:211 +0xda
github.com/weaveworks/flux/integrations/helm/operator.(*Controller).processNextWorkItem(0xc420328400, 0xc42046d8e0)
        /go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:221 +0x21e
github.com/weaveworks/flux/integrations/helm/operator.(*Controller).runWorker(0xc420328400)
        /go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:166 +0x2b
github.com/weaveworks/flux/integrations/helm/operator.(*Controller).(github.com/weaveworks/flux/integrations/helm/operator.runWorker)-fm()
        /go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:150 +0x2a
github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc420044250)
        /go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420044250, 0x3b9aca00, 0x0, 0x1b76a01, 0xc42008a1e0)
        /go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc420044250, 0x3b9aca00, 0xc42008a1e0)
        /go/src/github.com/weaveworks/flux/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/weaveworks/flux/integrations/helm/operator.(*Controller).Run
        /go/src/github.com/weaveworks/flux/integrations/helm/operator/operator.go:150 +0x332

[subnova]

I had helm-operator looking at the wrong branch in my repo, so neither the charts directory nor the chart will have been there.

[squaremo]

Thanks for this @subnova.

[squaremo]

Fix notes: It should certainly not panic; probably best to just log a warning*, and ignore from then on.
*Though not repeatedly, since it will be trying to look at the directory every minute or whatever.

[hiddeco]

Looks like this has already been taken care of in #1606 (stat error while checking for requirements.yaml in chart dir) and #1382 (err if path to chart could not be found).

ts=2019-02-25T13:11:22.84984835Z caller=chartsync.go:247 component=chartsync info="Start of releasesync"
ts=2019-02-25T13:11:22.908443585Z caller=chartsync.go:333 component=chartsync warning="Failed to update chart dependencies" namespace=default name=ghost error="stat /tmp/flux-working023605298/charts/ghost: no such file or directory"
ts=2019-02-25T13:11:22.910787782Z caller=chartsync.go:252 component=chartsync info="End of releasesync"

[hiddeco]

Superseded by #1764.