camel-k-cache crashes on minikube #736

mayurans · 2019-06-10T14:47:10Z

On minikube environment, camel-k-cache crashes with following error when "kamel install" is run:

$ kubectl logs camel-k-cache
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xf13712]

goroutine 1 [running]:
github.com/GoogleContainerTools/kaniko/pkg/cache.WarmCache(0x1977fe0, 0xf25480, 0xc42034fcd8)
	/go/src/github.com/GoogleContainerTools/kaniko/pkg/cache/warm.go:47 +0x372
github.com/GoogleContainerTools/kaniko/cmd/warmer/cmd.glob..func2(0x196f140, 0xc4202d5400, 0x0, 0x2)
	/go/src/github.com/GoogleContainerTools/kaniko/cmd/warmer/cmd/root.go:54 +0x2d
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).execute(0x196f140, 0xc4200380a0, 0x2, 0x2, 0x196f140, 0xc4200380a0)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x196f140, 0x0, 0x4, 0x116cca9)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).Execute(0x196f140, 0xc42034ff78, 0xc420096058)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
	/go/src/github.com/GoogleContainerTools/kaniko/cmd/warmer/main.go:26 +0x2d

When it craches, the following errors are thrown by camel-k-operator:

{"level":"error","ts":1560176689.3713505,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"integrationplatform-controller","request":"default/camel-k","error":"failed to warm up Kaniko cache","stacktrace":"github.com/apache/camel-k/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/nferraro/gopaths/camel-k/src/github.com/apache/camel-k/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/apache/camel-k/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/nferraro/gopaths/camel-k/src/github.com/apache/camel-k/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/apache/camel-k/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/nferraro/gopaths/camel-k/src/github.com/apache/camel-k/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/apache/camel-k/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/nferraro/gopaths/camel-k/src/github.com/apache/camel-k/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/apache/camel-k/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/nferraro/gopaths/camel-k/src/github.com/apache/camel-k/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/apache/camel-k/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/nferraro/gopaths/camel-k/src/github.com/apache/camel-k/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

EXPAND - kubectl describe pod camel-k-cache

$ kubectl  describe pod camel-k-cache 
Name:               camel-k-cache
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               minikube/192.168.122.123
Start Time:         Mon, 10 Jun 2019 14:24:04 +0000
Labels:             <none>
Annotations:        <none>
Status:             Failed
IP:                 172.17.0.6
Init Containers:
  create-kaniko-cache:
    Container ID:  docker://6dcaf166e78abb6afe232958b6646eedc6bb0cf0c7f29db5b61cef87d54acba3
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:4b6ad3a68d34da29bf7c8ccb5d355ba8b4babcad1f99798204e7abb43e54ee3d
    Port:          <none>
    Host Port:     <none>
    Command:
      mkdir
      -p
      /workspace/cache
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 10 Jun 2019 14:24:11 +0000
      Finished:     Mon, 10 Jun 2019 14:24:11 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5ctcz (ro)
      /workspace from camel-k-builder (rw)
Containers:
  warm-kaniko-cache:
    Container ID:  docker://9690d359ff45ab78ae5cf62dd3b2a5175245294b8baeb15609139f82114bba96
    Image:         gcr.io/kaniko-project/warmer:v0.9.0
    Image ID:      docker-pullable://gcr.io/kaniko-project/warmer@sha256:e146228a6b965e53d34abed82512f455791ec44ad305022a26c4adb632a61f00
    Port:          <none>
    Host Port:     <none>
    Args:
      --cache-dir=/workspace/cache
      --image=fabric8/s2i-java:3.0-java8
    State:          Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 10 Jun 2019 14:24:14 +0000
      Finished:     Mon, 10 Jun 2019 14:24:45 +0000
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5ctcz (ro)
      /workspace from camel-k-builder (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  camel-k-builder:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  camel-k
    ReadOnly:   false
  default-token-5ctcz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5ctcz
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age              From               Message
  ----     ------            ----             ----               -------
  Warning  FailedScheduling  7m (x5 over 7m)  default-scheduler  pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled         7m               default-scheduler  Successfully assigned default/camel-k-cache to minikube
  Normal   Pulled            7m               kubelet, minikube  Container image "busybox" already present on machine
  Normal   Created           7m               kubelet, minikube  Created container
  Normal   Started           7m               kubelet, minikube  Started container
  Normal   Pulled            7m               kubelet, minikube  Container image "gcr.io/kaniko-project/warmer:v0.9.0" already present on machine
  Normal   Created           7m               kubelet, minikube  Created container
  Normal   Started           7m               kubelet, minikube  Started container

Pods list:

$ kubectl  get pods --all-namespaces
NAMESPACE     NAME                                READY     STATUS    RESTARTS   AGE
default       camel-k-cache                       0/1       Error     0          21m
default       camel-k-operator-676999f5bc-9fpx7   1/1       Running   0          22m
kube-system   coredns-576cbf47c7-dmdbk            1/1       Running   3          2d18h
kube-system   coredns-576cbf47c7-ttdhv            1/1       Running   3          2d18h
kube-system   etcd-minikube                       1/1       Running   4          2d18h
kube-system   kube-addon-manager-minikube         1/1       Running   2          2d18h
kube-system   kube-apiserver-minikube             1/1       Running   3          2d18h
kube-system   kube-controller-manager-minikube    1/1       Running   3          2d18h
kube-system   kube-proxy-wj8g8                    1/1       Running   3          2d18h
kube-system   kube-scheduler-minikube             1/1       Running   3          2d18h
kube-system   registry-6kslv                      1/1       Running   1          2d18h
kube-system   storage-provisioner                 1/1       Running   5          2d18

Here are steps to reproduce:

Install MiniKubue
Run kamel install

The text was updated successfully, but these errors were encountered:

WillemJiang · 2019-06-10T22:46:24Z

@mayurans Could you specify the camel-k version that you use and the memory,storage setting of your minikube?

mayurans · 2019-06-11T17:57:56Z

camel-k (kamel) version: 0.3.3.
Minikube instance was deployed with the following configurations:
RAM: 8GB
CPU: 4

mayurans · 2019-06-11T20:51:44Z

I've re-deployed minikube and tried latest kamel binary (0.3.4).
But, I'm still getting the same runtime error in camel-k-cache.

minikube config:

$ minikube version
minikube version: v1.1.1

$ cat ~/.minikube/config/config.json 
{
    "cpus": 4,
    "disk-size": "50g",
    "kubernetes-version": "v1.12.0",
    "memory": 8192,
    "profile": "camelk",
    "registry": true,
    "vm-driver": "kvm2"
}

$minikube start -p camelk --extra-config=apiserver.enable-admission-plugins="LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook"

Kamel config:

$ kamel version
Camel K Client 0.3.4

$kamel install


$ kubectl get pods -w
NAME                               READY     STATUS    RESTARTS   AGE
camel-k-operator-66b855c5f-g4h6j   0/1       Pending   0          1s
camel-k-operator-66b855c5f-g4h6j   0/1       Pending   0         1s
camel-k-operator-66b855c5f-g4h6j   0/1       ContainerCreating   0         1s
camel-k-cache   0/1       Pending   0         0s
camel-k-cache   0/1       Pending   0         0s
camel-k-cache   0/1       Pending   0         0s
camel-k-cache   0/1       Init:0/1   0         0s
camel-k-operator-66b855c5f-g4h6j   1/1       Running   0         45s
camel-k-cache   0/1       PodInitializing   0         3s
camel-k-cache   1/1       Running   0         13s
camel-k-cache   0/1       Error     0         44s

camel-k-cache log:

$ kubectl logs camel-k-cache
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xf13712]

goroutine 1 [running]:
github.com/GoogleContainerTools/kaniko/pkg/cache.WarmCache(0x1977fe0, 0xf25480, 0xc42036dcd8)
	/go/src/github.com/GoogleContainerTools/kaniko/pkg/cache/warm.go:47 +0x372
github.com/GoogleContainerTools/kaniko/cmd/warmer/cmd.glob..func2(0x196f140, 0xc4202e7380, 0x0, 0x2)
	/go/src/github.com/GoogleContainerTools/kaniko/cmd/warmer/cmd/root.go:54 +0x2d
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).execute(0x196f140, 0xc4200a4070, 0x2, 0x2, 0x196f140, 0xc4200a4070)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x196f140, 0x0, 0x4, 0x116cca9)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).Execute(0x196f140, 0xc42036df78, 0xc42009e058)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
	/go/src/github.com/GoogleContainerTools/kaniko/cmd/warmer/main.go:26 +0x2d

docker images in minikube:

$ minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
busybox                                   latest              ef46e0caa533        22 hours ago        1.21MB
apache/camel-k                            0.3.4               4e98c80fd813        4 days ago          524MB
gcr.io/kaniko-project/warmer              v0.9.0              b454d1cbac22        4 months ago        22.8MB
k8s.gcr.io/kube-addon-manager             v9.0                119701e77cbc        4 months ago        83.1MB
k8s.gcr.io/kubernetes-dashboard-amd64     v1.10.1             f9aed6605b81        5 months ago        122MB
k8s.gcr.io/kube-apiserver                 v1.12.0             ab60b017e34f        8 months ago        194MB
k8s.gcr.io/kube-controller-manager        v1.12.0             07e068033cf2        8 months ago        164MB
k8s.gcr.io/kube-proxy                     v1.12.0             9c3a9d3f09a0        8 months ago        96.6MB
k8s.gcr.io/kube-scheduler                 v1.12.0             5a1527e735da        8 months ago        58.3MB
k8s.gcr.io/etcd                           3.2.24              3cab8e1b9802        8 months ago        220MB
k8s.gcr.io/coredns                        1.2.2               367cdc8433a4        9 months ago        39.2MB
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64    1.14.8              c2ce1ffb51ed        17 months ago       41MB
k8s.gcr.io/k8s-dns-sidecar-amd64          1.14.8              6f7f2dc7fab5        17 months ago       42.2MB
k8s.gcr.io/k8s-dns-kube-dns-amd64         1.14.8              80cc5ea4b547        17 months ago       50.5MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca1        17 months ago       742kB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        19 months ago       80.8MB

rejimathews · 2019-06-13T00:37:51Z

@mayurans Guess it worked for me on macOS. "kamel install" ran successfully.
However, I experienced some issues starting up camel apps. I will post details shortly

Check out these screenshots

jamesnetherton · 2019-06-13T06:46:16Z

It seems to be an intermittent problem (for me at least). Sometimes when starting a new cluster kamel install works fine and other times it fails warming the Kaniko cache.

The error given for the camel-k-cache pod is pod has unbound immediate PersistentVolumeClaims. Which is odd given that the init container successfully mounted the same PV and wrote to it.

WillemJiang · 2019-07-18T05:55:59Z

I hit this kind of issue today, with minikube -v 1.2.0 and kamel 0.3.4. Not sure how to resolve it.

jamesnetherton · 2019-07-23T09:30:01Z

The lazy fix I've been using locally is to set restart policy OnFailure here.

Pod completes successfully after 1 or 2 restarts.

fixes apache#736 fixes apache#898

mayurbm · 2019-09-20T09:32:55Z

fix: Use restart policy OnFailure for Kaniko cache warmer doesn't solve the problem in minikube.
I still get CrashLoopBackOff
camel-k-cache 0/1 CrashLoopBackOff 8 20m

in pod log I can see:

vagrant@ubukube:~/camel/camel-k/build/maven$ kubectl logs camel-k-cache
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xf13712]

goroutine 1 [running]:
github.com/GoogleContainerTools/kaniko/pkg/cache.WarmCache(0x1977fe0, 0xf25480, 0xc420331cd8)
/go/src/github.com/GoogleContainerTools/kaniko/pkg/cache/warm.go:47 +0x372
github.com/GoogleContainerTools/kaniko/cmd/warmer/cmd.glob..func2(0x196f140, 0xc4203ca240, 0x0, 0x2)
/go/src/github.com/GoogleContainerTools/kaniko/cmd/warmer/cmd/root.go:54 +0x2d
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).execute(0x196f140, 0xc4200380a0, 0x2, 0x2, 0x196f140, 0xc4200380a0)
/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x196f140, 0x0, 0x4, 0x116cca9)
/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).Execute(0x196f140, 0xc420331f78, 0xc42008a058)
/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
/go/src/github.com/GoogleContainerTools/kaniko/cmd/warmer/main.go:26 +0x2d

fixes apache#736 fixes apache#898

nicolaferraro added this to the 1.0.0 milestone Jul 22, 2019

nicolaferraro added kind/bug Something isn't working area/build-operator Related to the internal image build operator labels Jul 22, 2019

jamesnetherton mentioned this issue Aug 6, 2019

Camel-k-cache not able to create PVC (pod has unbound immediate PersistentVolumeClaims) #898

Closed

jamesnetherton added a commit to jamesnetherton/camel-k that referenced this issue Aug 13, 2019

fix: Use restart policy OnFailure for Kaniko cache warmer

15faee1

fixes apache#736 fixes apache#898

astefanutti closed this as completed in 47e60f5 Aug 13, 2019

jamesnetherton mentioned this issue Aug 13, 2019

Add option to disable Kaniko cache warming #907

Closed

asifdxtreme pushed a commit to asifdxtreme/camel-k that referenced this issue Sep 30, 2019

fix: Use restart policy OnFailure for Kaniko cache warmer

e18069b

fixes apache#736 fixes apache#898

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

camel-k-cache crashes on minikube #736

camel-k-cache crashes on minikube #736

mayurans commented Jun 10, 2019

WillemJiang commented Jun 10, 2019

mayurans commented Jun 11, 2019

mayurans commented Jun 11, 2019

rejimathews commented Jun 13, 2019 •

edited

Loading

jamesnetherton commented Jun 13, 2019

WillemJiang commented Jul 18, 2019

jamesnetherton commented Jul 23, 2019

mayurbm commented Sep 20, 2019

camel-k-cache crashes on minikube #736

camel-k-cache crashes on minikube #736

Comments

mayurans commented Jun 10, 2019

WillemJiang commented Jun 10, 2019

mayurans commented Jun 11, 2019

mayurans commented Jun 11, 2019

rejimathews commented Jun 13, 2019 • edited Loading

jamesnetherton commented Jun 13, 2019

WillemJiang commented Jul 18, 2019

jamesnetherton commented Jul 23, 2019

mayurbm commented Sep 20, 2019

rejimathews commented Jun 13, 2019 •

edited

Loading