metricbeat kubernetes.yml does not work with kubectl apply

[jordansissel]

When deploying metricbeat-kubernetes.yml with kubectl apply -f ... it deploys without error, but about one minute later, all resources defined by this yaml file disappear from kubernetes:

PS D:\projects\seceng\kubernetes\metricbeat> kubectl apply -f metricbeat.yml                                    
configmap "metricbeat-config" created                                                                          
configmap "metricbeat-daemonset-modules" created                                                                
daemonset.extensions "metricbeat" created                                                                      
configmap "metricbeat-deployment-modules" created                                                              
deployment.apps "metricbeat" created                                                                            
clusterrolebinding.rbac.authorization.k8s.io "metricbeat" configured                                            
clusterrole.rbac.authorization.k8s.io "metricbeat" configured                                                  
serviceaccount "metricbeat" unchanged                                                                          
PS D:\projects\seceng\kubernetes\metricbeat> kubectl get ds -n kube-system metricbeat                          
NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE                        
metricbeat   6         6         6         6            6           <none>          14s                        
PS D:\projects\seceng\kubernetes\metricbeat> kubectl get configmap -n kube-system metricbeat-config            
NAME                DATA      AGE                                                                              
metricbeat-config   1         24s                                                                              
PS D:\projects\seceng\kubernetes\metricbeat> date                                                              
Tue Jun  5 10:20:30 PDT 2018                                                                                    
PS D:\projects\seceng\kubernetes\metricbeat> date                                                              
Tue Jun  5 10:20:53 PDT 2018                                                                                    
PS D:\projects\seceng\kubernetes\metricbeat> kubectl get configmap -n kube-system metricbeat-config            
Error from server (NotFound): configmaps "metricbeat-config" not found                                          
PS D:\projects\seceng\kubernetes\metricbeat> kubectl get ds -n kube-system metricbeat                          
Error from server (NotFound): daemonsets.extensions "metricbeat" not found  

Some discussion on Slack recommended trying with kubectl create instead of kubectl apply, which appears to work.

Additionally, a second recommendation was to comment out all kubernetes.io/cluster-service: "true" lines. This recommendation appears to work for kubectl apply.

> kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.6-gke.1", GitCommit:"cb151369f60073317da686a6ce7de36abe2bda8d", GitTreeState:"clean", BuildDate:"2018-04-07T22:06:59Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Tested against Google Kubernetes Engine.

[exekias]

kubectl create is what we recommend in the docs, but I agree with removing kubernetes.io/cluster-service: "true" to avoid messing with addon manager by default