Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crashd overwrites the same resources from different API versions #182

Open
ncdc opened this issue Oct 2, 2020 · 5 comments
Open

crashd overwrites the same resources from different API versions #182

ncdc opened this issue Oct 2, 2020 · 5 comments

Comments

@ncdc
Copy link

ncdc commented Oct 2, 2020

For the following script

kc="{0}/.kube/config".format(os.home)
kube_capture(what="objects", kinds=["clusters"], kube_config=kube_config(path=kc))

With the following cluster:

$ k get clusters -A
NAMESPACE   NAME    PHASE
default     test1   Provisioning

Run this:

$ /crashd run --debug andy.crsh
time="2020-10-02T15:27:07-04:00" level=debug msg="creating working directory /tmp/crashd"
time="2020-10-02T15:27:07-04:00" level=debug msg="kube_capture(what=objects)"
time="2020-10-02T15:27:07-04:00" level=debug msg="Search filters groups:[]; kinds:[clusters]; namespaces:[]; versions:[]; names:[]; labels:[] containers:[]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching in 28 groups"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group core"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group apiregistration.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apiregistration.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apiregistration.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group extensions"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion extensions/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group apps"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apps/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group events.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion events.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion events.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group authentication.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion authentication.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion authentication.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group authorization.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion authorization.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion authorization.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group autoscaling"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion autoscaling/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion autoscaling/v2beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion autoscaling/v2beta2"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group batch"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion batch/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion batch/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group certificates.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion certificates.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion certificates.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group networking.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion networking.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion networking.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group policy"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion policy/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group rbac.authorization.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion rbac.authorization.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion rbac.authorization.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group storage.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion storage.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion storage.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group admissionregistration.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion admissionregistration.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion admissionregistration.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group apiextensions.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apiextensions.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apiextensions.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group scheduling.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion scheduling.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion scheduling.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group coordination.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion coordination.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion coordination.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group node.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion node.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group discovery.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion discovery.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group acme.cert-manager.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion acme.cert-manager.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion acme.cert-manager.io/v1alpha3"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion acme.cert-manager.io/v1alpha2"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group bootstrap.cluster.x-k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion bootstrap.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion bootstrap.cluster.x-k8s.io/v1alpha2"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group cert-manager.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion cert-manager.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion cert-manager.io/v1alpha3"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion cert-manager.io/v1alpha2"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group cluster.x-k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for GroupResource schema.GroupVersionResource{Group:\"cluster.x-k8s.io\", Version:\"v1alpha3\", Resource:\"clusters\"}"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace capi-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [capi-system]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [capi-webhook-system]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace cert-manager [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [cert-manager]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace default [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 1 clusters in namespace [default]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [kube-node-lease]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace kube-public [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [kube-public]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace kube-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [kube-system]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [local-path-storage]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for GroupResource schema.GroupVersionResource{Group:\"cluster.x-k8s.io\", Version:\"v1alpha3\", Resource:\"clusters/status\"}"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace capi-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:07-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in capi-system [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:07-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace cert-manager [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:07-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in cert-manager [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace default [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:07-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in default [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:08-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters/status in namespace kube-public [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:08-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-public [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters/status in namespace kube-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:08-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-system [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters/status in namespace local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:08-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching resources in GroupVersion cluster.x-k8s.io/v1alpha2"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for GroupResource schema.GroupVersionResource{Group:\"cluster.x-k8s.io\", Version:\"v1alpha2\", Resource:\"clusters\"}"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters in namespace capi-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:08-04:00" level=debug msg="Found 0 clusters in namespace [capi-system]"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters in namespace capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 0 clusters in namespace [capi-webhook-system]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace cert-manager [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 0 clusters in namespace [cert-manager]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace default [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 1 clusters in namespace [default]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 0 clusters in namespace [kube-node-lease]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace kube-public [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 0 clusters in namespace [kube-public]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace kube-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:10-04:00" level=debug msg="Found 0 clusters in namespace [kube-system]"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters in namespace local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:10-04:00" level=debug msg="Found 0 clusters in namespace [local-path-storage]"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for GroupResource schema.GroupVersionResource{Group:\"cluster.x-k8s.io\", Version:\"v1alpha2\", Resource:\"clusters/status\"}"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters/status in namespace capi-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:10-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in capi-system [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters/status in namespace capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:10-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters/status in namespace cert-manager [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:10-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in cert-manager [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters/status in namespace default [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in default [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching for clusters/status in namespace kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching for clusters/status in namespace kube-public [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-public [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching for clusters/status in namespace kube-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-system [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching for clusters/status in namespace local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in Group infrastructure.cluster.x-k8s.io"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion infrastructure.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion infrastructure.cluster.x-k8s.io/v1alpha2"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in Group addons.cluster.x-k8s.io"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion addons.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in Group controlplane.cluster.x-k8s.io"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion controlplane.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in Group exp.cluster.x-k8s.io"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion exp.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/capi-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/capi-webhook-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/cert-manager/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-node-lease/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-public/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/local-path-storage/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/capi-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/capi-webhook-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/cert-manager/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-node-lease/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-public/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/local-path-storage/clusters.json"

Look at clusters.json:

{
    "apiVersion": "cluster.x-k8s.io/v1alpha2",
    "items": [
        {
            "apiVersion": "cluster.x-k8s.io/v1alpha2",
            "kind": "Cluster",
            "metadata": {
                "annotations": {
                    "cluster.x-k8s.io/conversion-data": "{\"apiVersion\":\"cluster.x-k8s.io/v1alpha3\",\"kind\":\"Cluster\",\"spec\":{\"clusterNetwork\":{\"pods\":{\"cidrBlocks\":[\"192.168.0.0/16\"]}},\"controlPlaneEndpoint\":{\"host\":\"\",\"port\":0},\"infrastructureRef\":{\"apiVersion\":\"infrastructure.cluster.x-k8s.io/v1alpha3\",\"kind\":\"AWSCluster\",\"name\":\"test1\",\"namespace\":\"default\"}},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2020-10-02T19:23:23Z\",\"reason\":\"WaitingForInfrastructure\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2020-10-02T19:23:23Z\",\"reason\":\"WaitingForInfrastructure\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"InfrastructureReady\"}],\"controlPlaneInitialized\":false,\"infrastructureReady\":false,\"observedGeneration\":2,\"phase\":\"Provisioning\"}}",
                    "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"cluster.x-k8s.io/v1alpha3\",\"kind\":\"Cluster\",\"metadata\":{\"annotations\":{},\"name\":\"test1\",\"namespace\":\"default\"},\"spec\":{\"clusterNetwork\":{\"pods\":{\"cidrBlocks\":[\"192.168.0.0/16\"]}},\"infrastructureRef\":{\"apiVersion\":\"infrastructure.cluster.x-k8s.io/v1alpha3\",\"kind\":\"AWSCluster\",\"name\":\"test1\"}}}\n"
                },
                "creationTimestamp": "2020-10-02T19:19:59Z",
                "finalizers": [
                    "cluster.cluster.x-k8s.io"
                ],
                "generation": 2,
                "managedFields": [
                    {
                        "apiVersion": "cluster.x-k8s.io/v1alpha3",
                        "fieldsType": "FieldsV1",
                        "fieldsV1": {
                            "f:metadata": {
                                "f:annotations": {
                                    ".": {},
                                    "f:kubectl.kubernetes.io/last-applied-configuration": {}
                                }
                            },
                            "f:spec": {
                                ".": {},
                                "f:clusterNetwork": {
                                    ".": {},
                                    "f:pods": {
                                        ".": {},
                                        "f:cidrBlocks": {}
                                    }
                                },
                                "f:infrastructureRef": {
                                    ".": {},
                                    "f:apiVersion": {},
                                    "f:kind": {},
                                    "f:name": {}
                                }
                            }
                        },
                        "manager": "kubectl-client-side-apply",
                        "operation": "Update",
                        "time": "2020-10-02T19:19:59Z"
                    },
                    {
                        "apiVersion": "cluster.x-k8s.io/v1alpha3",
                        "fieldsType": "FieldsV1",
                        "fieldsV1": {
                            "f:metadata": {
                                "f:finalizers": {
                                    ".": {},
                                    "v:\"cluster.cluster.x-k8s.io\"": {}
                                }
                            },
                            "f:status": {
                                ".": {},
                                "f:conditions": {},
                                "f:observedGeneration": {},
                                "f:phase": {}
                            }
                        },
                        "manager": "manager",
                        "operation": "Update",
                        "time": "2020-10-02T19:23:23Z"
                    }
                ],
                "name": "test1",
                "namespace": "default",
                "resourceVersion": "2770",
                "selfLink": "/apis/cluster.x-k8s.io/v1alpha2/namespaces/default/clusters/test1",
                "uid": "677518fa-79c5-4742-862a-eadf12b3ff37"
            },
            "spec": {
                "clusterNetwork": {
                    "pods": {
                        "cidrBlocks": [
                            "192.168.0.0/16"
                        ]
                    }
                },
                "infrastructureRef": {
                    "apiVersion": "infrastructure.cluster.x-k8s.io/v1alpha3",
                    "kind": "AWSCluster",
                    "name": "test1",
                    "namespace": "default"
                }
            },
            "status": {
                "controlPlaneInitialized": false,
                "infrastructureReady": false,
                "phase": "Provisioning"
            }
        }
    ],
    "kind": "ClusterList",
    "metadata": {
        "continue": "",
        "resourceVersion": "3733",
        "selfLink": "/apis/cluster.x-k8s.io/v1alpha2/namespaces/default/clusters"
    }
}

Note that it's the v1alpha2 API version. It appears that crashd is finding both the v1alpha2 and v1alpha3 API versions of my Cluster, processing them both, and writing the results to the same file, with the last one in winning:

DEBU[0004] kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json
[snip]
DEBU[0004] kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json

If you use a RESTMapper and assume that when capturing objects, you always concatenate groups and kinds (which should really be resources) together, you can do something like this to get the server preferred version without having to loop through discovery data:

type Client struct {
  // other fields
  mapper meta.RESTMapper
}

func New(kubeconfig string) (*Client, error) {
  // other init
  res, err := restmapper.GetAPIGroupResources(disco)
  if err != nil {
    return nil, err
  }
  mapper := restmapper.NewDiscoveryRESTMapper(res)
  // store mapper in struct
}


func (k8sc *Client) _search(groups, kinds, namespaces, versions, names, labels, containers string) ([]SearchResult, error) {
	// prep code

	// (please ignore the quick hacks with variable names)
	groupsA := strings.Split(groups, ",")
	resourcesA := strings.Split(kinds, ",")
	for _, group := range groupsA {
		group = strings.TrimSpace(group)
		for _, resource := range resourcesA {
			resource = strings.TrimSpace(resource)
			gvk, err := k8sc.mapper.KindFor(schema.GroupVersionResource{Group: group, Resource: resource})
			if err != nil {
				// probably should collect errors, continue, and return an aggregate at the end
				return nil, err
			}
			mapping, err := k8sc.mapper.RESTMapping(gvk.GroupKind())
			if err != nil {
				// probably should collect errors, continue, and return an aggregate at the end
				return nil, err
			}

			// retrieve API objects based on GroupVersionResource and
			// filter by namespaces and names
			listOptions := metav1.ListOptions{
				LabelSelector: labels,
			}

			gvr := mapping.Resource

			// gather found resources
			var results []SearchResult
			if mapping.Scope.Name() == meta.RESTScopeNameNamespace {
				for _, ns := range splitParamList(namespaces) {
					logrus.Debugf("Searching for %s in namespace %s [GroupRes: %v]", resource, ns, gvr)
					list, err := k8sc.Client.Resource(gvr).Namespace(ns).List(listOptions)
					if err != nil {
						logrus.Debugf(
							"WARN: K8s.Search failed to get %s in %s [GroupRes: %s][labels: %v]: %s",
							resource, ns, gvr.GroupVersion(), listOptions.LabelSelector, err,
						)
						continue
					}
					logrus.Debugf("Found %d %s in namespace [%s]", len(list.Items), resource, ns)
					result := SearchResult{
						ListKind:             list.GetKind(),
						ResourceName:         resource,
						ResourceKind:         mapping.GroupVersionKind.GroupKind().Kind,
						Namespaced:           true,
						Namespace:            ns,
						GroupVersionResource: gvr,
						List:                 list,
					}
					results = append(results, result)
				}
			} else {
				logrus.Debugf("Searching for resource %s (non-namespaced)", resource)
				list, err := k8sc.Client.Resource(gvr).List(listOptions)
				if err != nil {
					logrus.Debugf(
						"WARN: K8s.Search failed to get %s: [GroupRes: %s] [labels: %v]: %s",
						resource, gvr.GroupVersion(), listOptions.LabelSelector, err,
					)
					continue
				}
				logrus.Debugf("Found %d %s (non-namespaced)", len(list.Items), resource)
				result := SearchResult{
					ListKind:             list.GetKind(),
					ResourceName:         resource,
					ResourceKind:         mapping.GroupVersionKind.GroupKind().Kind,
					Namespaced:           false,
					GroupVersionResource: gvr,
					List:                 list,
				}
				results = append(results, result)
			}

			// apply name filters
			for _, result := range results {
				filteredResult := result
				if len(containers) > 0 && result.ListKind == "PodList" {
					filteredResult = filterPodsByContainers(result, containers)
					logrus.Debugf("Found %d %s with container filter [%s]", len(filteredResult.List.Items), filteredResult.ResourceName, containers)
				}
				if len(names) > 0 {
					filteredResult = filterByNames(result, names)
					logrus.Debugf("Found %d %s with name filter [%s]", len(filteredResult.List.Items), filteredResult.ResourceName, names)
				}
				finalResults = append(finalResults, filteredResult)
			}
		}
	}

	return finalResults, nil
}
@ncdc
Copy link
Author

ncdc commented Oct 2, 2020

With my hacky changes from above (and limiting to just the default namespace to cut down on noise):

DEBU[0000] creating working directory /tmp/crashd
DEBU[0000] kube_capture(what=objects)
DEBU[0000] Search filters groups:[]; kinds:[clusters]; namespaces:[default]; versions:[]; names:[]; labels:[] containers:[]
DEBU[0000] Searching for clusters in namespace default [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]
DEBU[0000] Found 1 clusters in namespace [default]
DEBU[0000] kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json

@vladimirvivien
Copy link
Contributor

@ncdc thanks for the superb detailed issue.

@mcwumbly
Copy link

mcwumbly commented Jul 7, 2021

Just chiming in to note taht @gwang550 and I just ran into this problem when exploring how to provide guidance on how to troubleshoot issues bootstrapping a management cluster due to connectivity issues to the infrastructure APIs.

In some scenarios, we were looking at the vspherecluster objects on the cluster and seeing these conditions:

            "status": {
                "conditions": [
                    {
                        "lastTransitionTime": "2021-07-06T20:12:44Z",
                        "status": "True",
                        "type": "Ready"
                    },
                    {
                        "lastTransitionTime": "2021-07-06T20:12:44Z",
                        "status": "True",
                        "type": "LoadBalancerAvailable"
                    },
                    {
                        "lastTransitionTime": "2021-07-06T20:12:44Z",
                        "message": "Secret \"tkg-mgmt-vc\" not found",
                        "reason": "VCenterUnreachable",
                        "severity": "Error",
                        "status": "False",
                        "type": "VCenterAvailable"
                    }
                ],
                "ready": true
            }

But when we grabbed the resource with crashd, we saw this instead:

            "status": {
                "apiEndpoints": [
                    {
                        "host": "2620:124:6020:c00a:0:a:cccc:eebb",
                        "port": 6443
                    }
                ],
                "ready": true
            }

We were confused about why the crashd output was missing the conditions, but in looking at the output more closely, we saw that the apiVersion of the resource was older (v1alpha2 instead of v1alpha3).

@vladimirvivien
Copy link
Contributor

@mcwumbly will move this higher in priority.
Thanks for pointing this out.

@vladimirvivien vladimirvivien mentioned this issue Jul 14, 2021
Merged
@vladimirvivien
Copy link
Contributor

I @mcwumbly #207 implemented some changes that should fix your issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ncdc @vladimirvivien @mcwumbly