Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Operator crash after update to v0.7.0 #583

Closed
angelprada opened this issue Jul 4, 2023 · 5 comments · Fixed by #592
Closed

Operator crash after update to v0.7.0 #583

angelprada opened this issue Jul 4, 2023 · 5 comments · Fixed by #592
Labels
bug Something isn't working controller operations Dealing with the reconcile-loop and other internal aspects of the various Solr operator controllers
Milestone
v0.7.1

Comments

@angelprada
Copy link

angelprada commented Jul 4, 2023 •
edited
Loading

Hello.

I've tried to upgrade solr-operator from v0.6.0 to v0.7.0 in AWS EKS 1.27 but it crashes constantly:

2023-07-03T08:49:18Z	INFO	setup	solr-operator Version: v0.7.0
2023-07-03T08:49:18Z	INFO	setup	solr-operator Git SHA: 0e16c6b
2023-07-03T08:49:18Z	INFO	setup	solr-operator Build Time: Mon Apr 24 19:23:35 UTC 2023
2023-07-03T08:49:18Z	INFO	setup	Go Version: go1.19.8
2023-07-03T08:49:18Z	INFO	setup	Go OS/Arch: linux / amd64
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12ab950]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc00004f560, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc00004e3f0, 0x15}, {0xc000049060, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x1815ff3?)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc000001e00?)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x1a8a320, 0xc0002b32f0})
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc0004c50d0)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x1a8a320?, 0xc0002b32f0?})
	/go/pkg/mod/k8s.io/client-go@v0.26.1/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/client/apiutil/dynamicrestmapper.go:86 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/client/apiutil/dynamicrestmapper.go:119
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc000358870?, {0x0, 0x0, 0x2fb3e8039a338f01?})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/client/apiutil/dynamicrestmapper.go:99 +0x176
sigs.k8s.io/controller-runtime/pkg/cluster.setOptionsDefaults.func1(0xc00011d6c0?)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/cluster/cluster.go:217 +0x25
sigs.k8s.io/controller-runtime/pkg/cluster.New(0xc0001c3200, {0xc000143a58, 0x1, 0x0?})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/cluster/cluster.go:159 +0x18d
sigs.k8s.io/controller-runtime/pkg/manager.New(_, {0xc00011d6c0, 0x0, 0x0, {{0x1a87000, 0xc00012f1c0}, 0x0}, 0x1, {0x0, 0x0}, ...})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/manager/manager.go:351 +0xf9
main.main()
	/workspace/main.go:171 +0x7b5
@HoustonPutman
Copy link
Contributor

Unfortunately I can't replicate this on a KinD cluster running 1.27, but I did find this when googling:

elastic/cloud-on-k8s#6848

So maybe we just need to update the version of the kubernetes client for the operator.

@angelprada
Copy link
Author

Hello.

We had similar problems with external-secrets external-secrets/external-secrets#2344

Hope it helps.

@HoustonPutman HoustonPutman mentioned this issue Jul 17, 2023
Merged
@HoustonPutman HoustonPutman added this to the v0.7.1 milestone Jul 17, 2023
@HoustonPutman HoustonPutman added bug Something isn't working controller operations Dealing with the reconcile-loop and other internal aspects of the various Solr operator controllers labels Jul 17, 2023
@HoustonPutman
Copy link
Contributor

Fixed this. I'll start a v0.7.1 release soon.

@HoustonPutman
Copy link
Contributor

@angelprada The release candidate is out to try, please let me know if the docker image apache/solr-operator:v0.7.1-rc1 works for you. It should be a drop-in replacement for v0.7.0.

@angelprada
Copy link
Author

@HoustonPutman Tried and worked as expected!

2023-07-17T17:46:46Z	INFO	setup	solr-operator Version: v0.7.1
2023-07-17T17:46:46Z	INFO	setup	solr-operator Git SHA: 5205b93
2023-07-17T17:46:46Z	INFO	setup	solr-operator Build Time: Mon Jul 17 17:08:17 UTC 2023
2023-07-17T17:46:46Z	INFO	setup	Go Version: go1.19.11
2023-07-17T17:46:46Z	INFO	setup	Go OS/Arch: linux / amd64
2023-07-17T17:46:47Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
2023-07-17T17:46:47Z	INFO	setup	starting manager
2023-07-17T17:46:47Z	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-07-17T17:46:47Z	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
I0717 17:46:47.169060       1 leaderelection.go:248] attempting to acquire leader lease kube-system/88488bdc.solr.apache.org...
I0717 17:47:17.922432       1 leaderelection.go:258] successfully acquired lease kube-system/88488bdc.solr.apache.org

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working controller operations Dealing with the reconcile-loop and other internal aspects of the various Solr operator controllers
Projects
None yet
Milestone
v0.7.1
Development

Successfully merging a pull request may close this issue.

Upgrade kubernetes API for crash HoustonPutman/solr-operator
2 participants
@HoustonPutman @angelprada