Issue "Not the leader"

[sneko]

Hi,

I migrated my cluster with Velero to a new cluster and it seems since then the operator is unable to do anything. It doesn't recreate deployment for the OpenTelemetryCollector resource I create.

I tried to remove the operator deployment and recreate but here are the logs I get:

{"level":"info","ts":1587319935.2470012,"logger":"cmd","msg":"Starting the OpenTelemetry Operator","opentelemetry-operator":"v0.0.2","opentelemetry-collector":"0.0.2","build-date":"2019-10-07T12:05:27Z","go-version":"go1.12.9","go-arch":"amd64","go-os":"linux","operator-sdk-version":"v0.10.0"}
{"level":"info","ts":1587319935.24741,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1587319935.507039,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587319936.6374328,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587319939.018057,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587319943.5538852,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587319952.259436,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587319969.6250787,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587319987.8278975,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587320004.0431714,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587320020.5508134,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587320036.8663146,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587320053.8358185,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587320071.4925714,"logger":"leader","msg":"Not the leader. Waiting."}

Do you have any idea what could cause that?

Thank you,

[objectiser]

Looks like the relevant code is here: https://github.com/operator-framework/operator-sdk/blob/master/pkg/leader/leader.go#L45

Appears that a ConfigMap is being used as a lock - so that exists still preventing the new operator from becoming the leader.

[jpkrohling]

I've seen a few mentions to this logic being fixed due to quite a few bugs. I believe this might be made irrelevant by #23, but I'll keep this here open until we have a confirmation that the upgrade does fix the issue.

[jpkrohling]

@sneko, could you please try the operator from master? It has been updated to use kubebuilder, and I believe this problem should have been fixed there.

[Sam8301-arch]

I am having same issue while restoring keda from velero and getting same message.
Does someone fix that?

[Sam8301-arch]

Logs first install:-
{"level":"info","ts":1600305090.2607338,"logger":"cmd","msg":"KEDA Version: 1.5.0"}
{"level":"info","ts":1600305090.2607734,"logger":"cmd","msg":"Go Version: go1.14.4"}
{"level":"info","ts":1600305090.2607784,"logger":"cmd","msg":"Go OS/Arch: linux"}
{"level":"info","ts":1600305090.2607815,"logger":"cmd","msg":"Version of operator-sdk: v0.11.0"}
{"level":"info","ts":1600305090.2609844,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1600305092.6807027,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1600305092.6883569,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1600305095.0938156,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}

Issue Logs :-
{"level":"info","ts":1600307747.1605864,"logger":"cmd","msg":"KEDA Version: 1.5.0"}
{"level":"info","ts":1600307747.1606233,"logger":"cmd","msg":"Go Version: go1.14.4"}
{"level":"info","ts":1600307747.1606328,"logger":"cmd","msg":"Go OS/Arch: linux"}
{"level":"info","ts":1600307747.160639,"logger":"cmd","msg":"Version of operator-sdk: v0.11.0"}
{"level":"info","ts":1600307747.1608016,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1600307747.5870316,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1600307748.7144656,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1600307751.0974603,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1600307755.6356907,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1600307764.34399,"logger":"leader","msg":"Not the leader. Waiting

[jpkrohling]

What's Keda? What's Velero? Are you able to reproduce with the latest from master?

[Sam8301-arch]

Velero is for taking backup/restore k8's clusters. KEDA is for scaling up k8's. I am having this issue while restoring KEDA from Velero similar to @sneko(telemetry/velero) so thought to share my issues on this thread however I have also share same on KEDA and velero githubs.

[jpkrohling]

If you can reproduce this in the latest version from master, let us know. You might have found an edge-case problem that we might need to address.

[sneko]

@jpkrohling , sorry for the low reactvity 👎

Just taking a look at it, I wrote some notes months ago about how to install, it was basic:

kubectl create -n tracing -f https://raw.githubusercontent.com/open-telemetry/opentelemetry-operator/master/deploy/crds/opentelemetry.io_opentelemetrycollectors_crd.yaml
kubectl create -n tracing -f https://raw.githubusercontent.com/open-telemetry/opentelemetry-operator/master/deploy/service_account.yaml
kubectl create -n tracing -f https://raw.githubusercontent.com/open-telemetry/opentelemetry-operator/master/deploy/role.yaml
kubectl create -n tracing -f https://raw.githubusercontent.com/open-telemetry/opentelemetry-operator/master/deploy/role_binding.yaml
kubectl create -n tracing -f https://raw.githubusercontent.com/open-telemetry/opentelemetry-operator/master/deploy/operator.yaml

But now, by reading the README.md I'm not even sure what's need to be done to deploy the operator (and previous links are no longer existing).

Is there anything specific that is straight forward? Or maybe I'm too tired to see the right section ^^

Thank you,

EDIT: ok just took a look at the Makefile, saw some kubectl commands. That's probably the way to go but in the README it says:

Getting started
To run the operator locally, run:

I think I misunderstood the "to run the operator locally", locally? an operator deployed locally on the cluster, right?

[jpkrohling]

Right, I need to work on that readme file to clarify this.

Run locally means that the operator itself will be running locally, connecting to a Kubernetes cluster via the currently available kubeconfig (ie, whatever kubectl is configured to use). That's make install run. This does not create a Deployment with the operator, and having the webhooks to work here is tricky.

When you run make manifests docker-build docker-push deploy, it will deploy a "proper" operator in your Kubernetes cluster (the one your kubectl points to). This does create a Deployment in the operator and webhooks are going to be functional, as long as cert-manager is available (hint: make cert-manager).

That said, I'm right now working on the release scripts, and it should be way easier to deploy the operator later today or sometime tomorrow. You'll just need to feed a specific YAML into your Kubernetes cluster.

[jpkrohling]

I'm closing this one, as I haven't seen reports of this happening lately.