HorizontalPodAutoscaler continuously creates and terminates pods

[kevinearls]

I have been trying to work on #801 and in simply trying to get something to scale I saw some odd behavior.

The collector does seem to scale correctly (at least eventually) but collector pods are repeatedly being created and then immediately terminated. To reproduce this, do the following, which deploys a simple collector instance and uses tracegen to create traffic:

• Create/Login to an OpenShift instance. (NOTE: so far I have not been able to get this to scale at all with minikube despite following the official instructions and several blog posts based on those.)
• Install the otel-operator
• Download the attached files otel-tracegen.txt and otel-collector-simplest.txt and rename them to otel-tracegen.yaml and otel-collector-simplest.yaml
• Execute the following commands:

kubectl create namespace simple
kubectl apply --namespace simple -f ./otel-collector-simplest.yaml
kubectl apply --namespace simple -f ./otel-tracegen.yaml

• In separate terminal windows, execute the following commands:

kubectl get hpa --namespace simple --watch
kubectl get pods --namespace simple --watch

After 4 or 5 minutes the window with the kubectl get hpa... command should finally return an entry where there is a real value under TARGET rather than unknown. At that point the window with the kubectl get pods... command will start constantly streaming Pending/ContainerCreating/Terminating events.

Eventually the deployment will scale, but pod Pending/ContainerCreating/Terminating will continue

otel-collector-simplest.txt
otel-tracegen.txt

[kevinearls]

I found the problem with minikube: kubernetes/minikube#13969 So the solution is to start minikube with --extra-config=kubelet.housekeeping-interval=10s

This enables this problem to be reproduced with minikube. To reproduce:

• Download the attached files otel-tracegen.txt and otel-collector-simplest.txt and rename them to otel-tracegen.yaml and otel-collector-simplest.yaml
• Download reproduce.txt, rename it reproduce.sh and make it executable
• Run the reproduce.sh script
• In a separate terminal window execute:

kubectl get pods --namespace simple --watch

reproduce.txt

[pavolloffay]

@kevinearls was this resolved by #984 ?

[kevinearls]

@pavolloffay Yes, I just checked.