TAS Policy remove is not working as there are error logs in scheduler

[Rajpratik71]

Skipping extender &{https://tas-service.default.svc.cluster.local:9001 scheduler/filter scheduler/prioritize 1 0xc000866660 false map[telemetry/scheduling:{}] true} as it returned error Failed scheduler/filter with extender at URL https://tas-service.default.svc.cluster.local:9001/scheduler/filter, code 404 and has ignorable flag set

[togashidm]

@Rajpratik71
Could you give more details about it?
So the steps needed to reproduce the issue. Also, the logs from TAS.

[nitisgoe]

Seeing the same issue.
Steps:
Apply a policy and deploy an app.
remove the policy and scale the app.
Since the policy is removed there should not be any validation and scaling should schedule pods to multi nodes
Actual: It still look for policy and schedules only the valid node.
Logs from kube scheduler

1 generic_scheduler.go:378] Skipping extender &{https://tas-service.default.svc.cluster.local:9001 scheduler/filter scheduler/prioritize 1 0xc0007937d0 false map[telemetry/scheduling:{}] true} as it returned error Failed scheduler/filter with extender at URL https://tas-service.default.svc.cluster.local:9001/scheduler/filter, code 404 and has ignorable flag set
W0323 06:29:16.049715 1 generic_scheduler.go:378] Skipping extender &{https://tas-service.default.svc.cluster.local:9001 scheduler/filter scheduler/prioritize 1 0xc0007937d0 false map[telemetry/scheduling:{}] true} as it returned error Failed scheduler/filter with extender at URL https://tas-service.default.svc.cluster.local:9001/scheduler/filter, code 404 and has ignorable flag set

there are no new logs in TAS ext or cont. It still refers to the old policy stats.

[killianmuldoon]

Let me know if I rightly understand your use case:
You're trying to schedule new pods - created using a scale up - with a telemetry aware scheduling policy. You've removed the policy from the Kubernetes API and the pod is still being scheduled, but it looks like the policy is still being applied?

The error logs are expected as the pod is still requesting scheduling using TAS - deleting the policy doesn't change the scheduler behavior here. But it's concerning if the effects of the policy are persisting after the policy is deleted.

Can you output the description of the node that you think is invalid? And the policy you were using before deleting.

[nitisgoe]

Yes. even after the policy is deleted and trying to schedule the pods after that, policy effects still persist and doesn't schedule the pods on any other node except the one which was valid on policy apply.