With min-scale > 1 and activator only the first pod receive traffic

[SharpEdgeMarshall]

What version of Knative?

0.26.0

Expected Behavior

/kind bug

If you set min-scale=2 all pods receive traffic

Actual Behavior

• Only the first pod will receive traffic if there's activator proxing reqeusts

Steps to Reproduce the Problem

Istio Version: 1.11.4
Knative Version: 0.26.0
Kubernetes version: (use kubectl version): 1.21
OS (e.g. from /etc/os-release): bottlerocket

[jwcesign]

What gateway u are using? istio?

[SharpEdgeMarshall]

Yes

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[SharpEdgeMarshall]

This should be reopened it's an actual bug hitting production workloads

[dprotaso]

/triage accepted
/reopen
/milestone v1.11.0

Want to priortize an investigation in v1.11.0 timeframe

[dprotaso]

/assign @dprotaso

Found the issue - it seems like when we have exactly 2 pods that have equal weight we will always pick the first one

serving/pkg/activator/net/lb_policy.go

Lines 65 to 70 in 88cae7f

	pick, alt := targets[r1], targets[r2]
	// Possible race here, but this policy is for CC=0,
	// so fine.
	if pick.getWeight() > alt.getWeight() {
	pick = alt
	}

If you kick off a bunch of requests in parallel then you'll get better spreading since the weight's won't be even.

[dprotaso]

This should be reopened it's an actual bug hitting production workloads

@SharpEdgeMarshall do you have a way to repro your production issue - ie. similar to #14011 ?

It would be good to know if there are other scenarios where this happens