imc-dispatcher dial tcp <IP>: connect: cannot assign requested address

[maschmid]

Describe the bug

After about a day of cluster runtime, deploying brokers backed by InMemoryChannels (at the time of the errors there are 24 brokers on the cluster, each with a single trigger), sending events with replies, I have noticed the following errors present in imc-dispatcher logs:

"dial tcp : connect: cannot assign requested address"

in errors like these:

{"level":"error","ts":"2020-11-04T18:51:54.205Z","logger":"inmemorychannel-dispatcher","caller":"fanout/fanout_message_handler.go:189","msg":"Fanout had an error","error":"failed to forward reply to http://broker-ingress.knative-eventing.svc.cluster.local/foo10/broker: Post \"http://broker-ingress.knative-eventing.svc.cluster.local/foo10/broker\": dial tcp 172.30.107.69:80: connect: cannot assign requested address","stacktrace":"knative.dev/eventing/pkg/channel/fanout.(*MessageHandler).dispatch\n\t/opt/app-root/src/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:189\nknative.dev/eventing/pkg/channel/fanout.createMessageReceiverFunction.func1.1\n\t/opt/app-root/src/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:143"}

{"level":"error","ts":"2020-11-04T18:52:02.334Z","logger":"inmemorychannel-dispatcher","caller":"fanout/fanout_message_handler.go:189","msg":"Fanout had an error","error":"unable to complete request to http://broker-filter.knative-eventing.svc.cluster.local/triggers/foo6/counter/6c5ab867-5345-449a-aef7-7023693bf821: Post \"http://broker-filter.knative-eventing.svc.cluster.local/triggers/foo6/counter/6c5ab867-5345-449a-aef7-7023693bf821\": dial tcp 172.30.166.136:80: connect: cannot assign requested address","stacktrace":"knative.dev/eventing/pkg/channel/fanout.(*MessageHandler).dispatch\n\t/opt/app-root/src/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:189\nknative.dev/eventing/pkg/channel/fanout.createMessageReceiverFunction.func1.1\n\t/opt/app-root/src/go/src/knative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:143"}

suggesting there may be sockets leaking

Expected behavior
"cannot assign requested address" errors should not appear

To Reproduce
Currently unknown

Knative release version
0.17.2

Additional context

[slinkydeveloper]

/assign

[matzew]

related? cloudevents/sdk-go#98

[matzew]

looks like by default, the net/http allows 2 connections in its connection pool to a single host

see: golang/go#16012 (comment)

[slinkydeveloper]

Yeah that's what i was suspecting, my feeling is that we set this parameter too high

[slinkydeveloper]

So the actual limit config is:

transport.MaxIdleConns = 1000
transport.MaxIdleConnsPerHost = 100

Which means it can open a maximum of 1000 sockets, i guess not enough to leak all sockets on the machine... Let me try to figure out if we always use the same http client, or if somewhere we create new ones