Proxy protocol transport socket intermittently injects wrong addresses

[rchernobelskiy]

Title: Proxy protocol header intermittently uses wrong addresses

Description:
Proxy protocol injection (envoy.transport_sockets.upstream_proxy_protocol) works as expected to start out, but once you start connecting from different IPs, the injected proxy protocol header ends up randomly having addresses from previous connections.
The log shows the right IPs but proxy protocol header intermittently sends the IPs not from the current connection, but from one of the previous ones.
I thought maybe setting connection_pool_per_downstream_connection: true would help but it did not.

Repro steps:
Have envoy listen on 127.0.0.1:9000 and tcp proxy to a cluster that listens for connections, reads the proxy protocol header, and then promptly closes the connection.

# run this about 20 times:
curl --interface 127.0.0.2 127.0.0.1:9000
# then run this about 20 times
curl --interface 127.0.0.3 127.0.0.1:9000
# then run this about 20 times
curl --interface 127.0.0.4 127.0.0.1:9000

The expectation here and what the logs show is 20 connections each from the 3 separate addresses above. What the proxy protocol sends, ends up being a random mix.

Suspected cause:
I'm thinking this has something to do with connection pooling as the following comment suggests that it's not exactly a 1 to 1 setup currently: #4684 (comment)

Potentially related: #2800 #11694

[mattklein123]

This seems bad. cc @alyssawilk

[rchernobelskiy]

It seems that since tcp connection pooling was introduced, there have been a number of workarounds for use cases that need 1:1 semantics so one idea that seems attractive would be to make non-pooled tcp proxy functionality available as an option.