GRPCRoute conformance tests failing with client connection is closing

[ciarams87]

What happened:

Our GRPCRoute conformance tests are failing running against the main branch with

RPC finished with error: rpc error: code = Canceled desc = grpc: the client connection is closing

These are running and working successfully against v1.1.0.

What you expected to happen:

The requests are not cancelled and the tests pass

How to reproduce it (as minimally and precisely as possible):

Run the conformance test suite against any implementation using the default GRPC client

Anything else we need to know?:

As far as I can tell, this is happening as a result of #3095.

As part of the changes introduced in this PR, we are sharing one client between all requests instead of instantiating a new client for every GRPC request. However, each request is still calling defer c.Close() which means the client connection will be closed when this function returns even if another request is using the connection. Additionally, each call to sendRPC sets the connection specific to the call being made, and will reset the current connection if present. Because the tests run in parallel, this is causing conflicts and race conditions.

Example conformance test run with the failures: https://github.com/nginxinc/nginx-gateway-fabric/actions/runs/9312728851/job/25634135132

[ciarams87]

I believe a straight forward fix will be to return the default client instantiation to every request instead of having a top level shared client - like here.

Happy to work on this and open a PR if it makes sense!

[robscott]

Thanks for catching this @ciarams87! The fix looks reasonable to me, but would also like to have @snehachhabria and @gnossen review any follow up PR.

[ciarams87]

/assign @ciarams87