Feature Request: cache static auth creds in go/vt/grpcclient

[timvaillancourt]

Feature Description

This issue requests caching of the gRPC static auth credentials in go/vt/grpcclient to avoid high OS-thread usage when many gRPC clients are loaded concurrently - something that happens at vtgate startup

Currently, this blocking syscall shows up for every watched-tablet in profiles of vtgate starting up:

(dlv) stack
 0  0x000000000047f120 in syscall.Syscall6
    at syscall/asm_linux_amd64.s:43
 1  0x000000000047bb05 in syscall.openat
    at syscall/zsyscall_linux_amd64.go:68
 2  0x00000000004f7c5b in syscall.Open
    at syscall/syscall_linux.go:155
 3  0x00000000004f7c5b in os.openFileNolog
    at os/file_unix.go:216
 4  0x00000000004f6585 in os.OpenFile
    at os/file.go:338
 5  0x00000000004f694a in os.Open
    at os/file.go:318
 6  0x00000000004f694a in os.ReadFile
    at os/file.go:670
 7  0x0000000000de2f7a in vitess.io/vitess/go/vt/grpcclient.AppendStaticAuth
    at vitess.io/vitess/go/vt/grpcclient/client_auth_static.go:62
 8  0x0000000000de21cc in vitess.io/vitess/go/vt/grpcclient.DialContext
    at vitess.io/vitess/go/vt/grpcclient/client.go:105
 9  0x0000000001150a45 in vitess.io/vitess/go/vt/grpcclient.Dial
    at vitess.io/vitess/go/vt/grpcclient/client.go:63
10  0x0000000001150a45 in vitess.io/vitess/go/vt/vttablet/grpctabletconn.DialTablet
    at vitess.io/vitess/go/vt/vttablet/grpctabletconn/conn.go:80
11  0x0000000000e5a54c in vitess.io/vitess/go/vt/discovery.(*tabletHealthCheck).connectionLocked
    at vitess.io/vitess/go/vt/discovery/tablet_health_check.go:148
12  0x0000000000e5a430 in vitess.io/vitess/go/vt/discovery.(*tabletHealthCheck).Connection
    at vitess.io/vitess/go/vt/discovery/tablet_health_check.go:143
13  0x0000000000e5a2f1 in vitess.io/vitess/go/vt/discovery.(*tabletHealthCheck).stream
    at vitess.io/vitess/go/vt/discovery/tablet_health_check.go:127
14  0x0000000000e5ae9d in vitess.io/vitess/go/vt/discovery.(*tabletHealthCheck).checkConn
    at vitess.io/vitess/go/vt/discovery/tablet_health_check.go:276
15  0x0000000000e508aa in vitess.io/vitess/go/vt/discovery.(*HealthCheckImpl).AddTablet.func2
    at vitess.io/vitess/go/vt/discovery/healthcheck.go:380
16  0x00000000004692c1 in runtime.goexit
    at runtime/asm_amd64.s:1571

If you have enough blocking syscalls it is possible to hit the golang 10k thread limit, which is something we're seeing happen in production

Use Case(s)

vtgate that watches 10k+ tablets starting up

[timvaillancourt]

This was completed in #15030. Closing 🎉