pubsub: Pubsub Subscription Receive silently hangs without ca-certificates in Docker

[benjaminRomano]

Client
PubSub

Environment
Docker on GKE

Go Environment
go version go1.16 darwin/amd64

go version go1.16 darwin/amd64
bromano@C02DVD7TMD6R inferno % go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/bromano/Library/Caches/go-build"
GOENV="/Users/bromano/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/bromano/go/pkg/mod"
GONOPROXY="*.sc-corp.net"
GONOSUMDB="*.sc-corp.net"
GOOS="darwin"
GOPATH="/Users/bromano/go"
GOPRIVATE="*.sc-corp.net"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.16/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.16/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/bromano/Snapchat/Dev/inferno/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/gn/s3s4gx85327869lw6bkd1ct80000gp/T/go-build2205708766=/tmp/go-build -gno-record-gcc-switches -fno-common"

$ go version
$ go env

Code

	sub := p.pubsub.Subscription(p.conf.SubscriptionID)

	err := sub.Receive(context.Background() func(ctx context.Context, msg *pubsub.Message) {
                // This callback is never triggered
                logger.Info("msg received")
                msg.Ack()
        }

Expected behavior
Receive fails with error due to missing certificates

Actual behavior
Receive hangs with no error message

Additional context
I was using the ubuntu18.04 docker image as my base and when running my pubsub receive logic, the messages would never be received.

I was able to track down the root cause to #791. I resolved it by installing ca-certificates using the following:

$ apt-get install ca-certificates

For future developers who run into this issue the quickest fix is to use Google's Ubuntu 18.04 instead which has this dependency already baked in.

[codyoss]

This is not exclusive to pubsub. This is due to how grpc-go handles these types of errors. There are some more rational to why these errors are retried in grpc/grpc-go#1742 and the linked issues. I don't think there is currently much we can do from a library standpoint. All gRPC based clients, those found in this repository, will require the presence of certs.