Add coverage data to codecov

[justinvp]

No description provided.

[abhinav]

Status update:

#469 works fine for unit test coverage tracking—that's the straightforward bit.
However, it does not currently work for integration test coverage tracking.
The reason for this is that go test does not allow using GOCOVERDIR (for integration tests) and -coverprofile (for unit tests) at the same time. It specifically overrides GOCOVERDIR:

https://github.com/golang/go/blob/c19c4c566c63818dfd059b352e52c4710eecf14d/src/cmd/go/internal/test/test.go#L1337-L1341

So if there's a mix of unit and integration tests in the same directory, and we run the following:

GOCOVERDIR=$(pwd)/coverage go test -coverprofile foo.out

The value of GOCOVERDIR will not be propagated to the coverage-instrumented binary that we're trying to get integration testing coverage information out of.

From the original proposal for this feature, this is a known limitation that is intended to be addressed in a follow up. Per this comment, a workaround for this is to build the test binary without running the tests (go test -c), and then call it with the undocumented -test.gocoverdir flag.

# Build the test binary with coverage instrumentation:
go test -coverpkg=example.com/foo/... -c -o ./test

# Run the test binary with GOCOVERDIR set for integration test coverage
# and the -test.gocoverdir flag set for unit test coverage.
GOCOVERDIR=$(pwd)/coverage ./test -test.gocoverdir=$(pwd)/coverage

This works, but it has to be done on a per-package basis so some tooling is necessary to build and run the test binaries for each test.

I prototyped such a tool and it appears to do the job, but I'm not certain that this is a good idea long-term. It's a couple hundred lines of code built on an unsupported, undocumented workaround.

On top of that, the pulumi-language-yaml binary in particular seems to do some kind of unclean exit which leaves without writing coverage data, so even if GOCOVERDIR is propagated all the way to it, it doesn't report anything yet except 0%. I'm currently debugging that.

[abhinav]

Update: With the graceful shutdown work that was landed on pu/pu master recently, and the tool I prototyped above, we are able to successfully get reliable coverage data out of the pulumi-language-yaml binary.
The tool will be necessary only until the relevant upstream issue in go test is resolved.