Updated containerd1.7; google.golang.org/protobuf

[helsaawy]

Update to containerd 1.7 and move to google.golang.org/protobuf from github.com/gogo/protobuf/gogoproto.

Changes are broken out in two commits: second commit has vendor updates, so they don't clog reviewing the original change

These two changes are intertwined, since containerd 1.7 moves and changes its ttrpc task server definitions (containerd/containerd#6827) and protobuff generation (containerd/containerd#6841), as well as some other API changes.
Containerd updated to "github.com/containerd/typeurl/v2", so now the types it registers by default (in "github.com/containerd/containerd/runtime"'s init) are not available to the shim code unless the hcsshim repo either (1) switches to typeurl/v2 or (2) duplicates the containerd typerurl registration code.
Since 1.7 does not use gogoproto, all references to it (eg "github.com/gogo/protobuf/types".Empty) have to be updated.
Upgrading to google.golang.org/protobuf also requires updating the containerd/cgroups dependency to v3
(github.com/containerd/cgroups/v3/cgroup1/stats/), since the older v1 still uses gogoproto.
Additionally, the containerd protoc-gen-gogoctrd is now deprecated and not included in containerd 1.7, so that generator is removed.

The changes could be isolated, so only ./cmd/containerd-shim-runhcs-v1 (and internal code it relies on) that implements the containerd task server and interfaces with containerd is updated.
However, this would result in bifurcating the project in two, with parts of the code base using the deprecated protobuf packages and marshalling, and requiring two different protobuf generation procedures (including cloning containerd 1.6 to use the now-deprecated protoc-gen-gogoctrd generator).
This would likely add more work and complicate project management.
It is also not guaranteed that there would be a clean split between all the code, as there is likely shared code that would need to be updated to either use the new protobuf type ("google.golang.org/protobuf/types/known/emptypb", "google.golang.org/protobuf/types/known/timestamppb", and co) or keep the old types from "github.com/gogo/protobuf/types"

The protoc-gen-go-grpc generator do not allow directives such as gogoproto.customname, so the go-fix-acronym command is used to update acronym customization (similar to what containerd does).

Added protobuild and protobuf grpc and ttrpc generators to tools.go so they are tracked and vendored, and can be trivially installed via go install.

Added an Update-Proto.ps1 script to re-gerenate protobuf files locally and in GitHub CI.

[kevpar]

Can we split this into two PRs:

• Moving off of gogo
• Updating to containerd 1.7, and any other changes specific to that

Edit: To clarify, I see from your description that we need to move to gogo to move to 1.7, but I'm wondering if we also have to move to 1.7 to move to gogo?

[helsaawy]

Edit: To clarify, I see from your description that we need to move to gogo to move to 1.7, but I'm wondering if we also have to move to 1.7 to move to gogo?

Yes, since if we move off of gogo, then our marshalling will differ from containerd1.6, and we wont be able to communicate with them (and likely won't implement task server correctly, since the return types will be different).

edit: the task server interface specifies gogo return types: https://github.com/containerd/containerd/blob/de33abf0547cb56bed2b7ea74cbbf0fe903c00b8/runtime/v2/task/shim.pb.go#L3429
plus, the requests themselves have gogo types within them:
https://github.com/containerd/containerd/blob/de33abf0547cb56bed2b7ea74cbbf0fe903c00b8/runtime/v2/task/shim.pb.go#L36

[kevpar]

Yes, since if we move off of gogo, then our marshalling will differ from containerd1.6, and we wont be able to communicate with them (and likely won't implement task server correctly, since the return types will be different).

I'm confused by this. As long as both sides are speaking protobuf, it shouldn't matter what internal implementation types they use, right?

[katiewasnothere]

I'm confused by this. As long as both sides are speaking protobuf, it shouldn't matter what internal implementation types they use, right?

@kevpar I think there's some bug in the protobuf code. With the cri-containerd tests, I've been seeing errors like this even on main because the protobuf versions are different:

runpodsandbox_test.go:1927: failed to exec `ps` in the uvm with ttrpc: cannot marshal unknown type: *shimdiag.ExecProcessRequest

[helsaawy]

I'm confused by this. As long as both sides are speaking protobuf, it shouldn't matter what internal implementation types they use, right?

technically since grpc isnt fully self-describing, it needs a type to unpack it into when unmarshalling the byte payload.
if the (ttrpc) server is expecting a "time".Time instead of "google.golang.org/protobuf/types/known/timestamppb".Timestamp (or similar for Any), it will error out since the underlying data won't fit. (I think the Any and Empty types are the same, but I doubt theres a guarantee for that compatibility)

[kevpar]

technically since grpc isnt fully self-describing, it needs a type to unpack it into when unmarshalling the byte payload. if the (ttrpc) server is expecting a "time".Time instead of "google.golang.org/protobuf/types/known/timestamppb".Timestamp (or similar for Any), it will error out since the underlying data won't fit. (I think the Any and Empty types are the same, but I doubt theres a guarantee for that compatibility)

I must be missing something here. time.Time would never be sent via protobuf, instead we would send google.protobuf.Timestamp, and only on the client or server end does it convert it into a Go type to work with. As long as the same protobuf definitions are used on both ends, and the protobuf wire format is used to transport between, I don't see how the client's or server's choice of language package would make a difference.

Is there a specific example of the issue you can share?

[helsaawy]

I must be missing something here. time.Time would never be sent via protobuf, instead we would send google.protobuf.Timestamp, and only on the client or server end does it convert it into a Go type to work with. As long as the same protobuf definitions are used on both ends, and the protobuf wire format is used to transport between, I don't see how the client's or server's choice of language package would make a difference.

oh, youre right, I was mistaken.
but the issue still remains that the generated protobuf structs live in containerd for the taskserver
so if we wanted to switch to google.golang.org/protobuf while on containerd1.6, we'd either need to

1. fork their runtime/v2/task/shim.proto file and generate it with the new protobuf stack; or
2. have half our code deal with the old gogo types and the other half use the new one

[helsaawy]

rebased to fix merge conflicts

[katiewasnothere]

lgtm. Should we make like a last release of the shim pre-containerd 1.7 change?

[ambarve]

Minor question but LGTM otherwise!

[helsaawy]

Rebased to get mockgen changes, and update dependencies to the latest version