Proposal: Check-in generated golang protos in envoy/api

[nareddyt]

Hi envoy-api shepards,

Given the recent improvement to the golang APIs via #8003 and the upcoming goproto compatible version of go-control-plane via envoyproxy/go-control-plane#213, I wanted to propose another change: Is it possible that we could check in generate golang protos into the api/ folder?

Most golang code uses the built-in go build system instead of Bazel. This is a standard in the open-source golang community. This means that the generated golang protos (.pb.go files) must be checked into these repos, allowing consumers to depend upon them and build them using the go build system.

Currently, envoy's api repository does not have the generated golang protos. Instead, both go-control-plane/envoy and gprc-go/xds/internal/proto/envoy contain the generated envoy golang protos. This is a problem because:

1. Each repo needs to manually update their generated protos everytime any protos in envoy are updated
2. There is no single source-of-truth for generated protos. We could modify either go-control-plane or grpc-go to serve as the SOT, but it would bring in more dependencies than required. The ideal solution is to have the data-plane-api be the SOT.
3. We recently split the UDPA apis into cncf/udpa. With this change, grpc-go was also forced to add generation for the UDPA protos. Here are the generated protos in grpc-go. Once again, this breaks the single source-of-truth.
4. In the worst case, it may be impossible for external golang code to import both go-control-plane and grpc-go due to "duplicate proto type registered" errors. This would be extremely painful to solve.

Checking in generated files is OK for golang protos. Other than go-control-plane and grpc-go, some other repositories also follow the same structure for non-envoy protos:

• census-instrumentation/opencensus-proto in the gen-go/ folder
• googleapis/go-genproto in the googleapis/ folder

It would be ideal to have golang protos auto-generated and checked-in to the api/ folder whenever any of the underlying protos files are updated. Proto generation would occur with protoc using grpc and protoc-gen-validate.

I know that envoy primarily supports C++, but this change would help drive golang adoption for the control plane.

[htuch]

@nareddyt I get that this is the way that the Go proto ecosystem tends to work. Here is how I can see it being realistic for Envoy if you want this:

1. We do this in a mirror repository, e.g. https://github.com/envoyproxy/data-plane-api. This avoid https://github.com/envoyproxy/envoy noise each time we do some API changes.
2. We figure out how to do this with Bazel (not a parallel ad-hoc proto generation system).
3. We have an OWNER for this who can do the original work and be an on-going support/maintenance person from the Go world.

If we have all the above satisfied, I think this would be totally reasonable and I'm supportive.

[htuch]

CC other @envoyproxy/api-shepherds

[kyessenov]

I agree that the generated go protobufs need to be published somewhere to avoid bifurcation in go/envoy ecosystem. I was going to add a script to mirror generated go files from bazel to a repo (something like https://github.com/kythe/kythe/blob/master/kythe/proto/generate_go_protobufs.py). The question would remain about versioning scheme. I also had work in progress to move gogo ad-hoc script into bazel world.

[nareddyt]

Thanks for the quick reply @htuch. Some of my thoughts:

1. That makes sense. If these could be auto-generated from any changes to envoy/api but checked into only envoyproxy/data-plane-api, that would be perfect.
2. With api: organize go_proto_libraries #8003 merged it, it should be possible to generate golang protos in the correct directory structure using Bazel. I can work with Kuat on a prototype if needed.
3. I can help maintain the setup (if needed), but I am new to the golang ecosystem. I'd certainly need help doing so, and help with the initial setup of this.

[mattklein123]

I'm strongly in favor of the solution that @htuch proposes which is to mirror the Go protos on every master commit into a new repo. I think this should be pretty easy and would be great for the community. I know we would use this at Lyft and I was actually discussing this with @lita and @ryancox today. It's possible we could help out with this also.

[kyessenov]

It should be possible to create a new branch in go-control-plane (v2?) to host the standard protos. The import path would be a bit strange, e.g. github.com/envoyproxy/go-control-plane/v2/envoy/api/v2/core.

[kyessenov]

PR #8163 deleted all gogoproto annotations to help with separating go/gogo proto builds.

[nareddyt]

Status Update:

• Transition to use golang/protobuf to support automated code generation go-control-plane#226 updated the master branch of go-control-plane with golang/protobuf protos. gogo/protobuf compatible code is in the gogo branch.
• These changes were made by manually running the script in api: add go proto generation script #8155. @kyessenov plans on automating this in CI to generate the protos automatically on commits to envoy/api.

@kyessenov: Should this issue be closed now, or will you close it after the CI change?

[mattklein123]

Let's close this after we get CI auto-merge working.