Transition to use golang/protobuf to support automated code generation #226

kyessenov · 2019-09-09T17:46:38Z

Fixes #213

Warning This is a breaking change due to the switch from gogo/protobuf to golang/protobuf. Please use branch gogo if you would like to continue using gogo/protobuf.

Signed-off-by: Kuat Yessenov <kuat@google.com>

Needed to be compatible with internal build rules. Signed-off-by: Teju Nareddy <nareddyt@google.com> Signed-off-by: Kuat Yessenov <kuat@google.com>

Signed-off-by: Kuat Yessenov <kuat@google.com>

rshriram · 2019-09-10T17:20:52Z

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "data-plane-api"]
-	path = data-plane-api
-	url = https://github.com/envoyproxy/data-plane-api


?? why remove this?

The stubs will be pushed from envoy repo directly. I want to have the first commit done manually before I automate the pushes since we still need to validate go module build (which bazel doesn't do).

See envoyproxy/envoy#8155

yangminzhu

I'm not following with the changes so just some questions to learn about the change, do we need to run generate_go_protobuf.py manually to generate all these go files in this repo? Or is it done by some bots automatically? If so how often is it pushed? Thanks

kyessenov · 2019-09-10T22:06:30Z

This PR was done by manually running the script. The automation is a follow-up. The script will be triggered by any change to APIs.

jyotimahapatra · 2019-09-17T21:09:19Z

@kyessenov I am trying this out with my sample control plane which was working before this change.
I am getting errors like this

./main.go:13:2: imported and not used: "github.com/envoyproxy/go-control-plane/envoy/api/v2/auth" as envoy_api_v2_auth
./main.go:14:2: imported and not used: "github.com/envoyproxy/go-control-plane/envoy/api/v2/core" as envoy_api_v2_core
./main.go:15:2: imported and not used: "github.com/envoyproxy/go-control-plane/envoy/api/v2/route" as envoy_api_v2_route
./main.go:24:11: undefined: core
./main.go:31:24: undefined: core
./main.go:96:8: undefined: core
./main.go:108:3: cannot use &duration (type *time.Duration) as type *duration.Duration in field value
./main.go:111:28: undefined: core
./main.go:112:26: undefined: auth
./main.go:117:20: undefined: route
./main.go:117:20: too many errors

Is there an easy way to fix this?

kyessenov · 2019-09-17T21:15:32Z

Name every import, e.g.

import (
  auth "github.com/envoyproxy/go-control-plane/envoy/api/v2/auth"
)

jyotimahapatra · 2019-09-17T21:18:45Z

@kyessenov
I tried that, but this doesn't work

v2 "github.com/envoyproxy/go-control-plane/envoy/api/v2"
"github.com/gogo/protobuf/jsonpb"
"github.com/gogo/protobuf/proto"
..
..
xDS := map[string]proto.Message{
                "rds": &v2.RouteConfiguration{},
                "cds": &v2.Cluster{},
        }

./tools/xds-validator/xds-validator -t rds -p generated/routes/ file: xyz.json error: unknown field "exact_match" in envoy_api_v2_route.HeaderMatcher

kyessenov · 2019-09-17T21:23:39Z

Please use golang/protobuf/jsonpb and not gogo/protobuf/jsonpb to parse JSON into golang/protos.

jyotimahapatra · 2019-09-18T00:07:50Z

Thank you @kyessenov . While I was migrating all usages of gogo to golang, my project's glide up failed with this piece of error

^[[0;32m[INFO]  ^[[m--> Setting version for github.com/envoyproxy/go-control-plane to v0.9.0.
[DEBUG] Unlocking git-gh.neting.cc-lyft-go-analytics
[DEBUG] Unlocking https-gh.neting.cc-onsi-gomega
^[[0;32m[INFO]  ^[[m--> Setting version for github.com/golang/protobuf to v1.3.2.
...
...
...
[DEBUG] Trying to open github.com/envoyproxy/go-control-plane/envoy/api/v2 (/Users/jmahapatra/.glide/cache/src/https-gh.neting.cc-envoyproxy-go-control-plane/envoy/api/v2)
[DEBUG] Package github.com/envoyproxy/go-control-plane/envoy/api/v2 imports github.com/envoyproxy/go-control-plane/envoy/api/v2/auth
[DEBUG] In vendor: github.com/envoyproxy/go-control-plane/envoy/api/v2/auth
[DEBUG] Package github.com/envoyproxy/go-control-plane/envoy/api/v2 imports github.com/envoyproxy/go-control-plane/envoy/api/v2/cluster
[DEBUG] In vendor: github.com/envoyproxy/go-control-plane/envoy/api/v2/cluster
[DEBUG] Package github.com/envoyproxy/go-control-plane/envoy/api/v2 imports github.com/envoyproxy/go-control-plane/envoy/api/v2/core
[DEBUG] In vendor: github.com/envoyproxy/go-control-plane/envoy/api/v2/core
[DEBUG] Package github.com/envoyproxy/go-control-plane/envoy/api/v2 imports github.com/envoyproxy/go-control-plane/envoy/api/v2/endpoint
[DEBUG] In vendor: github.com/envoyproxy/go-control-plane/envoy/api/v2/endpoint
[DEBUG] Package github.com/envoyproxy/go-control-plane/envoy/api/v2 imports github.com/envoyproxy/go-control-plane/envoy/api/v2/listener
[DEBUG] In vendor: github.com/envoyproxy/go-control-plane/envoy/api/v2/listener
[DEBUG] Package github.com/envoyproxy/go-control-plane/envoy/api/v2 imports github.com/envoyproxy/go-control-plane/envoy/api/v2/route
[DEBUG] In vendor: github.com/envoyproxy/go-control-plane/envoy/api/v2/route
[DEBUG] Package github.com/envoyproxy/go-control-plane/envoy/api/v2 imports github.com/envoyproxy/go-control-plane/envoy/type
[DEBUG] In vendor: github.com/envoyproxy/go-control-plane/envoy/type
[DEBUG] Package github.com/envoyproxy/go-control-plane/envoy/api/v2 imports github.com/envoyproxy/protoc-gen-validate/validate
[DEBUG] In vendor: github.com/envoyproxy/protoc-gen-validate/validate
[DEBUG] Marking github.com/envoyproxy/protoc-gen-validate/validate to be scanned.
^[[0;32m[INFO]  ^[[m--> Fetching updates for github.com/envoyproxy/protoc-gen-validate
[DEBUG] Package github.com/envoyproxy/go-control-plane/envoy/api/v2 imports github.com/golang/protobuf/proto
[DEBUG] In vendor: github.com/golang/protobuf/proto
[DEBUG] Marking github.com/golang/protobuf/proto to be scanned.
[DEBUG] Dependency github.com/golang/protobuf has already been pinned. Fetching updates skipped
^[[0;32m[INFO]  ^[[m--> Setting version for github.com/golang/protobuf to v1.1.0.
.....
.....
.....
[[0;31m[ERROR] ^[[mFailed to generate lock file: Generating lock produced conflicting versions of github.com/golang/protobuf. import (v1.1.0), testImport (v1.3.2)

Is it something because of this ? or is it a mistake on my part?

kyessenov · 2019-09-18T00:09:36Z

It could be, but I'm not an expert in the interplay between glide and go mod. If you have a suggestion, we can merge a fix here.

jyotimahapatra · 2019-09-19T23:44:42Z

@kyessenov You've been immensely helpful and we could finally migrate from go-control-plane v0.8.0 to v0.9.0.

kyessenov force-pushed the v2 branch from f4985a8 to 8ec1732 Compare September 9, 2019 17:52

kyessenov and others added 10 commits September 9, 2019 10:53

delete gogo stubs

6f7f570

Signed-off-by: Kuat Yessenov <kuat@google.com>

rebase to goproto

30b0b6a

Signed-off-by: Kuat Yessenov <kuat@google.com>

add warning

b55f559

Signed-off-by: Kuat Yessenov <kuat@google.com>

Update README.md

4f62216

Signed-off-by: Kuat Yessenov <kuat@google.com>

update README

16df96b

Signed-off-by: Kuat Yessenov <kuat@google.com>

small code re-org to fit best practices

7fdb3a3

Signed-off-by: Kuat Yessenov <kuat@google.com>

enable deterministic serialization

497b127

Signed-off-by: Kuat Yessenov <kuat@google.com>

Rename grpc imports (#225)

ce1cd06

Needed to be compatible with internal build rules. Signed-off-by: Teju Nareddy <nareddyt@google.com> Signed-off-by: Kuat Yessenov <kuat@google.com>

drop v2 suffix in the branch

0360445

Signed-off-by: Kuat Yessenov <kuat@google.com>

update README

f28b41f

Signed-off-by: Kuat Yessenov <kuat@google.com>

kyessenov force-pushed the v2 branch from 8ec1732 to f28b41f Compare September 9, 2019 17:54

kyessenov requested review from rshriram and yangminzhu September 9, 2019 17:56

rshriram reviewed Sep 10, 2019

View reviewed changes

rshriram approved these changes Sep 10, 2019

View reviewed changes

yangminzhu approved these changes Sep 10, 2019

View reviewed changes

kyessenov merged commit 52b167a into master Sep 10, 2019

kyessenov deleted the v2 branch September 10, 2019 22:06

nareddyt mentioned this pull request Sep 11, 2019

Proposal: Check-in generated golang protos in envoy/api envoyproxy/envoy#8151

Closed

This was referenced Feb 6, 2020

Makefile: Adding target go-test-coverage openservicemesh/osm#123

Merged

Alias go-control-plane imports in xds.go openservicemesh/osm#124

Merged

wozz mentioned this pull request May 26, 2020

Upgrade Protobuf Dependency bufbuild/protoc-gen-validate#338

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transition to use golang/protobuf to support automated code generation #226

Transition to use golang/protobuf to support automated code generation #226

kyessenov commented Sep 9, 2019

rshriram Sep 10, 2019

kyessenov Sep 10, 2019

kyessenov Sep 10, 2019

yangminzhu left a comment

kyessenov commented Sep 10, 2019

jyotimahapatra commented Sep 17, 2019 •

edited

Loading

kyessenov commented Sep 17, 2019

jyotimahapatra commented Sep 17, 2019 •

edited

Loading

kyessenov commented Sep 17, 2019

jyotimahapatra commented Sep 18, 2019 •

edited

Loading

kyessenov commented Sep 18, 2019

jyotimahapatra commented Sep 19, 2019

Transition to use golang/protobuf to support automated code generation #226

Transition to use golang/protobuf to support automated code generation #226

Conversation

kyessenov commented Sep 9, 2019

rshriram Sep 10, 2019

Choose a reason for hiding this comment

kyessenov Sep 10, 2019

Choose a reason for hiding this comment

kyessenov Sep 10, 2019

Choose a reason for hiding this comment

yangminzhu left a comment

Choose a reason for hiding this comment

kyessenov commented Sep 10, 2019

jyotimahapatra commented Sep 17, 2019 • edited Loading

kyessenov commented Sep 17, 2019

jyotimahapatra commented Sep 17, 2019 • edited Loading

kyessenov commented Sep 17, 2019

jyotimahapatra commented Sep 18, 2019 • edited Loading

kyessenov commented Sep 18, 2019

jyotimahapatra commented Sep 19, 2019

jyotimahapatra commented Sep 17, 2019 •

edited

Loading

jyotimahapatra commented Sep 17, 2019 •

edited

Loading

jyotimahapatra commented Sep 18, 2019 •

edited

Loading