proto: Generate faster code using vtprotobuf #8173
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Vicent Marti <vmg@strn.cat>
Signed-off-by: Vicent Marti <vmg@strn.cat>
Signed-off-by: Vicent Marti <vmg@strn.cat>
|
I've updated the repo for the new protobuf compiler with a README and examples: https://github.com/planetscale/vtprotobuf |
systay
reviewed
May 25, 2021
Signed-off-by: Vicent Marti <vmg@strn.cat>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Following up on last weeks upgrade to the ProtoBuf APIv2 and the newest version of GRPC, observing minds will have probably noticed a significant regression on CPU usage and performance during RPC since the upgrade.
This performance regression has multiple sources, but the most notable of them is the removal of the
gogo/protobufgenerated code that we were using with the ProtoBuf APIv1. In #7519, I replaced the serialization and de-serialization code in Vitess with code autogenerated by the Gogo compiler; this was a significant performance improvement with very little cost, although the change itself was contentious: the Gogo compiler has been abandoned and it's looking for new ownership, so adding it as a dependency to Vitess was a questionable choice, particularly since as soon as we upgraded to ProtoBuf API v2, the generated Gogo compiler was no longer usable.Lo and behold, a couple months later, and we've upgraded to ProtoBuf APIv2, with the consequent performance regression because the autogenerated code from
gogo/protobufis no longer usable. So what do we do about this?Introducing
protoc-gen-go-vtprotoWe're replacing the now-deprecated
gogo/protobufcompiler with a proper implementation that has been written from scratch for ProtoBuf APIv2. The new compiler, available at https://github.com/planetscale/vtprotobuf, has been designed as a helper generator that is run alongside the upstreamprotoc-gen-go, instead of a replacement, and it focuses on generating high-performance (un)marshalling code -- which is the core feature that Vitess used from the old compiler.There are still more improvements on the pipeline for the new compiler, including, huh, a README, but for now the generated code fully passes both the required and optional conformance suites for ProtoBuf and brings back most of the performance we've missed since the upgrade.
Let's see how it behaves in practice!
Related Issue(s)
Checklist
Deployment Notes