-
Notifications
You must be signed in to change notification settings - Fork 15.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Report use of unknown fields #5967
Comments
What are the inputs and return value for the callback you would like to proposal? I think it's some kind of hard to just providing one callback to fully support all users concerns. |
It would really depend on what is most idiomatic for protobuf, but the effect that we want is for C++ code calling into functions like Also CC @lizan |
JsonStringToMessage has an option for ignore_unknown_fields. If you set it to false for (1) and set it to true for (2), would that give you what you need? The parse failure message should inform you approximately where the error occurs in JSON. |
We need to programatically obtain a list of unknown fields to use for more detailed logging and stats. We want to accept the unknown fields, but provide an indicator in our own access logs of all accepted unknown fields, and scraping error strings for this seems a bit fragile. |
The converter::ErrorListener is for this purpose, though it is in internal header so ideally we want to make it as a part of public API. |
What kind of information would you like to log? Do you need to know the JSON key that is unknown + the full hierarchy to get there? Do you need to know the line number + column number of that unknown field? Random idea: if I give you a traverser that can traverse the entire JSON, and for each JSON field that I visit, I'll give invoke your visitor with the JSON key + JSON value + the corresponding FieldDescriptor for that JSON (or absent of FieldDescriptor for unknown field), would that be sufficient for your logging/stats purpose? |
I think we just want to know the field names for JSON and the field number for proto, that should be sufficient for logging. The idea you present would definitely work for JSON, is there a way of also tackling this for proto? |
For regular proto message, you can do access unknown fields using: |
@haon4 ack; in that case, what you suggest in #5967 (comment) sounds great. Do you think this can be added to the protobuf C++ libs? |
I had a chat with Yilun about this. We have some ideas, but not enough free cycles :( Once Yilun finishes up his current project he might be able to look designing this. |
Great. Meanwhile, we will add support in Envoy for handling the proto3 ingestion path using reflection and then we can consume the JSON support for that path as it becomes available. |
We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment. This issue is labeled |
We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it. This issue was closed and archived because there has been no new activity in the 14 days since the |
What language does this apply to?
C++ (possibly others)
Describe the problem you are trying to solve.
We would like to allow unknown fields during proto parsing from wire (and formats such as JSON/YAML), but to report the details for the purposes of logging. Our use case is Envoy (https://github.com/envoyproxy/envoy), where configuration with unknown fields can fall into two categories:
Intentional delivery of a newer field from a server with a newer API proto than the client. Client should just ignore.
Accidental misconfiguration of protos (typically the JSON/YAML representation, but also text proto) by users.
We would be confident enabling unknown fields if we could track their use to support (1) and still detect (2).
Describe the solution you'd like
A callback during proto parsing when an unknown field is encountered.
Describe alternatives you've considered
Today, we only allow a binary choice by the user. We encourage them to allow unknown fields if the protos are machine generated and to disallow them if not. This is often too coarse, with a mixture of configuration proto provenance not uncommon.
CC @mattklein123
The text was updated successfully, but these errors were encountered: