-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add target and remove_field options to decode_json_field #3134
Comments
I would see the current behaviour even as a bug as this is probably going to break as soon as the template is loaded. We must definitively change this. Instead of allowing the flexibility to decide under which namespace the processed data should go, I suggest we put it under json as this namespace already exists. This allows us to define this already in the template and prevents most type conflicts. There could be type conflicts in case two different json documents have different types for the same fields, but I would argue this is more an issue in the log structure. In case someone wants to move the fields to a different place (and breaks templates with it) this can be done on the Logstash or Ingest side. If someone wants to remove the message field, this could already be done with the filters we have. I see that it is nice to have as a short cut for usability, but I'm hesitant to add a feature in two places. Also a user can use again ingest or LS to remove fields. In general we should be careful allowing users to modify the structure of events as this will make the events incompatible with the template and will lead to potential type / namespace conflicts with overwrites, which I prefer not to handle on the beats side. |
Hi @andrewkroh, i actually testing try to decode json message field. Here is my output
You see that message is format with Update: |
@andrewkroh, i'll tack back my comment. I see this coming from the multiline pattern. And json decode is not able to decode it properly. I assume this is what you meaning with
|
Like @ruflin says, i also think this is a bug. I have tested something
console output
It always doesn't decode the last line from the multiline pattern, or if you specify only on entry it doesn't decode. See my first commit. (is this another bug??). If i publish to elasticsearch the only record is the with |
I was trying to use the new
decode_json_field
processor to decode pretty multiline json messages (as in #1208). And without a configurable "target" it's kind of hard to use since the mapping for themessage
field is a string, but decode_json_field makesmessage
an object.So I propose we add a configuration options for
target
andremove_field
like Logstash has.For example:
filebeat.yml
input.json
output
The text was updated successfully, but these errors were encountered: