-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kubernetes Pods Logging (escaped nested) JSON to Std Out Isn't Correctly Parsed #691
Comments
I have tested this with the new 0.13.5 release and found the same issue. Log field in Kibana is displayed as follows for the following configurations: Config 1:
In Kibana: Config 2:
In Kibana: Config 3:
In Kibana: I'm continuing to experiment on my end with alternate approaches to find a work around but would greatly appreciate some assistance. I hope I'm not the only person trying to parse nested escaped JSON logs. |
So, it seems that without this getting fixed, using Fluent Bit with Kubernetes is pretty much useless. It's surprising that basic use cases have not been considered and are not part of the tests... |
@nikolay Yes, you are right. Tried with multiple possibilities but no luck. |
@nikolay Pull requests are always welcome. However, I have noticed your pattern of feedback is often not particularly helpful or necessary |
@StevenACoffman Your ability to do pattern recognition is sub-par. I no longer program in archaic and unsafe programming languages - I did C/C++ in the 90s. Anyway, in 2018, I may consider doing a Rust-based alternative. I already work on a pull request to improve the Docker build. And last, but not least: contribution to Open Source projects is not only in the form of pull requests! My frustration was only about a missing disclaimer - I didn't say "false advertisement"! |
@nikolay Most open source Maintainers appreciate feedback, so I'm suggesting you might try expressing your frustration or disappointment in more constructive ways. Pull requests are only one (highly recommended) possibility. Perhaps just being more careful that your comments are more constructive or polite.
I enjoy Rust as well! It plays nice with C, and if you extended fluent-bit in some way using it, I'm sure the maintainers would appreciate hearing about it. However, your choice of calling the maintainers choice of C as archaic and unsafe does not pass the T.H.I.N.K. test. I would appreciate it if you refrained from that kind of stuff. |
@StevenACoffman I appreciate this model of the T.H.I.N.K. test, I'd like to instate it in our own internal guidelines. |
@StevenACoffman My point was that I personally am not going back to C just to make Fluent Bit handle basic use cases. My other point was that the Kubernetes world doesn't need a fancy log shipper - it needs something really basic, which is well-suited for its use cases. For example, the inability to reload configuration, the centralized parsers definition, the inability to have wild cards for parser files, and others are not well-suited for Kubernetes features or limitations. The documentation is also outdated and incomplete. Anyway, I am doing everything to my abilities to help the project, but you cannot expect all people to submit pull requests! Anyway, can you explain to me how your THINK test applies to your handling of a very simple PR (#692) that's been ignored for almost two weeks, and which obviously fixes a very obvious bug in Fluent Bit? |
@nikolay utf-8 decoder patch will be reviewed after 0.14 release. |
JSMN in strict mode cannot be used to validate a JSON string without filing the tokens, it behavior is unexpected, more details about this in the following jsmn pull request: zserge/jsmn#119 (comment) this patch remove the flb_pack_json_valid() function. Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
@therealdwright thanks for providing specific details to reproduce the issue. the fix is already in place in GIT master and will be backported to 0.13 branch for 0.13.7 release: with the above fixes the following config works as expected:
more detail about the root cause of the problem with jsmn project read here: zserge/jsmn#119 (comment) |
JSMN in strict mode cannot be used to validate a JSON string without filing the tokens, it behavior is unexpected, more details about this in the following jsmn pull request: zserge/jsmn#119 (comment) this patch remove the flb_pack_json_valid() function. Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
@nikolay I used "default" config, but i changed escaped to escaped_utf8:
And logs look good to me |
are we ok to close this ticket? |
#692 is merged, so we should be good. Fixed |
* source: add Github links Signed-off-by: Patrick Stephens <pat@calyptia.com> * source: switch to Github Signed-off-by: Patrick Stephens <pat@calyptia.com>
Bug Report
Describe the bug
When running a docker container inside a Kubernetes cluster where the container already outputs JSON lines, it is not possible to parse the nested JSON in a way that Elasticsearch will correctly index the key:value pairs. Running the Docker parser over the log using the escaped json decoder correctly parses everything in the log except for the original JSON message.
If this is a misconfiguration on my end, assistance would be greatly appreciated, however I think I have run in to a limitation that if the original log is JSON which results in escaped JSON with a nested escaped JSON object, there appears to be no way to solve this in fluent-bit.
To Reproduce
kubectl run echo-chamber --quiet --image=quay.io/therealdwright/echo-chamber:latest
kubectl logs -f deployments/echo-chamber
and see the logs that will be output as follows:Expected behavior
I expect the logs to be parsed in a way that Elasticsearch is able to index the key "Message" in a way that can be queried by Lucene.
Screenshots
Your Environment
fluent-bit input config:
docker parser config:
Additional context
I have tried many variants of configuration based on the GitHub issues I could see, I also have looked through the source and it looks like only a single "Decode_Field_As" match can be applied to a single log.
Additionally when running the Merge_JSON_Log on the Kubernetes filter, I see many errors: "could not merge JSON log as requested" in the fluentbit container.
An example raw log sent to logz.io (retrieved looking at the JSON payload) is as follows:
The text was updated successfully, but these errors were encountered: