-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiline log guidance #100
Comments
Fluent Bit unfortunately does not yet have generic multiline logging support that can be used with FireLens. We are planning to work on it. For now, you must use Fluentd: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluentd/multiline-logs |
@zhonghui12 , @PettitWesley we are using firelens configuration with aws-for-fluent-bit for multi-destination log routing which includes cloudwatch as one of the sources. Multiline log grouping we need to make the most out of our logs. whether is any custom parser we can use to achieve it also fine. |
@belangovan there has been no change in guidance since my last comment on this issue. Fluent Bit still only have multiline support that works when tailing a log file. It does not have generic multiline support that works with FireLens. We are planning to work on that some time in the next few months. Until then, you have to use Fluentd for multiline. |
Is this feature you guys are planning on working soon? Just want to know on how to plan for our organization Do we switch to fluentd or wait. If we wait - how long |
@HaroonSaid We have begun investigation for this project. We hope to get it launched within 2 months, however, there are no guarantees. |
@PettitWesley Any updates re: whether this project is launching as intended? Debating whether we have to change an internal logging system to support fluentd or if we can wait for fluentbit multiline support to land. |
@corleyma The upstream maintainers are working on it apparently- I've been told that it should be ready/launched sometime in May. |
@PettitWesley May... this year? Any update on this? It'd be a very useful feature for us. |
@silvervest Yeah it was supposed to be May of this year. Progress has been made upstream but the launch is delayed till sometime in June. |
This is launching very soon: fluent/fluent-bit#337 (comment) |
Just to clarify, is the multi-line support now available for use in this image? Or are we still awaiting that implementation? |
Hi @aaronrl95, it was included in v2.18.0. |
Ah great, thank you. Could you point me to the documentation around implementing that feature in our firelens configuration? I'm struggling to find any |
You can follow this Firelens example. |
@hossain-rayhan thank you for that, that's just what I'm looking for |
@hossain-rayhan Does this solution also applicable for JSON format logs produced by Docker container ? |
@zhonghui12 or @PettitWesley can you answer this? |
I assume that if the JSON format logs are split into multiple lines, then it can be concatenated as there is no obvious limit here: https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace. But maybe @PettitWesley can give a more certain answer here. Or maybe we should help to test it out. |
@hossain-rayhan @zhonghui12 @PettitWesley hi guys, I've been trying to use multiline support to concat partial messages splitted by containerd (AWS Fargate), however it didn't work. I've been using approach described by @hossain-rayhan with the following config:
Could you please take a look, thanks! More details on my setup and what I'm trying to achieve: |
@StasKolodyuk you to create a custom multiline parser I think. I don't know exactly how to solve this use case with the new multiline support. I suspect with a custom parser with a custom regex it should be possible. https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace |
@vinaykrish25aws Yes the new filter will work with json logs from Docker. In that case, the log content is in the
If the content of that key is itself nested json that need to be recombined or something then that's a more complicated use case which might need custom parser and/or additional parsing steps. |
Hi, I have similar problem. |
@shijupaul Unfortunately, since this feature is new, we are still learning and understanding as well, and there are very few working examples that we have as well... so right now everyone is figuring it out. So actually, if you or anyone in this thread get a working example for a use case you think is decently common, please do share it. This will benefit the community. I'm also slowly working on slowly improving our FireLens/Fluent Bit FAQ/examples, and this data can be used for that. Can you share what these java stack traces look like? And I recommend that you (and everyone) test their own logs with the regular expressions that you write in the multiline parser using the rubular website: https://rubular.com/ If the regex's don't work there with your logs... then that's the problem. That should be your first debug step. |
Hello 👋 I thought I'd share my attempts as well here, as it might be useful to someone. I've been trying to get this to work for a couple days now as well, but so far without any luck. I have a pretty much identical setup as @shijupaul (I don't have the grep filter). I've playing around with these regexes quite a bit, but it doesn't seem to have any effect at all. Even if I put in a regex like To verify my hypothesis, I have been trying a couple of things:
I also ran it locally using
The interesting thing is that there I do see that it has as an effect. I can see how multiple log lines are combined. I have a couple of theories now:
Any tips or tricks are appreciated! In the meantime, I'll keep debugging |
@lbunschoten @PettitWesley This is what I experienced as well... Correct me if I'm wrong but I believe that the issue is the Source of the logs - Our images only get it as Forwarded messages from the emitter (https://github.com/aws/aws-for-fluent-bit/blob/mainline/fluent-bit.conf#L1-L4). This might make it pointless to try to concat it through the use of metadata (like CRI's logtag or Docker's partial_message) because those could be filtered out or not forwarded to us in the first place. That would match our experienced behavior here. |
@PettitWesley can you share how to configure buffer mode in fluent-bit with multiline support? I know the aws image already has it preconfigured. According to this thread and your link I should enabled buffer mode, but the official fluent bit documentation does not mention what are the valid values for buffer is it on/off or true/false?
Do you think this configuration will work? |
I have a very strange issue.
|
On/off/true/false case insensitive are all supported for Fluent Bit bool config values. Your configuration looks correct to me. Looks like my doc changes only got deployed for 1.9 pre release: https://docs.fluentbit.io/manual/v/1.9-pre/pipeline/filters/multiline-stacktrace @7nwwrkdnakht3 Can you please open a separate issue to troubleshoot your problem and please provide us with more details like the FB config. |
hi @PettitWesley I have succesfully tested with both images (the aws image and the fluent bit one) the multiline parser with ECS containers in fargate to send logs to OpenSearch but the downside is when I used a regular parser I was able to match fields I declared on my regex (i.e. time, source, level, uuid) and see them on OpenSearch this regular parser uses the same regex as my multiline parser on my comment above
results into
but now with multiline parsers I get this on OpenSearch
Do you know if matching is not enabled on multiline parsers or why my configuration can't match the fields I put on my regex |
@magg I think your use case is for parsing the log to split out the data into separate fields? That's a use case for the "norma'" parsers and the normal filter parser. I think this is not a use case for multiline- multiline is for joining multiple events into one event. Also, you can use both. You can have two filters. You can concat events together with multiline first, and then split out fields from the log using normal parser, or the reverse. |
@PettitWesley yes I get the idea, I'm able to see the full java stack trace thanks the to multiline line parser. But all my log lines including the first one that contain the stacktrace have the format Do you have any example of using both filters? I cannot find it anywhere :( |
So from what I understand it is currently not possible to use the input plugin tail with I'm using AWS EKS and I want to merge Java stack trace records into a single log record in CloudWatch, but I'm failing to do so.
|
Using two filters is just a matter of have two filter definitions like:
|
You should be able to use both the tail multiline functionality and also the filter. You shouldn't use the same multiline parser with each, use a different parser, but it should work. |
Yes! It works! Thanks a lot @PettitWesley, no idea what I did wrong last time I tried it. I will add my config here, maybe it can help someone in the future Logs before getting it to work:
Now I add the multiline filter to my config:
And it merges the stack traces. Here is a screenshot from CloudWatch: Relevant line from fluent-bit logs:
Edit: As a "micro-optimization" you can also set |
Note please see: fluent/fluent-bit#5235 If you have more than one multiline filter definition and they match the same records, it can cause all sorts of trouble. I am trying to figure out how to fix this. |
I want to combine docker logs. https://docs.fluentbit.io/manual/v/1.9-pre/pipeline/filters/multiline-stacktrace
Because of this caution, I want to write |
@tatsuo48 that statement only applies to tail, because tail gets the logs all at once in chunks read from the file and so it's most efficient to do the multiline processing there. For forward plugin, based on my understanding of the underlying code, there shouldn't really be much of a difference between implementing buffering and multiline concat directly in forward vs in my filter. The way filters actually work is that they are sort of like run in the same context as the input, they're sort of like extensions attached to an input. If that makes sense/helps. So please just use the filter :) |
I understand what you are saying. |
if you set "Path_Key" in input, can you still collect logs? |
Btw I'm having issues with multiline atm:
@James96315, I don't see why it shouldn't work. But I haven't tried it. |
@James96315 yea |
I use "amazon/aws-for-fluent-bit:2.24.0", if I add "path_key" in "[INPUT]", it can't work. I got some info from fluent bit log:
Here is my config:
The sample log is as follows:
If the parsing is correct, the number of output log records should be 2, but it has not been successful |
@James96315 Thanks for the report, just to be clear, when you remove Path_Key, you see these records concatenated? Can you share an example of concatenated/unconcatenated results please. |
2.Even if I set the buffer to False and remove the Path_Key, the log parsing is not right. The number of parsed records is sometimes 3, sometimes 5, which seem to be random. The logs keep getting split and not parsed correctly.
|
Is it possible to pass first through the multiline filter and then through the rewrite_tag filter?
Fluentbit version 2.24.0
|
@clouddev-code https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#rewrite_tag-filter-and-cycles-in-the-log-pipeline Rewrite tag moves data to head of the pipeline, so it moves through the filter again with new tag. |
Hi, I'm looking for a guide of multiline for eks fargate logging. Here is the official documentation . Is it possible to apply the mutiline parser in the built-in log router, I checked the document, and it's not allowed to add a multiline filter in the filter.config.
|
@TWCM Supporting custom multiline parsers on EKS Fargate would be a feature request, can you please open a request here for that: https://github.com/aws/containers-roadmap/issues And @ mention me in it. |
I'm going to close this issue as it is very old and the full multiline support was launched last year. Please open a new issue for new multiple issues or requests. |
@PettitWesley
|
We have the following configuration
We want to have multiline logs for stack trace etc.
How should I configure
fluentbit
The text was updated successfully, but these errors were encountered: