Add option to log to file

[nightkr]

Description

Fixes #795

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

Preview Give feedback

1. Integration tests passed (for non trivial changes)

Reviewer

Preview Give feedback

1. Code contains useful comments
2. Changelog updated
3. Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Preview Give feedback

1. Feature Tracker has been updated
2. Proper release label has been added

[nightkr]

One downside here is that tracing-appender rotates based on time rather than disk space, so we can no longer provide strict guarantees that the logs will never outgrow their respective volumes... There are PRs up for addressing this already (see tokio-rs/tracing#2972: tokio-rs/tracing#2904, tokio-rs/tracing#2497) but none that have been merged yet.

[sbernauer]

Well that's a pity :/

[sbernauer]

Only looked at the Rust code

[sbernauer]

@siegfriedweber would you be willing to review this PR? I think you are the best person to do so, as you can best asses the so we can no longer provide strict guarantees that the logs will never outgrow their respective volumes problem. Maybe this is a blocker (given customers already had problems with the volume sizes), maybe not.

Also you know the vector transformation far better

[siegfriedweber]

Only minor things. The transform rule is implemented very well.

[lfrancke]

I just browsed through the comments here but: No rotation by size is a pretty big blocker for me. Is that really the case?

[nightkr]

Currently, yes. :/

Looks like tokio-rs/tracing#2497 is seeing some movement again, but it's still ultimately blocked on upstream tracing actually merging it. Or we could fork it ourselves, technically (sliiightly less of a mess than h2, since this is a direct dependency rather than a transitive one, but still).

[lfrancke]

Hmmm....this is only for operators though, correct?

I was thinking of stackabletech/issues#574 but for the operators I'm way less worried. Still not great.

[siegfriedweber]

LGTM if no one vetoes it because of size-based rolling.

[lfrancke]

No veto from me

[nightkr]

Hmmm....this is only for operators though, correct?

Operators and sidecars (like the OPA bundle builder).

[sbernauer]

I also wouldn't veto this, but have a somewhat bad feeling about this. The log rate of the bunde-builder is not that high, so it should be fine.

My bad feeling can be much improved by having it running for some hours (e.g. in the e2e-security-demo), collect some basic usage numbers and do some napkin math if the emptyDir is big enough ;)

[nightkr]

Generally, bundle builder's log rate depends primarily on how often the regorule configmaps are modified. It should be very quiet for a demo, but I'm not sure how well that correlates to practical user environments.

[nightkr]

Well in practice I'd expect regorules to be fairly static.. After about a minute's hammering (constant writes every ~100ms) the logs grew to ~400KB. Extrapolating from that into the six allowed hourly files, that's ~144MB.

Not great, but also hopefully not a super realistic scenario.

That said, I agree that, long term, returning to size rotation would be ideal.

[nightkr]

Created #828 for rotation followup.