Reduce stdout log volume

[tillrohrmann]

Current ideas:

• Introduce a mechanism to log per-request spans only on debug mode, but keeping them at info level for tracing.
• Mark per-request log events via an attribute that the tracing subscriber subscribes to and that the stdout logger can opt-in to.

[slinkydeveloper]

So I did some investigation on the subject. In theory, given this bug is fixed, one could easily build filters such as:

• restate[{restate.invocation.sid}]=warn,restate=info: enables >= warn logging for all logs related to specific invocations, but keeps all the other restate logs at info level.
• restate[{restate.invocation.sid}]=off,restate=info: completely disables logs related to specific invocations.

I would like to possibly move ahead with this solution as it's far more flexible than adding any custom built flag ourselves (plus i'm not even sure such a solution would work given what i read on that issue), and we can easily provide some examples to describe how to set this up. Problem is, we need this fix :) tokio-rs/tracing#2095

[tillrohrmann]

Any ETA on the fix for the bug?

What would be the default log filter then?

[slinkydeveloper]

Any ETA on the fix for the bug?

Can't say. We can use a fork in the meantime i guess.

What would be the default log filter then?

Not sure, let's see when we get to it. Perhaps by default we do restate[{restate.invocation.sid}]=warn,restate=info,warn and then specify in the log lines how to improve it.

[tillrohrmann]

Ok, sounds good to me. I guess the important bit is to properly document how to configure the log filter to get the desired behavior.