Can JsonTemplateLayout output JSON boolean values?

[aabmass]

I am using the JsonTemplateLayout with the pattern resolver and I'm trying to output a boolean value, specifically something like {"logging.googleapis.com/trace_sampled": true, ...}. It looks like the PatternResolver can only write strings to the JSON stream, but I wanted to check if there is another solution before opening a feature request.

Some additional context:

• I am using the OpenTelemetry Java auto instrumentation agent which automatically adds the W3C traceflags to the MDC key trace_flags as a hex string.
• This hex string can be either 00 (trace is not sampled) or 01 (trace is sampled).
• The GCP Cloud Logging agent parses JSON And expects the key logging.googleapis.com/trace_sampled to have a boolean value.

My current config outputs "true" or "false" instead of booleans:

<JsonTemplateLayout eventTemplateUri="classpath:GcpLayout.json">
  <EventTemplateAdditionalField
    key="logging.googleapis.com/trace_sampled"
    format="JSON"
    value='{"$resolver": "pattern", "pattern": "%equals{%equals{%mdc{trace_flags}}{01}{true}}{00}{false}"}'
  />
</JsonTemplateLayout>

Thanks in advance!

[vy]

@aabmass, I don't think this is possible using PatternResolver – see its resolve() method uses JsonWriter#writeString(), which effectively will encode the PatternLayout output. There are a couple of approaches I can think of to address your use case:

Create your own event template resolver

✓ Complexity: 1/10
✓ You will have the full control
✓ Doesn't require a change on the Log4j side
❌ You need to deliver Java code next to the Log4j configuration

This is something you can easily do yourself.

Add a flag to PatternResolver to inject PatternLayout output without encoding it

✓ Complexity: 1/10
❌ Requires a change on the Log4j side

I am not a big fan of PatternResolver in general. It literally communicates "JsonTemplateLayout falls short on this feature, let me delegate this to PatternLayout" and this doesn't sound good at many levels. PL is not designed to produce JSON-safe strings. Though due to its age, PL ships several utility directives JTL lack. I am more inclined to learn from PL and port proven utilities to JTL instead. This brings me to my next suggestion:

Add a new ScriptResolver

✓ Users will have the complete programmatic control in the Log4j configuration file
❌ Complexity: 6/10
❌ Requires a change on the Log4j side

We can provide MVEL, Groovy, Kotlin, JavaScript, etc. support out of the box. If we can get this "right" (read as "safe and easy to use"), I bet this will address 90% of any potential JTL feature request.

Add a new TypeConverterResolver

✓ Complexity: 2/10
❌ Requires a change on the Log4j side

We can add a new resolver that can convert from String-to-Number and vice versa. These sort of utility methods always remind me of Greespun's tenth rule:

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

[aabmass]

Thanks for the thorough response @vy!

Create your own event template resolver

I was hoping to submit a PR to the JTL included GcpLayout.json to set this field, along with support for logging.googleapis.com/trace and logging.googleapis.com/spanId which don't require any munging. I guess this would boil down to including a new one-off event template resolver in that PR. Idk if that sounds good to you.

Add a flag to PatternResolver to inject PatternLayout output without encoding it

I kind of like this approach but understand your hesitance. It would be a nice escape hatch. One other prior art design that wouldn't require any flags–the logstash logback encoder supports #asBoolean{} in pattern strings.

This isn't a high priority thing for GCP support btw, the logging.googleapis.com/trace_sampled is more of a nice-to-have feature. How would you like me to proceed here? Should we wait to see if this is a common ask? I am happy to send a PR for one of the simpler options, but ScriptResolver sounds like a bigger undertaking.

[vy]

including a new one-off event template resolver

Sorry, I can't do that. I think we should not add a resolver just for GCP Layout.

Add a flag to PatternResolver to inject PatternLayout output without encoding it

I just noticed this won't cut it. Consider the following:

{
  "$resolver": "pattern",
  "pattern": "%equals{%equals{%mdc{trace_flags}}{01}{true}}{00}{false}",
  "encoded": false
}

What if trace_flags contains neither 01, nor 00? Then, depending on the value of trace_flags MDC entry, the JSON structure can be totally corrupted.

This isn't a high priority thing for GCP support btw, the logging.googleapis.com/trace_sampled is more of a nice-to-have feature. How would you like me to proceed here? Should we wait to see if this is a common ask? I am happy to send a PR for one of the simpler options, but ScriptResolver sounds like a bigger undertaking.

The fact that delegating to PatternLayout will not produce a bullet-proof solution disqualifies the following proposal: Add a flag to PatternResolver to inject PatternLayout output without encoding it

Checking logstash-logback-encoder you cited... It uses Logback JaninoEventEvaluator, which uses Janino under the hood. That is, for conditional branching, we need scripting.

Checking asBoolean() of logstash-logback-encoder... We can implement something similar using TypeConverterResolver BooleanResolver I suggested earlier:

{
  "$resolver": "booleanResolver",
  "matcher": {
    "values": [ "00" ],
    "regexes": [ "\\s*00\\s*" ]
  }
  "target": true,
  "source": {
    "$resolver": "mdc",
    "key": "trace_flags",
    "stringified": true
  }
}

Note that above either match.values or matcher.regexes would suffice, and target is true by default. I provided all these knobs for demonstration purposes.

I can accept a PR implementing the BooleanResolver. (You can check CounterResolver for inspiration.) If you would like to go that route, let me know. I prefer to assist you a bit first rather than reviewing a fully-blown PR without any prior discussion.

Using filters

A very ugly hack you can implement is to

1. Repeat your <Appender block twice with different JTL event templates – one for if (trace_flags == 01), and another one for else
2. Use RoutingAppender to switch between the two

[aabmass]

Sorry, I can't do that. I think we should not add a resolver just for GCP Layout.

No worries 👍

What if trace_flags contains neither 01, nor 00? Then, depending on the value of trace_flags MDC entry, the JSON structure can be totally corrupted.

Yes it's definitely a bit of a hack. The pattern string solution also has the issue of forward compatibility. Right now, trace flags can only be 00 or 01 but it is supposed to be a bitset with the first bit alone being the sampled flag so 03 should also become true.

I can accept a PR implementing the BooleanResolver. (You can check CounterResolver for inspiration.) If you would like to go that route, let me know. I prefer to assist you a bit first rather than reviewing a fully-blown PR without any prior discussion.

The more I think about it, ideally GCP would just accept the hex string as is. I'm trying to push on that internally. If I can't find another way forward, I agree BooleanResolver would be a good short term fix, and ScriptResolver would be a really nice solution.

Thanks again, I really appreciate you taking the time to answer my questions!