Using custom pattern converters for JSON template layout

[kf-ping]

We have an application that uses log4j's Pattern Layout and have created custom Pattern Converters to handle certain use cases (e.g. logging HTTP headers, parameters, etc.). I'm trying to add support for JSON Template Layout to the application and wish to support the same logging features available with our custom Pattern Converters.

I know log4j JSON Template Layout supports Pattern resolver as an escape-hatch solution. However, I'd like to know if there's a recommended way to do this without using the Pattern resolver? Will I need to write an equivalent custom Event Resolver that does the same thing as the existing custom Pattern Converters?

Thanks for any suggestions.

[vy]

@kf-ping, thanks for reaching out to us. It always helps us a lot to learn how users integrate Log4j into their systems.

We have an application that uses log4j's Pattern Layout and have created custom Pattern Converters to handle certain use cases (e.g. logging HTTP headers, parameters, etc.)

Curious, would you mind providing a little bit more detail about these converters, please? Some examples of what they inject will help a lot. This will also help us to understand the type these converters will preferably output in a structured layout such as JTL.

Will I need to write an equivalent custom Event Resolver that does the same thing as the existing custom Pattern Converters?

Well, it depends. If your custom converters...

1. ... produce String-typed output, you can include them in your JTL templates using the pattern resolver.
2. ... produce not-String-typed (i.e., numbers, arrays, and/or objects) output,
   or are performance sensitive and you want to take advantage of JTL's recycling capabilities,
   then you indeed need to implement custom JTL event template resolvers. You can refactor your code base to make both JTL event template resolvers and PL converters share the same business logic, instead of duplicating it.

[kf-ping]

@vy thanks for the response

Curious, would you mind providing a little bit more detail about these converters, please? Some examples of what they inject will help a lot. This will also help us to understand the type these converters will preferably output in a structured layout such as JTL

The custom PL converters extend LogEventPatternConverter and overrides the public void format(LogEvent event, StringBuilder toAppendTo) method. It appends to the toAppendTo with things like the HTTP header, parameters, etc. for logging purposes. The application uses Log4j to output these values into a log file and currently supports only PL. I'm adding support for JTL.

1. ... produce String-typed output, you can include them in your JTL templates using the pattern resolver.

Since we are just outputting string values I was thinking of just using the pattern resolvers in JTL.
Are there any drawbacks to using this approach relying on pattern resolvers? I noticed the note below in the Log4j docs and is why I posted this question:

This resolver is mostly intended as an emergency lever when all other JSON Template Layout resolvers fall short of addressing your need. If you find yourself using this, it is highly likely you are either doing something wrong, or JSON Template Layout needs some improvement. In either case, you are advised to share your use case with maintainers in one of our support channels.

[vy]

[Custom converters] appends to the toAppendTo with things like the HTTP header, parameters, etc. for logging purposes. The application uses Log4j to output these values into a log file and currently supports only PL. I'm adding support for JTL.
...
Since we are just outputting string values I was thinking of just using the pattern resolvers in JTL. Are there any drawbacks to using this approach relying on pattern resolvers?

PL doesn't have a structure, hence in toAppendTo, I presume, you are just composing a free text, e.g.:

buffer.append("HttpHeaders{");
for (int i = 0; i < httpHeaders.size(); i++) {
    if (i > 0) { buffer.append(", "); }
    var header = httpHeaders.get(i);
    buffer.append("\"%s\"=\"%s\", header.getName(), header.getValue());
}
buffer.append("}");

With JTL, you wouldn't want to do like this. That is, the whole point of JTL is delivering a structured layout – by collapsing individual values into a single string you are counteracting to this goal. Instead, generate a JSON object containing HTTP header key-value pairs. I can think of two ways:

1. Inject these values to the thread context and consume them using %X in PL and mdc in JTL
2. Write custom PL converters (emitting strings) and JTL resolvers (emitting JSON objects)

My advice would be going route #1. Curious: Why is leveraging the thread context not an option for you?

I noticed the note below in the Log4j docs and is why I posted this question:

This resolver is mostly intended as an emergency lever ...

Yes, I am aware that you do your due diligence, and I am very grateful for that! 🙇

[kf-ping]

Inject these values to the thread context and consume them using %X in PL and mdc in JTL ... Curious: Why is leveraging the thread context not an option for you?

I believe for custom Pattern Converters you rely on the format(LogEvent event, StringBuilder toAppendTo) method, so it did not occur to me to use ThreadContext, but that is a good idea. I assume you mean something like this?

@ConverterKeys({"header"})
public class HttpHeaderPatternConverter extends LogEventPatternConverter
{
    format(LogEvent event, StringBuilder toAppendTo)
    {
        toAppendTo.append("HttpHeaders: ");
        String httpHeaderValues =  ... process HTTP headers into String ... 
        toAppendTo.append(httpHeaderValues);

        ThreadContext.put("httpHeaders", httpHeaderValues);
    }
}

Can I achieve structured JSON using multiple pattern resolvers in my JSON template?
For example, with multiple custom Pattern Converters (e.g. "header" and "parameters"), I can use a pattern like this:

<PatternLayout>
    <pattern>%d | %header{Content-Type} | %header{User-Agent} | %parameters{param1} </pattern>
</PatternLayout>

I can refer to these Pattern Converters in my JSON template like this correct?

{
  "timestamp": {
    "$resolver": "timestamp",
    "pattern": {
      "format": "yyyy-MM-dd'T'HH:mm:ss.SSSXX"
    }
  },
  "headerContentType": {
    "$resolver": "pattern",
    "pattern": "%header{Content-Type}"
  },
  "headerContentType": {
    "$resolver": "pattern",
    "pattern": "%header{User-Agent}"
  },
  "headerContentType": {
    "$resolver": "pattern",
    "pattern": "%parameters{param1}"
  }
}

I appreciate your insights so far on this topic, thanks!

[vy]

I believe for custom Pattern Converters you rely on the format(LogEvent event, StringBuilder toAppendTo) method, so it did not occur to me to use ThreadContext, but that is a good idea. I assume you mean something like this?

@ConverterKeys({"header"})
public class HttpHeaderPatternConverter extends LogEventPatternConverter
{
    format(LogEvent event, StringBuilder toAppendTo)
    {
        toAppendTo.append("HttpHeaders: ");
        String httpHeaderValues =  ... process HTTP headers into String ... 
        toAppendTo.append(httpHeaderValues);

        ThreadContext.put("httpHeaders", httpHeaderValues);
    }
}

No. You better implement your custom thread context data provider.

Can I achieve structured JSON using multiple pattern resolvers in my JSON template? For example, with multiple custom Pattern Converters (e.g. "header" and "parameters"), I can use a pattern like this:

<PatternLayout>
    <pattern>%d | %header{Content-Type} | %header{User-Agent} | %parameters{param1} </pattern>
</PatternLayout>

I can refer to these Pattern Converters in my JSON template like this correct?

{
  "timestamp": {
    "$resolver": "timestamp",
    "pattern": {
      "format": "yyyy-MM-dd'T'HH:mm:ss.SSSXX"
    }
  },
  "headerContentType": {
    "$resolver": "pattern",
    "pattern": "%header{Content-Type}"
  },
  "headerContentType": {
    "$resolver": "pattern",
    "pattern": "%header{User-Agent}"
  },
  "headerContentType": {
    "$resolver": "pattern",
    "pattern": "%parameters{param1}"
  }
}

Yes, this also is a valid approach. That said, my preference is still to leverage the thread context.

[kf-ping]

Thanks! I think I got a good idea of what works for our use case. I appreciate your help.