Disabling Attributes logging not working

[AntonioLyubchev]

Describe the bug
This relates to #6772 and the setup/environment is the same

Having disabled all additional parameters under logback-appender:

otel:
  exporter:
    otlp:
      endpoint: ${OTLP_ENDPOINT}
      protocol: ${OTLP_PROTOCOL}
  metrics:
    exporter: "otlp"
  traces:
    exporter: "otlp"
  propagators: "tracecontext"
  resource:
    attributes:
      environment: ${APP_ENV}
  instrumentation:
    logback-appender:
      experimental-log-attributes: ${LOGBACK_EXPERIMENTAL_LOG_ATTRIBUTES}
      experimental:
        capture-mdc-attributes: false
        capture-code-attributes: false
        capture-marker-attribute: false
        capture-logger-context-attributes: false
        capture-key-value-pair-attributes: false

and explicitly excluded "log.body.parameters,log.body.template":

otel:
  experimental:
    resource:
      disabled:
        keys: "telemetry.sdk.language,telemetry.sdk.name,telemetry.sdk.version,log.body.parameters,log.body.template"

After logging a message:

log.info("Service B about to call HelloService to do work that {}.", Boolean.TRUE.equals(fail) ? "fails" : "succeeds");

I end up with the following in Elastic:

I expect that:
Attributes.log.body.parameters
Attributes.log.body.template
would not be present in the structured JSON log.

Additional context
Full config yaml:

# Simulated environment variable injection
APP_ENV: dev
LOGBACK_EXPERIMENTAL_LOG_ATTRIBUTES: false # Enable the capture of experimental log attributes thread.name and thread.id
LOGBACK_CAPTURE_MDC_ATTRIBUTES: "*" # Comma separated list of MDC attributes to capture. Use the wildcard character * to capture all attributes.
LOGBACK_CAPTURE_CODE_ATTRIBUTES: false # Enable the capture of source code attributes. Note that capturing source code attributes at logging sites might add a performance overhead.
LOGBACK_CAPTURE_MARKER_ATTRIBUTE: false # Enable the capture of Logback markers as attributes.
LOGBACK_CAPTURE_LOGGER_CONTEXT_ATTRIBUTES: false  # Enable the capture of Logback logger context properties as attributes. (This feature adds MDC properties as top level attributes instead of nested in _source)
LOGBACK_CAPTURE_KEY_VALUE_PAIR_ATTRIBUTES: false  # Enable the capture of Logback key value pairs as attributes.
OTLP_PROTOCOL: grpc
OTLP_ENDPOINT: http://localhost:4317

spring:
  application:
    name: PoC-OpenTelemetry-REST-KAFKA

logging:
  include-application-name: false
  level:
    io.opentelemetry: DEBUG

# NOTE: Always use comma separated lists for multiple values, this is how otel SDKs expect them
otel:
  exporter:
    otlp:
      endpoint: ${OTLP_ENDPOINT}
      protocol: ${OTLP_PROTOCOL}
  metrics:
    exporter: "otlp"
  traces:
    exporter: "otlp"
  propagators: "tracecontext"
  resource:
    attributes:
      environment: ${APP_ENV}
  instrumentation:
    logback-appender:
      experimental-log-attributes: ${LOGBACK_EXPERIMENTAL_LOG_ATTRIBUTES}
      experimental:
        capture-mdc-attributes: ${LOGBACK_CAPTURE_MDC_ATTRIBUTES}
        capture-code-attributes: ${LOGBACK_CAPTURE_CODE_ATTRIBUTES}
        capture-marker-attribute: ${LOGBACK_CAPTURE_MARKER_ATTRIBUTE}
        capture-logger-context-attributes: ${LOGBACK_CAPTURE_LOGGER_CONTEXT_ATTRIBUTES}
        capture-key-value-pair-attributes: ${LOGBACK_CAPTURE_KEY_VALUE_PAIR_ATTRIBUTES}
  java:
    enabled:
      resource:
        providers: "io.opentelemetry.sdk.autoconfigure.internal.EnvironmentResourceProvider"
  experimental:
    resource:
      disabled:
        keys: "telemetry.sdk.language,telemetry.sdk.name,telemetry.sdk.version,log.body.parameters,log.body.template"
  service:
    name: ${spring.application.name} # absolutely MANDATORY for every service so telemetry data can be easily associated with the correct service through visual tools

Class producing the log:

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequiredArgsConstructor
@Slf4j
public class HelloController {

    private final HelloService helloService;

    @GetMapping("/hello")
    public String helloGet(@RequestParam(value = "fail", defaultValue = "false") Boolean fail) {

        log.info("Service B about to call HelloService to do work that {}.", Boolean.TRUE.equals(fail) ? "fails" : "succeeds");
        helloService.doServiceWorkSpanAnnotation(fail);

        return "Hello from Service B!";
    }
}

[jack-berg]

Transferring to opentelemetry-java-instrumentation which is where the source code lives for the logback appender.

[trask]

hi @AntonioLyubchev!

log.body.parameters and log.body.template shouldn't be captured by default (see #11865).

can you try with normal configuration (not the new experimental yaml configuration)? thanks

[AntonioLyubchev]

hi @AntonioLyubchev!

log.body.parameters and log.body.template shouldn't be captured by default (see #11865).

can you try with normal configuration (not the new experimental yaml configuration)? thanks

Hi!

I completely removed the experimental logback config from the yaml and it seems that I am still getting these log attributes:

EDIT:

Ah.. it seems it is enabled by default

I will try to overwrite it but it seems it's not working. At first glance, it seems that overwriting is not implemented at all

addOpenTelemetryAppender() calls initializeOpenTelemetryAppenderFromProperties() which does not touch at all captureArguments state of OpenTelemetryAppender so it remains true

If I am on the right track, I can try to create a PR :)

[jeanbisutti]

The Java agent disables the argument capture by default:

opentelemetry-java-instrumentation/instrumentation/logback/logback-appender-1.0/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/logback/appender/v1_0/LogbackSingletons.java

Line 41 in 8a10097

"otel.instrumentation.logback-appender.experimental.capture-arguments", false);

I would say that the OpenTelemetry starter should have the same behavior as the Java agent.

The documention of the OpenTelemetry Logback appender says that the argument capture is disabled by default (and is not consistent with the implementation): https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/logback/logback-appender-1.0/library

[jeanbisutti]

initializeOpenTelemetryAppenderFromProperties() which does not touch at all captureArguments state of OpenTelemetryAppender so it remains true

@AntonioLyubchev

initializeOpenTelemetryAppenderFromProperties() is to be able to initialize the Logback instrumentation of the OTel starter with properties. It overrides the configuration defined in a Logback XML file. The override is not implemented for the argument capture. It should be.

As a workaround, you could disable the capture of the log attributes from a Logback XML file:

  <appender name="OpenTelemetry" class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
    <captureArguments>false</captureArguments>
  </appender>

[AntonioLyubchev]

initializeOpenTelemetryAppenderFromProperties() which does not touch at all captureArguments state of OpenTelemetryAppender so it remains true

@AntonioLyubchev

initializeOpenTelemetryAppenderFromProperties() is to be able to initialize the Logback instrumentation of the OTel starter with properties. It overrides the configuration defined in a Logback XML file. The override is not implemented for the argument capture. It should be.

As a workaround, you could disable the capture of the log attributes from a Logback XML file:

  <appender name="OpenTelemetry" class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
    <captureArguments>false</captureArguments>
  </appender>

Thank you for identifying the issue and providing the workaround.

[laurit]

Resolved with #12445