Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for R2DBC #4673

Open
lvitaly opened this issue Feb 6, 2023 · 15 comments
Open

Support for R2DBC #4673

lvitaly opened this issue Feb 6, 2023 · 15 comments

Comments

@lvitaly
Copy link

lvitaly commented Feb 6, 2023

R2DBC project reached 1.0 release and already has support for popular databases such as Oracle, PostgreSQL, MySQL, H2, MsSQL, and MariaDB. R2DBC provides a nice feature for observability as r2dbc-proxy. But anyway, it would be nice to have instrumentation out of the box.

https://r2dbc.io

@jurriaan
Copy link

OpenTelemetry also solves this using a ProxyConnectionFactory: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/r2dbc-1.0/library/README.md

Hoping this can be added to dd-trace-java as well!

@d-wire
Copy link

d-wire commented Oct 31, 2023

Any chance there's an update on this? It'd be really nice to have r2dbc support in our DataDog traces, since the current behavior spawns a new trace id every time an r2dbc operation is performed.

@TheBatman09
Copy link

Do you have a workaround to avoid generating a new trace id each time ? I have the same issue.

@ahmadwrites
Copy link

Any updates or work arounds for this issue?

@bm1549 bm1549 mentioned this issue Dec 7, 2023
@bm1549
Copy link
Contributor

bm1549 commented Dec 7, 2023

Hello 👋 Brian from Datadog here

Unfortunately, there are no updates on R2DBC support at this time. That being said, we'll be sure to update this issue when we make progress

@ahmadwrites
Copy link

Any chance there's an update on this? It'd be really nice to have r2dbc support in our DataDog traces, since the current behavior spawns a new trace id every time an r2dbc operation is performed.

Have you found any workaround?

@d-wire
Copy link

d-wire commented Feb 14, 2024

Any chance there's an update on this? It'd be really nice to have r2dbc support in our DataDog traces, since the current behavior spawns a new trace id every time an r2dbc operation is performed.

Have you found any workaround?

Not yet unfortunately. Still hoping for full support for r2dbc at this point 🤞

@mcculls
Copy link
Contributor

mcculls commented Aug 25, 2024

Hi - since 1.38.1 you can now add the OTel R2DBC instrumentation at runtime to dd-java-agent: https://docs.datadoghq.com/opentelemetry/interoperability/instrumentation_libraries/?tab=java

For example:

# this demo requires Java 21 and Maven 3 on your PATH
git clone https://github.com/eugenp/tutorials

cd tutorials/spring-reactive-modules/spring-reactive-data

curl -L -O https://github.com/DataDog/dd-trace-java/releases/download/v1.38.1/dd-java-agent-1.38.1.jar

curl -L -O \
    https://repo1.maven.org/maven2/io/opentelemetry/javaagent/instrumentation/opentelemetry-javaagent-r2dbc-1.0/2.5.0-alpha/opentelemetry-javaagent-r2dbc-1.0-2.5.0-alpha.jar

mvn spring-boot:run \
    -Dstart-class=com.baeldung.pagination.PaginationApplication \
    -Dspring-boot.run.jvmArguments="-javaagent:dd-java-agent-1.38.1.jar \
        -Ddd.trace.otel.enabled=true \
        -Dotel.javaagent.extensions=opentelemetry-javaagent-r2dbc-1.0-2.5.0-alpha.jar"

Open http://127.0.0.1:8080/products to exercise the product query - you should see R2DBC spans in the Datadog UI

Note that we currently only support versions up to 2.5.0-alpha of the R2DBC instrumentation. This is because in 2.6.0-alpha OTel refactored some of the instrumentation config classes and we need to adjust our mappings to account for that.

@geezylucas
Copy link

Hi

I've implemented the solution suggested by @mcculls. However, I'm getting these errors.

java.lang.IllegalStateException: java.lang.NoClassDefFoundError: datadog/trace/bootstrap/otel/instrumentation/r2dbc/v1_0/internal/shaded/R2dbcTelemetry (wrong name: io/opentelemetry/instrumentation/r2dbc/v1_0/internal/shaded/R2dbcTelemetry) at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$UsingUnsafeInjection.defineClass(ClassInjector.java:1045) at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.injectRaw(ClassInjector.java:284) at datadog.trace.agent.tooling.HelperInjector.injectClassLoader(HelperInjector.java:154) at datadog.trace.agent.tooling.HelperInjector.transform(HelperInjector.java:119) at datadog.trace.agent.tooling.AdviceStack.transform(AdviceStack.java:31) at datadog.trace.agent.tooling.SplittingTransformer.transform(SplittingTransformer.java:29) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12508) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12445) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12154) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12936) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12866) at java.base/java.security.AccessController.doPrivileged(Unknown Source) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12388) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source) at datadog.trace.agent.tooling.bytebuddy.DDJava9ClassFileTransformer.transform(DDJava9ClassFileTransformer.java:60) at java.instrument/sun.instrument.TransformerManager.transform(Unknown Source) at java.instrument/sun.instrument.InstrumentationImpl.transform(Unknown Source) at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(Unknown Source) at java.base/java.security.SecureClassLoader.defineClass(Unknown Source) at java.base/java.net.URLClassLoader.defineClass(Unknown Source) at java.base/java.net.URLClassLoader$1.run(Unknown Source) at java.base/java.net.URLClassLoader$1.run(Unknown Source) at java.base/java.security.AccessController.doPrivileged(Unknown Source) at java.base/java.net.URLClassLoader.findClass(Unknown Source) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.loadClass(JarUrlClassLoader.java:107) at org.springframework.boot.loader.launch.LaunchedClassLoader.loadClass(LaunchedClassLoader.java:91) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) at org.springframework.boot.r2dbc.ConnectionFactoryBuilder$OptionsCapableWrapper.buildAndWrap(ConnectionFactoryBuilder.java:224) at org.springframework.boot.r2dbc.ConnectionFactoryBuilder$PoolingAwareOptionsCapableWrapper.buildAndWrap(ConnectionFactoryBuilder.java:237) at org.springframework.boot.r2dbc.ConnectionFactoryBuilder.build(ConnectionFactoryBuilder.java:206) at org.springframework.boot.autoconfigure.r2dbc.ConnectionFactoryConfigurations.createConnectionFactory(ConnectionFactoryConfigurations.java:76) at org.springframework.boot.autoconfigure.r2dbc.ConnectionFactoryConfigurations$PoolConfiguration$PooledConnectionFactoryConfiguration.connectionFactory(ConnectionFactoryConfigurations.java:102) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:146) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1355) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1185) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:904) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:782) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:542) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1355) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1185) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:904) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:782) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:237) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1375) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1212) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:975) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:971) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:625) at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:66) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1363) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1352) at com.gentera.domain.ServiceApplication.main(ServiceApplication.java:30) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102) at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64) at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:40) Caused by: java.lang.NoClassDefFoundError: datadog/trace/bootstrap/otel/instrumentation/r2dbc/v1_0/internal/shaded/R2dbcTelemetry (wrong name: io/opentelemetry/instrumentation/r2dbc/v1_0/internal/shaded/R2dbcTelemetry) at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(Unknown Source) at java.base/java.lang.ClassLoader$ByteBuddyAccessor$V1.defineClass(Unknown Source) at java.base/jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$UsingUnsafeInjection.defineClass(ClassInjector.java:1041) ... 94 more

and this:

[dd.trace 2024-10-31 18:59:23:810 +0000] [main] ERROR datadog.trace.agent.tooling.HelperInjector - Failed to inject helper classes - instrumentation.class=R2dbcInstrumentationModule instrumentation.target.classloader=org.springframework.boot.loader.launch.LaunchedClassLoader@4d6ccc97 instrumentation.target.class=class io.r2dbc.spi.ConnectionFactories

Any ideas for solving these errors?

Thank you.

@mcculls
Copy link
Contributor

mcculls commented Oct 31, 2024

@geezylucas which OpenTelenetry jar did you download?

Also can you list the exact command(s) you used and the version of dd-java-agent?

@geezylucas
Copy link

@mcculls

dd-java-agent.jar: https://dtdg.co/latest-java-tracer
opentelemetry-javaagent-r2dbc: https://repo1.maven.org/maven2/io/opentelemetry/javaagent/instrumentation/opentelemetry-javaagent-r2dbc-1.0/2.5.0-alpha/opentelemetry-javaagent-r2dbc-1.0-2.5.0-alpha.jar

I've been using a Docker (GKE) for deploy this API.

FROM eclipse-temurin:17.0.13_11-jre-noble

WORKDIR /opt/app

# Install prerequisites
RUN apt-get update && apt-get install -y curl

# adding user
RUN groupadd -r spring && useradd -r -g spring spring

# java profiling agent
RUN mkdir -p /opt/cprof && \
    wget -q -O- https://storage.googleapis.com/cloud-profiler/java/latest/profiler_java_agent.tar.gz \
    | tar xzv -C /opt/cprof

RUN chown -R spring:spring /opt/cprof

RUN curl -Lo dd-java-agent.jar 'https://dtdg.co/latest-java-tracer'

RUN curl -Lo opentelemetry-javaagent-r2dbc.jar 'https://repo1.maven.org/maven2/io/opentelemetry/javaagent/instrumentation/opentelemetry-javaagent-r2dbc-1.0/2.5.0-alpha/opentelemetry-javaagent-r2dbc-1.0-2.5.0-alpha.jar'

USER spring
VOLUME /tmp

ARG JAR_FILE="${CI_PROJECT_DIR}/build/libs/*.jar"

COPY ${JAR_FILE} app.jar

EXPOSE 8080
ENTRYPOINT ["sh", "-c", "java ${JAVA_OPTS} -javaagent:dd-java-agent.jar -Ddd.trace.otel.enabled=true -Dotel.javaagent.extensions=opentelemetry-javaagent-r2dbc.jar -XX:FlightRecorderOptions=stackdepth=256 -jar app.jar"]

and also I use flux (helm) to add these variables

datadog: # Datadog APM config
  enabled: true # Enable Datadog APM
  labels: # Add required Datadog labels
    tags.datadoghq.com/env: "qas" # Environment
    tags.datadoghq.com/service: "service-name" # Application name
    tags.datadoghq.com/version: "12" # Default to version 12
    admission.datadoghq.com/enabled: "true" # Enable injection
  annotations: # Add required Datadog annotations
    admission.datadoghq.com/java-lib.version: "v1.35.2" # Tell the APM java version
  env: # Insert environments vars inside pod
    - name: DD_LOGS_INJECTION
      value: "true"
    - name: DD_ENV
      valueFrom:
        fieldRef:
          fieldPath: metadata.labels['tags.datadoghq.com/env'] # Take values from labels
    - name: DD_SERVICE
      valueFrom:
        fieldRef:
          fieldPath: metadata.labels['tags.datadoghq.com/service'] # Take values from labels
    - name: DD_VERSION
      valueFrom:
        fieldRef:
          fieldPath: metadata.labels['tags.datadoghq.com/version'] # Take values from labels

finally, I see this env variables

Image

@mcculls
Copy link
Contributor

mcculls commented Nov 1, 2024

Hi @geezylucas - so far I can only recreate this exception when I use version 1.38.0 of dd-trace-java, if I use version 1.38.1 or later I don't see this exception.

I'm wondering whether a different -javaagent option is being added via ${JAVA_OPTS} which refers to the older dd-java-agent. The JVM will activate each Java agent in the order they appear on the command-line, so if the older dd-java-agent appears first then that's the version that will be used, and if that version is 1.38.0 then that would explain the error you see.

Could you check which version is shown in the DATADOG TRACER CONFIGURATION log message?

@geezylucas
Copy link

@mcculls

I see this version in DATADOG TRACER CONFIGURATION

Image

@mcculls
Copy link
Contributor

mcculls commented Nov 1, 2024

OK, that would explain it - the version of dd-java-agent there is 1.35.2 which is missing this fix which went into 1.38.1

If the older version is coming in via JAVA_OPTS then you could try changing the last line to:

ENTRYPOINT ["sh", "-c", "java -javaagent:dd-java-agent.jar ${JAVA_OPTS} -Ddd.trace.otel.enabled=true -Dotel.javaagent.extensions=opentelemetry-javaagent-r2dbc.jar -XX:FlightRecorderOptions=stackdepth=256 -jar app.jar"]

i.e. put the newer version first, so it takes precedence

Otherwise if the older version is being injected by single-step then you'd need to update the version in your single-step configuration.

Hope that helps

@geezylucas
Copy link

Thank you so much @mcculls

Last question,

I read this article https://docs.datadoghq.com/database_monitoring/connect_dbm_and_apm/?tab=java and I'm wondering if I can use your solution with OpenTelemetry (R2DBC) to correlate dbm and traces.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants