Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added NPE checkers in Instrumentation Gateway #5383

Merged
merged 6 commits into from
Jun 21, 2023

Conversation

ValentinZakharov
Copy link
Contributor

@ValentinZakharov ValentinZakharov commented Jun 14, 2023

What Does This Do

The temporary solution. Added Null checker to avoid an NPE (logged to debug level).

Motivation

Reported NPE:

java.lang.NullPointerException: Cannot invoke "com.datadog.appsec.event.EventDispatcher$DataSubscriberInfoImpl.isEventDispatcher(com.datadog.appsec.event.EventDispatcher)" 
because "subscribers" is null. 
	at com.datadog.appsec.event.EventDispatcher.publishDataEvent(EventDispatcher.java:185)
	at com.datadog.appsec.event.ReplaceableEventProducerService.publishDataEvent(ReplaceableEventProducerService.java:33)
	at com.datadog.appsec.gateway.GatewayBridge.lambda$init$5(GatewayBridge.java:224)
	at com.datadog.appsec.gateway.GatewayBridge$$Lambda$151 (Source)
	at datadog.trace.api.gateway.InstrumentationGateway$6.apply(InstrumentationGateway.java:255)
	at datadog.trace.api.gateway.InstrumentationGateway$6.apply(InstrumentationGateway.java:250)
	at org.glassfish.jersey.server.internal.routing.UriRoutingContext.getPathParameters(UriRoutingContext.java:319)
	at org.glassfish.jersey.server.internal.routing.UriRoutingContext.getPathParameters(UriRoutingContext.java:297)
	at com.fsmatic.launcher.JettyEnvironment.decorateRequest(JettyEnvironment.java:284)
	at com.fsmatic.launcher.JettyEnvironment$2.decorateRequest(JettyEnvironment.java:341)
	at io.opentracing.contrib.jaxrs2.server.ServerTracingFilter.filter(ServerTracingFilter.java:78)
	at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132)
	at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68)
	at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
	at org.glassfish.jersey.server.ServerRuntime$1run(ServerRuntime.java:269)

Additional Notes

  • This is a temporary fix, while we are investigating the root of issue. It seems, there are something is wrong with the jersey instrumentation, that cause the problem.
  • Hitting this problem should have no impact to the application other than the logged exceptions.

@ValentinZakharov ValentinZakharov added type: bug comp: asm waf Application Security Management (WAF) labels Jun 14, 2023
@ValentinZakharov ValentinZakharov requested a review from a team as a code owner June 14, 2023 09:04
@ValentinZakharov ValentinZakharov self-assigned this Jun 14, 2023
Copy link
Contributor

@cataphract cataphract left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've looked at the code, at there seems to be a race condition indeed. Code follows this pattern:

    while (true) {
      if (this.respDataSubInfo == null) {
        this.respDataSubInfo =
            producerService.getDataSubscribers(
                KnownAddresses.RESPONSE_STATUS, KnownAddresses.RESPONSE_HEADERS_NO_COOKIES);
      }

      try {
        return producerService.publishDataEvent(this.respDataSubInfo, ctx, bundle, false);
      } catch (ExpiredSubscriberInfoException e) {
        this.respDataSubInfo = null;
      }
    }

where respDataSubInfo is a volatile field. Problem is some thread may have set the variable to null after the null check on another thread. The field needs to be copied to a local variable before the null check.

@pr-commenter
Copy link

pr-commenter bot commented Jun 15, 2023

Benchmarks

Parameters

Baseline Candidate
commit 1.17.0-SNAPSHOT~314c79e928 1.16.0-SNAPSHOT~765e2c3b02
config baseline candidate
See matching parameters
Baseline Candidate
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 3 performance regressions! Performance is the same for 19 cases.

scenario Δ mean execution_time
scenario:Startup-iast-IAST worse
[+0.330ms; +0.598ms] or [+2.835%; +5.141%]
scenario:Startup-iast-Remote Config worse
[+34.010µs; +66.745µs] or [+5.720%; +11.226%]
scenario:Startup-waf-Remote Config worse
[+143.454µs; +230.061µs] or [+20.934%; +33.572%]

@ValentinZakharov
Copy link
Contributor Author

I'll try to fix the race condition issue in next PR

@ValentinZakharov ValentinZakharov enabled auto-merge (squash) June 16, 2023 09:14
@ValentinZakharov ValentinZakharov requested a review from a team as a code owner June 20, 2023 12:16
@smola smola requested a review from cataphract June 20, 2023 14:38
@ValentinZakharov ValentinZakharov merged commit 74b3e2f into master Jun 21, 2023
@ValentinZakharov ValentinZakharov deleted the vzakharov/ig_npe_checker branch June 21, 2023 16:56
@github-actions github-actions bot added this to the 1.17.0 milestone Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp: asm waf Application Security Management (WAF) type: bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants