Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup autoconfigured resources in case of exception #5117

Merged
merged 6 commits into from
Jan 22, 2023

Conversation

jack-berg
Copy link
Member

Resolves #5090.

Wraps entire autoconfigure code block in a try / catch. Adds all Closeable components to a List<Closeable>. If an exception is thrown, catch it and close all the closeables in the list, then rethrow the exception.

@jack-berg jack-berg requested a review from a team January 13, 2023 00:18
}
}
}
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These testsuites were split out because they left behind resources that causes issues with other tests. Being able to merge them into the other test suites was a decent proxy for confirming that all tests were properly cleaning up resources.

SpanLimits config =
TracerProviderConfiguration.configureSpanLimits(
LogLimits config =
LoggerProviderConfiguration.configureLogLimits(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A test bug from bad copy / paste.

@Test
@SetSystemProperty(key = "otel.propagators", value = "cat")
void invalidPropagator() {
// TODO(jack-berg): confirm log warnings go away once exporters are properly shutdown (#5113)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's misleading log messages in the test output that look like this:

[Test worker] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export metrics. The request could not be executed. Full error message: executor rejected
    [Test worker] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export metrics. The request could not be executed. Full error message: executor rejected

They can happen when a test like this initializes a periodic metric reader with an exporter, and the exporter is shutdown before the periodic metric reader. When the periodic metric reader is shutdown it flushes to the exporter which has already had its executor threads shutdown. #5113 fixes this by adjusting all exporters to reject exports after being shutdown. This really is the correct behavior trying to export is bound to produce various exceptions after the network resources have been closed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you intending to keep the TODO here? Do we want to wait until 5113 is in the can?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed since #5113 is merged!


try {
try (SdkLoggerProvider loggerProvider = builder.build()) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This type of change is a common theme in this PR. Switching to try-with-resources when available to cleanup the code, and also block (for at most 10 seconds) until the resource is closed.


@Test
void configuration() {
void configureExporter_spiExporter() {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I'm in the neighborhood, I've renamed a bunch of vaguely named tests to better describe what they do.

@codecov
Copy link

codecov bot commented Jan 13, 2023

Codecov Report

Base: 91.13% // Head: 91.14% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (0d63dc6) compared to base (f28e0ad).
Patch coverage: 98.09% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #5117   +/-   ##
=========================================
  Coverage     91.13%   91.14%           
- Complexity     4907     4911    +4     
=========================================
  Files           549      547    -2     
  Lines         14506    14544   +38     
  Branches       1393     1399    +6     
=========================================
+ Hits          13220    13256   +36     
- Misses          890      891    +1     
- Partials        396      397    +1     
Impacted Files Coverage Δ
...nfigure/AutoConfiguredOpenTelemetrySdkBuilder.java 96.22% <96.00%> (-0.96%) ⬇️
...try/exporter/internal/grpc/OkHttpGrpcExporter.java 82.00% <100.00%> (ø)
...y/exporter/internal/grpc/UpstreamGrpcExporter.java 91.30% <100.00%> (ø)
...metry/exporter/internal/okhttp/OkHttpExporter.java 90.14% <100.00%> (ø)
...porter/jaeger/thrift/JaegerThriftSpanExporter.java 81.13% <100.00%> (ø)
...logging/otlp/OtlpJsonLoggingLogRecordExporter.java 85.71% <100.00%> (ø)
...er/logging/otlp/OtlpJsonLoggingMetricExporter.java 88.46% <100.00%> (ø)
...rter/logging/otlp/OtlpJsonLoggingSpanExporter.java 90.90% <100.00%> (ø)
...emetry/exporter/logging/LoggingMetricExporter.java 82.75% <100.00%> (ø)
...elemetry/exporter/logging/LoggingSpanExporter.java 95.12% <100.00%> (ø)
... and 26 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@jack-berg
Copy link
Member Author

I added an additional commit which reduces the log level when exporters are shutdown multiple times from WARNING to INFO. This PR can cause exporters to be shutdown multiple times since both an exporter and a provider are closeables that are cleaned up, and providers shutdown exporters as a side affect.

Reducing the log level to INFO reduces the log output of the tests, and seems more appropriate than WARNING anyway. There's other ways to reduce the log out of the tests with more effort.

@jack-berg jack-berg merged commit 6edba79 into open-telemetry:main Jan 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Autoconfigure should cleanup if configuration fails
3 participants