fix: no default otel endpoint for the operator #154
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The operator no longer has a default opentelemetry endpoint and will only collect and send traces if an endpoint is configured. This might address the memory leak but either way its a nice quality of life improvement as we have not been collecting traces from the operator anyway.
|
Turns out this breaks logging within the keramik operator. Still trying to figure out why. |
With this change logs will still be printed to STDOUT even if no OTLP endpoint is configured.
nathanielc
commented
Feb 19, 2024
| @@ -127,9 +127,6 @@ spec: | |||
| containerPort: 9464 | |||
| protocol: TCP | |||
| env: | |||
| # We are pointing to tempo or grafana tracing agent's otlp grpc receiver port | |||
| - name: OPERATOR_OTLP_ENDPOINT | |||
| value: "https://otel:4317" | |||
There was a problem hiding this comment.
We need to remove this config option from any operators we have configured otherwise they will still try to send traces.
|
@3benbox @smrz2001 I could use another review. The second commit fixes the issue with no longer logging any events to stdout. Local testing shows this fixed the OOM issues. The traces must have been being cached forever since they could not be flushed. In order to fix this in our prod envs we will need to remove the OTLP_ENDPOINT env var from the deployment of the operator. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The operator no longer has a default opentelemetry endpoint and will only collect and send traces if an endpoint is configured.
This might address the memory leak but either way its a nice quality of life improvement as we have not been collecting traces from the operator anyway.