Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid infinity loop when exporter use http to send spans #332

Closed
OlivierAlbertini opened this issue Sep 24, 2019 · 9 comments · Fixed by #335
Closed

Avoid infinity loop when exporter use http to send spans #332

OlivierAlbertini opened this issue Sep 24, 2019 · 9 comments · Fixed by #335
Assignees

Comments

@OlivierAlbertini
Copy link
Member

OlivierAlbertini commented Sep 24, 2019

Is your feature request related to a problem? Please describe.

Right now, I must use this config to avoid infinity loop:

const config = { 
 plugins: {
            http: {
                enabled: true,
                path: '@opentelemetry/plugin-http',
                ignoreOutgoingUrls: [/spans/]
            }
        }
}

Describe the solution you'd like
We should handle this internally. Perhaps we could use a specific header that will be handled by http plugin for ignoring outgoing request with that header?

@vmarchaud
Copy link
Member

@OlivierAlbertini
Copy link
Member Author

should we set x-opentelemetry-outgoing-request ?

OlivierAlbertini added a commit to VilledeMontreal/opentelemetry-js that referenced this issue Sep 25, 2019
closes open-telemetry#332

Signed-off-by: Olivier Albertini <olivier.albertini@montreal.ca>
OlivierAlbertini added a commit to VilledeMontreal/opentelemetry-js that referenced this issue Sep 25, 2019
closes open-telemetry#332

Signed-off-by: Olivier Albertini <olivier.albertini@montreal.ca>
@OlivierAlbertini OlivierAlbertini changed the title Avoid infinity network loop when exporter use http to send spans Avoid infinity loop when exporter use http to send spans Sep 25, 2019
@vmarchaud
Copy link
Member

@OlivierAlbertini i'm good with the header you set in the PR

OlivierAlbertini added a commit to VilledeMontreal/opentelemetry-js that referenced this issue Sep 26, 2019
closes open-telemetry#332

Signed-off-by: Olivier Albertini <olivier.albertini@montreal.ca>
danielkhan pushed a commit that referenced this issue Sep 27, 2019
* fix: dont trace ourselves

closes #332

Signed-off-by: Olivier Albertini <olivier.albertini@montreal.ca>

* fix: add mayurkale22 recommendations

Signed-off-by: Olivier Albertini <olivier.albertini@montreal.ca>
@ljcremer
Copy link

This still seems to be an issue and caused our platform to log 50M+ spans in two weeks with very little traffic on the platform.
Here is the related issue and the way we resolved it:

@dyladan
Copy link
Member

dyladan commented Jul 17, 2020

@ljcremer we are working on a new way to prevent these infinite loops by allowing exporters to set a context property during export which disables tracing. A quick glance at the exporter code looks like the header to disable tracing is set properly in the exporter, but I'll make sure the http/https plugins are properly respecting it.

@aabmass
Copy link
Member

aabmass commented Jul 17, 2020

@dyladan
Copy link
Member

dyladan commented Jul 17, 2020

Yes I meant span processor. I often say "Exporter" to refer to the whole export pipeline or process.

lukaswelinder pushed a commit to agile-pm/opentelemetry-js that referenced this issue Jul 24, 2020
* Error for mismatching env variables

Signed-off-by: Yevgeny Petukhov <petukhov.yevgeny@gmail.com>

* Warning instead of error

Signed-off-by: Yevgeny Petukhov <petukhov.yevgeny@gmail.com>

* Warning text and var name are fixed

Signed-off-by: Yevgeny Petukhov <petukhov.yevgeny@gmail.com>

* fixup! Warning text and var name are fixed

Signed-off-by: Yevgeny Petukhov <petukhov.yevgeny@gmail.com>

* fixup! Warning text and var name are fixed

Signed-off-by: Yevgeny Petukhov <petukhov.yevgeny@gmail.com>

* Move check to _validateEnv method and call it at initTracer

Signed-off-by: Yevgeny Petukhov <petukhov.yevgeny@gmail.com>

* fixup! Move check to _validateEnv method and call it at initTracer

Signed-off-by: Yevgeny Petukhov <petukhov.yevgeny@gmail.com>
@Hariharan-Neutrinos
Copy link

what is the solution for this? Fix!?

@Flarna
Copy link
Member

Flarna commented Nov 2, 2023

current solution is that SpanProcessor/exporter set the suppressTracing marker on context which causes that tracer doesn't create a span.
see e.g. here, here and here.

There might be other implementations of Exporters/SpanProcessors/Tracers in the wild which are not covered by the SDK implementation so they should adapt this approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants