[EXPORTER] Async exporting for otlp grpc #2407

owent · 2023-11-15T13:13:26Z

Fixes #2406

Changes

Add arena for OTLP exporters.
Add async exporting for OTLP gRPC exporter, and move data lifetime with RPC calls.

For significant contributions please make sure you have completed the following items:

CHANGELOG.md updated for non-trivial changes
Unit tests have been added
Changes in public API reviewed

… by using a modern malloc library.

codecov · 2023-11-15T13:26:47Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (497eaf4) 87.12% compared to head (d86cb39) 87.09%.
Report is 2 commits behind head on main.

❗ Current head d86cb39 differs from pull request most recent head 37f7dbd. Consider uploading reports for the commit 37f7dbd to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2407      +/-   ##
==========================================
- Coverage   87.12%   87.09%   -0.02%     
==========================================
  Files         200      200              
  Lines        6109     6103       -6     
==========================================
- Hits         5322     5315       -7     
- Misses        787      788       +1

see 3 files with indirect coverage changes

…tlp_grpc_exporter

marcalff

Thanks for the fix.

It will take me several review passes to go over this change,
please find a first batch of comments for now.

exporters/otlp/include/opentelemetry/exporters/otlp/otlp_grpc_client.h

exporters/otlp/src/otlp_grpc_client.cc

exporters/otlp/src/otlp_grpc_exporter.cc

exporters/otlp/src/otlp_grpc_log_record_exporter.cc

exporters/otlp/src/otlp_grpc_metric_exporter.cc

…tlp_grpc_exporter # Conflicts: # CHANGELOG.md

…tlp_grpc_exporter

marcalff

Impressive work.

Please find here more comments / questions.

exporters/otlp/include/opentelemetry/exporters/otlp/otlp_grpc_client.h

exporters/otlp/src/otlp_grpc_client.cc

owent · 2024-01-10T02:46:24Z

Ping - do we need to support GRPC < 1.39, or can we just document a recent GRPC is needed instead ?

@marcalff It would be better if we can suppor a low version of gRPC. We have some project with old toolchains which can only use gRPC 1.33.

marcalff · 2024-01-10T09:09:30Z

Ping - do we need to support GRPC < 1.39, or can we just document a recent GRPC is needed instead ?

@marcalff It would be better if we can suppor a low version of gRPC. We have some project with old toolchains which can only use gRPC 1.33.

Thanks for the clarification. Resolved then.

marcalff

Excellent work, thanks for the fix.

I will not pretend I know grpc async in depth to fully appreciate the code here.
The patch looks good to me, based on my limited knowledge of this area.

Given the patch size, a second opinion is needed.

marcalff · 2024-01-10T10:00:21Z

@lalitb @ThomsonTan @esigo

Added the please review label, for a second review.

ThomsonTan · 2024-01-18T20:35:03Z

exporters/otlp/include/opentelemetry/exporters/otlp/otlp_grpc_exporter.h

@@ -23,6 +23,8 @@ namespace exporter
 namespace otlp
 {

+class OtlpGrpcClient;


Can this be moved to otlp_grpc_exporter_options.h as a central place?

Users use the factory and otlp_grpc*_options.h to create OTLP gRPC exporters, but I think users do not need to know the internal class OtlpGrpcClient?

esigo

Thanks @owent for yet another great work.
I had a minor question only.

exporters/otlp/test/otlp_grpc_log_record_exporter_test.cc

exporters/otlp/src/otlp_grpc_client.cc

…tlp_grpc_exporter

…porter' into async_exporting_for_otlp_grpc_exporter

owent added 2 commits November 14, 2023 16:36

Use protobuf arena to reduce memory fragments and improve performence…

3e840b0

… by using a modern malloc library.

Add async exporting for OTLP gRPC exporter

9b3b057

owent requested a review from a team November 15, 2023 13:13

owent added 2 commits November 15, 2023 21:23

Fix OTLP HTTP exporter without async exporting.

74175a3

Fix condition_variable including

3cc2776

owent added 7 commits November 16, 2023 10:23

Fix compile problem without ENABLE_ASYNC_EXPORT

aeb7b55

Use callback API for OTLP gRPC exporter.

25419f6

Fix unit test for OTLP gRPC async exporting.

e7f119a

Fix grpcpp header

4301f23

There is no compatible way to detect gRPC version.

1862010

Fix style and benchmark

5fef32a

Fix styles

0dc3f73

owent changed the title ~~[WIP] Async exporting for otlp grpc exporter~~ Async exporting for otlp grpc exporter Nov 16, 2023

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

0951836

esigo assigned marcalff Nov 20, 2023

owent added 3 commits November 21, 2023 20:18

Merge remote-tracking branch 'github/main' into async_exporting_for_o…

a9c538d

…tlp_grpc_exporter

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

ae7dc10

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

57ebf34

marcalff reviewed Nov 30, 2023

View reviewed changes

owent added 7 commits December 1, 2023 22:58

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

a3c78d2

Fix feedbacks.

0dadc1c

Fix style

ac964f3

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

5c7c352

Merge remote-tracking branch 'github/main' into async_exporting_for_o…

54fb976

…tlp_grpc_exporter # Conflicts: # CHANGELOG.md

Fixes format

692cfed

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

49c3b7d

lalitb mentioned this pull request Dec 11, 2023

Otel threads spawn before "main" impact signal handling #2445

Closed

owent and others added 2 commits December 16, 2023 02:59

Merge remote-tracking branch 'github/main' into async_exporting_for_o…

be4bcd1

…tlp_grpc_exporter

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

3a53ed9

owent and others added 4 commits December 27, 2023 23:01

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

7c66023

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

6587bd3

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

4c892a7

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

63489fa

marcalff reviewed Jan 9, 2024

View reviewed changes

Cleanup unused methods

05525fd

marcalff approved these changes Jan 10, 2024

View reviewed changes

marcalff added the pr:please-review This PR is ready for review label Jan 10, 2024

marcalff changed the title ~~Async exporting for otlp grpc exporter~~ [EXPORTER] Async exporting for otlp grpc Jan 10, 2024

Make constructor and destructors always available.

d07cdf6

marcalff assigned esigo Jan 10, 2024

owent and others added 3 commits January 12, 2024 22:47

Fix wair_for in destructor

cdb8816

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

7dd1bd3

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

1cc3f1c

ThomsonTan reviewed Jan 18, 2024

View reviewed changes

owent and others added 3 commits January 21, 2024 13:06

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

d0f1add

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

daccf33

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

d86cb39

esigo reviewed Jan 29, 2024

View reviewed changes

exporters/otlp/test/otlp_grpc_log_record_exporter_test.cc Show resolved Hide resolved

exporters/otlp/src/otlp_grpc_client.cc Show resolved Hide resolved

owent added 3 commits February 2, 2024 01:10

Merge remote-tracking branch 'github/main' into async_exporting_for_o…

95f7614

…tlp_grpc_exporter

Add comments for old version of gRPC, make is_shutdown_ thread-safety

0f4c77f

Merge remote-tracking branch 'origin/async_exporting_for_otlp_grpc_ex…

277b73b

…porter' into async_exporting_for_otlp_grpc_exporter

ThomsonTan approved these changes Feb 8, 2024

View reviewed changes

Merge branch 'main' into async_exporting_for_otlp_grpc_exporter

37f7dbd

ThomsonTan merged commit 6e8f7c4 into open-telemetry:main Feb 8, 2024
47 checks passed

chenrui333 mentioned this pull request Feb 18, 2024

opentelemetry-cpp 1.14.0 Homebrew/homebrew-core#163187

Merged

owent deleted the async_exporting_for_otlp_grpc_exporter branch February 27, 2024 11:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EXPORTER] Async exporting for otlp grpc #2407

[EXPORTER] Async exporting for otlp grpc #2407

owent commented Nov 15, 2023 •

edited

Loading

codecov bot commented Nov 15, 2023 •

edited

Loading

marcalff left a comment

marcalff left a comment

owent commented Jan 10, 2024 •

edited

Loading

marcalff commented Jan 10, 2024

marcalff left a comment

marcalff commented Jan 10, 2024

ThomsonTan Jan 18, 2024

owent Jan 21, 2024 •

edited

Loading

esigo left a comment

[EXPORTER] Async exporting for otlp grpc #2407

[EXPORTER] Async exporting for otlp grpc #2407

Conversation

owent commented Nov 15, 2023 • edited Loading

Changes

codecov bot commented Nov 15, 2023 • edited Loading

Codecov Report

marcalff left a comment

Choose a reason for hiding this comment

marcalff left a comment

Choose a reason for hiding this comment

owent commented Jan 10, 2024 • edited Loading

marcalff commented Jan 10, 2024

marcalff left a comment

Choose a reason for hiding this comment

marcalff commented Jan 10, 2024

ThomsonTan Jan 18, 2024

Choose a reason for hiding this comment

owent Jan 21, 2024 • edited Loading

Choose a reason for hiding this comment

esigo left a comment

Choose a reason for hiding this comment

owent commented Nov 15, 2023 •

edited

Loading

codecov bot commented Nov 15, 2023 •

edited

Loading

owent commented Jan 10, 2024 •

edited

Loading

owent Jan 21, 2024 •

edited

Loading