Skip to content

Commit

Permalink
[Metrics] Add OTLP http metric exporter (#1487)
Browse files Browse the repository at this point in the history
* Add OTLP http metric exporter

Signed-off-by: owentou <owentou@tencent.com>

* Add changelog for OTLP HTTP metric exporter

Signed-off-by: owentou <owentou@tencent.com>

* Add unit test of `SumPointData` for `OtlpHttpMetricExporter`

Signed-off-by: owentou <owentou@tencent.com>

* Fix dependency of otlp recordable, add `Gauge` support for `OtlpMetricUtils::PopulateRequest`

Signed-off-by: owentou <owentou@tencent.com>

* Add unit test for `OtlpHttpMetricExporter`

Signed-off-by: owentou <owentou@tencent.com>

* Fix format

Signed-off-by: owentou <owentou@tencent.com>

* Fix unit test `OtlpMetricSerializationTest.ObservableGauge`

Signed-off-by: owentou <owentou@tencent.com>
  • Loading branch information
owent authored Jul 19, 2022
1 parent 4062237 commit 24d5100
Show file tree
Hide file tree
Showing 19 changed files with 1,570 additions and 59 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Increment the:

* [TRACE SDK] Add trace sdk builders (#1393) [#1471](https://github.com/open-telemetry/opentelemetry-cpp/pull/1471)
* [EXAMPLE] Fix memory ownership of InMemorySpanExporter (#1473) [#1471](https://github.com/open-telemetry/opentelemetry-cpp/pull/1471)
* [EXPORTER] Add metrics OTLP/HTTP exporter [#1487](https://github.com/open-telemetry/opentelemetry-cpp/pull/1487)
* [EXPORTER] OTLP http exporter allow concurrency session ([#1209](https://github.com/open-telemetry/opentelemetry-cpp/pull/1209))
* [EXT] `curl::HttpClient` use `curl_multi_handle` instead of creating a thread
for every request and it's able to reuse connections now. ([#1317](https://github.com/open-telemetry/opentelemetry-cpp/pull/1317))
Expand Down
5 changes: 5 additions & 0 deletions cmake/opentelemetry-cpp-config.cmake.in
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,11 @@
# opentelemetry-cpp::otlp_recordable - Imported target of opentelemetry-cpp::otlp_recordable
# opentelemetry-cpp::otlp_grpc_exporter - Imported target of opentelemetry-cpp::otlp_grpc_exporter
# opentelemetry-cpp::otlp_grpc_log_exporter - Imported target of opentelemetry-cpp::otlp_grpc_log_exporter
# opentelemetry-cpp::otlp_grpc_metrics_exporter - Imported target of opentelemetry-cpp::otlp_grpc_metrics_exporter
# opentelemetry-cpp::otlp_http_client - Imported target of opentelemetry-cpp::otlp_http_client
# opentelemetry-cpp::otlp_http_exporter - Imported target of opentelemetry-cpp::otlp_http_exporter
# opentelemetry-cpp::otlp_http_log_exporter - Imported target of opentelemetry-cpp::otlp_http_log_exporter
# opentelemetry-cpp::otlp_http_metric_exporter - Imported target of opentelemetry-cpp::otlp_http_metric_exporter
# opentelemetry-cpp::ostream_log_exporter - Imported target of opentelemetry-cpp::ostream_log_exporter
# opentelemetry-cpp::ostream_metrics_exporter - Imported target of opentelemetry-cpp::ostream_metrics_exporter
# opentelemetry-cpp::ostream_span_exporter - Imported target of opentelemetry-cpp::ostream_span_exporter
Expand Down Expand Up @@ -84,9 +86,12 @@ set(_OPENTELEMETRY_CPP_LIBRARIES_TEST_TARGETS
in_memory_span_exporter
otlp_recordable
otlp_grpc_exporter
otlp_grpc_log_exporter
otlp_grpc_metrics_exporter
otlp_http_client
otlp_http_exporter
otlp_http_log_exporter
otlp_http_metric_exporter
ostream_log_exporter
ostream_metrics_exporter
ostream_span_exporter
Expand Down
48 changes: 47 additions & 1 deletion exporters/otlp/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,14 @@ cc_library(
name = "otlp_recordable",
srcs = [
"src/otlp_log_recordable.cc",
"src/otlp_metric_utils.cc",
"src/otlp_populate_attribute_utils.cc",
"src/otlp_recordable.cc",
"src/otlp_recordable_utils.cc",
],
hdrs = [
"include/opentelemetry/exporters/otlp/otlp_log_recordable.h",
"include/opentelemetry/exporters/otlp/otlp_metric_utils.h",
"include/opentelemetry/exporters/otlp/otlp_populate_attribute_utils.h",
"include/opentelemetry/exporters/otlp/otlp_recordable.h",
"include/opentelemetry/exporters/otlp/otlp_recordable_utils.h",
Expand All @@ -39,6 +41,7 @@ cc_library(
"//sdk/src/resource",
"//sdk/src/trace",
"@com_github_opentelemetry_proto//:logs_service_proto_cc",
"@com_github_opentelemetry_proto//:metrics_service_proto_cc",
"@com_github_opentelemetry_proto//:trace_service_proto_cc",
],
)
Expand Down Expand Up @@ -161,6 +164,30 @@ cc_library(
],
)

cc_library(
name = "otlp_http_metric_exporter",
srcs = [
"src/otlp_http_metric_exporter.cc",
],
hdrs = [
"include/opentelemetry/exporters/otlp/otlp_environment.h",
"include/opentelemetry/exporters/otlp/otlp_http_metric_exporter.h",
"include/opentelemetry/exporters/otlp/protobuf_include_prefix.h",
"include/opentelemetry/exporters/otlp/protobuf_include_suffix.h",
],
strip_include_prefix = "include",
tags = [
"otlp",
"otlp_http_metric",
],
deps = [
":otlp_http_client",
":otlp_recordable",
"//sdk/src/metrics",
"@com_github_opentelemetry_proto//:metrics_service_proto_cc",
],
)

cc_library(
name = "otlp_grpc_log_exporter",
srcs = [
Expand Down Expand Up @@ -204,7 +231,10 @@ cc_test(

cc_test(
name = "otlp_log_recordable_test",
srcs = ["test/otlp_log_recordable_test.cc"],
srcs = [
"test/otlp_log_recordable_test.cc",
"test/otlp_metrics_serialization_test.cc",
],
tags = [
"otlp",
"test",
Expand Down Expand Up @@ -294,6 +324,22 @@ cc_test(
],
)

cc_test(
name = "otlp_http_metric_exporter_test",
srcs = ["test/otlp_http_metric_exporter_test.cc"],
tags = [
"otlp",
"otlp_http_metric",
"test",
],
deps = [
":otlp_http_metric_exporter",
"//api",
"//ext/src/http/client/nosend:http_client_nosend",
"@com_google_googletest//:gtest_main",
],
)

cc_test(
name = "otlp_grpc_log_exporter_test",
srcs = ["test/otlp_grpc_log_exporter_test.cc"],
Expand Down
32 changes: 32 additions & 0 deletions exporters/otlp/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,21 @@ if(WITH_OTLP_HTTP)
list(APPEND OPENTELEMETRY_OTLP_TARGETS opentelemetry_exporter_otlp_http_log)

endif()
if(NOT WITH_METRICS_PREVIEW)
add_library(opentelemetry_exporter_otlp_http_metric
src/otlp_http_metric_exporter.cc)

set_target_properties(opentelemetry_exporter_otlp_http_metric
PROPERTIES EXPORT_NAME otlp_http_metric_exporter)

target_link_libraries(
opentelemetry_exporter_otlp_http_metric
PUBLIC opentelemetry_otlp_recordable
opentelemetry_exporter_otlp_http_client)

list(APPEND OPENTELEMETRY_OTLP_TARGETS
opentelemetry_exporter_otlp_http_metric)
endif()
endif()

install(
Expand Down Expand Up @@ -240,5 +255,22 @@ if(BUILD_TESTING)
TEST_PREFIX exporter.otlp.
TEST_LIST otlp_http_log_exporter_test)
endif()

if(NOT WITH_METRICS_PREVIEW)
add_executable(otlp_http_metric_exporter_test
test/otlp_http_metric_exporter_test.cc)
target_link_libraries(
otlp_http_metric_exporter_test
${GTEST_BOTH_LIBRARIES}
${CMAKE_THREAD_LIBS_INIT}
${GMOCK_LIB}
opentelemetry_exporter_otlp_http_metric
opentelemetry_metrics
http_client_nosend)
gtest_add_tests(
TARGET otlp_http_metric_exporter_test
TEST_PREFIX exporter.otlp.
TEST_LIST otlp_http_metric_exporter_test)
endif()
endif()
endif() # BUILD_TESTING
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,53 @@ inline OtlpHeaders GetOtlpDefaultLogHeaders()

return result;
}

inline const std::string GetOtlpDefaultHttpMetricEndpoint()
{
constexpr char kOtlpMetricsEndpointEnv[] = "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT";
constexpr char kOtlpEndpointEnv[] = "OTEL_EXPORTER_OTLP_ENDPOINT";
constexpr char kOtlpEndpointDefault[] = "http://localhost:4318/v1/metrics";

auto endpoint = opentelemetry::sdk::common::GetEnvironmentVariable(kOtlpMetricsEndpointEnv);
if (endpoint.empty())
{
endpoint = opentelemetry::sdk::common::GetEnvironmentVariable(kOtlpEndpointEnv);
if (!endpoint.empty())
{
endpoint += "/v1/metrics";
}
}
return endpoint.size() ? endpoint : kOtlpEndpointDefault;
}

inline const std::chrono::system_clock::duration GetOtlpDefaultMetricTimeout()
{
constexpr char kOtlpMetricsTimeoutEnv[] = "OTEL_EXPORTER_OTLP_METRICS_TIMEOUT";
constexpr char kOtlpTimeoutEnv[] = "OTEL_EXPORTER_OTLP_TIMEOUT";

auto timeout = opentelemetry::sdk::common::GetEnvironmentVariable(kOtlpMetricsTimeoutEnv);
if (timeout.empty())
{
timeout = opentelemetry::sdk::common::GetEnvironmentVariable(kOtlpTimeoutEnv);
}
return GetOtlpTimeoutFromString(timeout.c_str());
}

inline OtlpHeaders GetOtlpDefaultMetricHeaders()
{
constexpr char kOtlpMetricsHeadersEnv[] = "OTEL_EXPORTER_OTLP_METRICS_HEADERS";
constexpr char kOtlpHeadersEnv[] = "OTEL_EXPORTER_OTLP_HEADERS";

OtlpHeaders result;
std::unordered_set<std::string> metric_remove_cache;
DumpOtlpHeaders(result, kOtlpHeadersEnv, metric_remove_cache);

metric_remove_cache.clear();
DumpOtlpHeaders(result, kOtlpMetricsHeadersEnv, metric_remove_cache);

return result;
}

} // namespace otlp
} // namespace exporter
OPENTELEMETRY_END_NAMESPACE
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,11 @@ class OtlpHttpClient
std::function<bool(opentelemetry::sdk::common::ExportResult)> &&result_callback,
std::size_t max_running_requests) noexcept;

/**
* Force flush the HTTP client.
*/
bool ForceFlush(std::chrono::microseconds timeout = (std::chrono::microseconds::max)()) noexcept;

/**
* Shut down the HTTP client.
* @param timeout an optional timeout, the default timeout of 0 means that no
Expand All @@ -159,6 +164,12 @@ class OtlpHttpClient
*/
inline const OtlpHttpClientOptions &GetOptions() const noexcept { return options_; }

/**
* Get if this OTLP http client is shutdown.
* @return return true after Shutdown is called.
*/
bool IsShutdown() const noexcept;

private:
struct HttpSessionData
{
Expand Down Expand Up @@ -212,11 +223,11 @@ class OtlpHttpClient
*/
bool cleanupGCSessions() noexcept;

bool isShutdown() const noexcept;

// For testing
friend class OtlpHttpExporterTestPeer;
friend class OtlpHttpLogExporterTestPeer;
friend class OtlpHttpMetricExporterTestPeer;

/**
* Create an OtlpHttpClient using the specified http client.
* Only tests can call this constructor directly.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ class OtlpHttpExporter final : public opentelemetry::sdk::trace::SpanExporter
* timeout is applied.
* @return return the status of this operation
*/
bool Shutdown(std::chrono::microseconds timeout = std::chrono::microseconds(0)) noexcept override;
bool Shutdown(
std::chrono::microseconds timeout = std::chrono::microseconds::max()) noexcept override;

private:
// The configuration options associated with this exporter.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,8 @@ class OtlpHttpLogExporter final : public opentelemetry::sdk::logs::LogExporter
* Shutdown this exporter.
* @param timeout The maximum time to wait for the shutdown method to return
*/
bool Shutdown(std::chrono::microseconds timeout = std::chrono::microseconds(0)) noexcept override;
bool Shutdown(
std::chrono::microseconds timeout = std::chrono::microseconds::max()) noexcept override;

private:
// Configuration options for the exporter
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

#pragma once
#ifndef ENABLE_METRICS_PREVIEW

# include "opentelemetry/sdk/metrics/metric_exporter.h"

# include "opentelemetry/exporters/otlp/otlp_http_client.h"

# include "opentelemetry/exporters/otlp/otlp_environment.h"

# include <chrono>
# include <cstddef>
# include <memory>
# include <string>

OPENTELEMETRY_BEGIN_NAMESPACE
namespace exporter
{
namespace otlp
{

/**
* Struct to hold OTLP exporter options.
*/
struct OtlpHttpMetricExporterOptions
{
// The endpoint to export to. By default
// @see
// https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md
// @see https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver
std::string url = GetOtlpDefaultHttpMetricEndpoint();

// By default, post json data
HttpRequestContentType content_type = HttpRequestContentType::kJson;

// If convert bytes into hex. By default, we will convert all bytes but id into base64
// This option is ignored if content_type is not kJson
JsonBytesMappingKind json_bytes_mapping = JsonBytesMappingKind::kHexId;

// If using the json name of protobuf field to set the key of json. By default, we will use the
// field name just like proto files.
bool use_json_name = false;

// Whether to print the status of the exporter in the console
bool console_debug = false;

// TODO: Enable/disable to verify SSL certificate
std::chrono::system_clock::duration timeout = GetOtlpDefaultMetricTimeout();

// Additional HTTP headers
OtlpHeaders http_headers = GetOtlpDefaultMetricHeaders();

# ifdef ENABLE_ASYNC_EXPORT
// Concurrent requests
// https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md#otlpgrpc-concurrent-requests
std::size_t max_concurrent_requests = 64;

// Requests per connections
std::size_t max_requests_per_connection = 8;
# endif
};

/**
* The OTLP exporter exports metrics data in OpenTelemetry Protocol (OTLP) format in HTTP.
*/
class OtlpHttpMetricExporter final : public opentelemetry::sdk::metrics::MetricExporter
{
public:
/**
* Create an OtlpHttpMetricExporter with default exporter options.
*/
OtlpHttpMetricExporter();

/**
* Create an OtlpHttpMetricExporter with user specified options.
* @param options An object containing the user's configuration options.
*/
OtlpHttpMetricExporter(const OtlpHttpMetricExporterOptions &options);

opentelemetry::sdk::common::ExportResult Export(
const opentelemetry::sdk::metrics::ResourceMetrics &data) noexcept override;

/**
* Force flush the exporter.
*/
bool ForceFlush(
std::chrono::microseconds timeout = (std::chrono::microseconds::max)()) noexcept override;

bool Shutdown(
std::chrono::microseconds timeout = (std::chrono::microseconds::max)()) noexcept override;

private:
// Configuration options for the exporter
const OtlpHttpMetricExporterOptions options_;

// Object that stores the HTTP sessions that have been created
std::unique_ptr<OtlpHttpClient> http_client_;
// For testing
friend class OtlpHttpMetricExporterTestPeer;

/**
* Create an OtlpHttpMetricExporter using the specified http client.
* Only tests can call this constructor directly.
* @param http_client the http client to be used for exporting
*/
OtlpHttpMetricExporter(std::unique_ptr<OtlpHttpClient> http_client);
};
} // namespace otlp
} // namespace exporter
OPENTELEMETRY_END_NAMESPACE
#endif
Loading

1 comment on commit 24d5100

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'OpenTelemetry-cpp sdk Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite Current: 24d5100 Previous: 4062237 Ratio
BM_BaselineBuffer/2 11427702.903747559 ns/iter 3235454.559326172 ns/iter 3.53
BM_BaselineBuffer/4 13585989.475250244 ns/iter 5168609.619140625 ns/iter 2.63
BM_LockFreeBuffer/4 8000767.230987549 ns/iter 1649455.8212604928 ns/iter 4.85

This comment was automatically generated by workflow using github-action-benchmark.

Please sign in to comment.