Skip to content

Commit

Permalink
[EXPORTER] Elastic Search exporter follow ECS guidelines (#3107)
Browse files Browse the repository at this point in the history
* [EXPORTERS]: elastic search log message within `message` key instead of
`body`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-base.html#field-message

Refs #3091

* [EXPORTERS]: elastic search set severity within `log.level` key instead of
    `severity`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html#field-log-level

Refs #3091

* [EXPORTERS]: elastic search set timestamp within `@timestamp` instead of
`timestamp`

Also changes the format to be a Date string.

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-base.html#field-timestamp

Refs #3091

* [EXPORTERS]: elastic search set instrumentation scope within `log.logger` instead of
`name`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html#field-log-logger

Refs #3091

* EXPORTERS]: elastic search recorable constructor sets `ecs.version` field
to 8.11.0

According to ECS guidelines this field is mandatory
https://www.elastic.co/guide/en/ecs/8.11/ecs-guidelines.html

Refs #3091

* [EXPORTERS]: elastic search put attributes in json root instead of under
`attributes`

This allows user to set other fields that are part of the [ECS log documentation](https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html).
For instance, it allows to have an attribute with key `log.file`, that
will, thanks to `nlohmann::json`, appear as :
```
{
  "log": {
    "file": "xxx"
  }
}
```

Closes #3091
  • Loading branch information
ShadowMaxLeb authored Oct 27, 2024
1 parent 2c912d5 commit 5e62859
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 6 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ class ElasticSearchRecordable final : public sdk::logs::Recordable
void WriteValue(const opentelemetry::common::AttributeValue &value, const std::string &name);

public:
ElasticSearchRecordable() noexcept;

/**
* Returns a JSON object contain the log information
*/
Expand Down
50 changes: 44 additions & 6 deletions exporters/elasticsearch/src/es_log_recordable.cc
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,11 @@ void ElasticSearchRecordable::WriteValue(const opentelemetry::common::AttributeV
}
}

ElasticSearchRecordable::ElasticSearchRecordable() noexcept : sdk::logs::Recordable()
{
json_["ecs"]["version"] = "8.11.0";
}

nlohmann::json ElasticSearchRecordable::GetJSON() noexcept
{
return json_;
Expand All @@ -205,7 +210,38 @@ nlohmann::json ElasticSearchRecordable::GetJSON() noexcept
void ElasticSearchRecordable::SetTimestamp(
opentelemetry::common::SystemTimestamp timestamp) noexcept
{
json_["timestamp"] = timestamp.time_since_epoch().count();
const std::chrono::system_clock::time_point timePoint{timestamp};

// If built with with at least cpp 20 then use std::format
// Otherwise use the old style to format the timestamp in UTC
#if __cplusplus >= 202002L
const std::string dateStr = std::format("{:%FT%T%Ez}", timePoint);
#else
const static int dateToSecondsSize = 19;
const static int millisecondsSize = 8;
const static int timeZoneSize = 1;
const static int dateSize = dateToSecondsSize + millisecondsSize + timeZoneSize;

std::time_t time = std::chrono::system_clock::to_time_t(timePoint);
std::tm tm = *std::gmtime(&time);

char bufferDate[dateSize]; // example: 2024-10-18T07:26:00.123456Z
std::strftime(bufferDate, sizeof(bufferDate), "%Y-%m-%dT%H:%M:%S", &tm);
auto microseconds =
std::chrono::duration_cast<std::chrono::microseconds>(timePoint.time_since_epoch()) %
std::chrono::seconds(1);

char bufferMilliseconds[millisecondsSize];
std::snprintf(bufferMilliseconds, sizeof(bufferMilliseconds), ".%06ld",
static_cast<long>(microseconds.count()));

std::strcat(bufferDate, bufferMilliseconds);
std::strcat(bufferDate, "Z");

const std::string dateStr(bufferDate);
#endif

json_["@timestamp"] = dateStr;
}

void ElasticSearchRecordable::SetObservedTimestamp(
Expand All @@ -216,23 +252,25 @@ void ElasticSearchRecordable::SetObservedTimestamp(

void ElasticSearchRecordable::SetSeverity(opentelemetry::logs::Severity severity) noexcept
{
auto &severityField = json_["log"]["level"];

// Convert the severity enum to a string
std::uint32_t severity_index = static_cast<std::uint32_t>(severity);
if (severity_index >= std::extent<decltype(opentelemetry::logs::SeverityNumToText)>::value)
{
std::stringstream sout;
sout << "Invalid severity(" << severity_index << ")";
json_["severity"] = sout.str();
severityField = sout.str();
}
else
{
json_["severity"] = opentelemetry::logs::SeverityNumToText[severity_index];
severityField = opentelemetry::logs::SeverityNumToText[severity_index];
}
}

void ElasticSearchRecordable::SetBody(const opentelemetry::common::AttributeValue &message) noexcept
{
WriteValue(message, "body");
WriteValue(message, "message");
}

void ElasticSearchRecordable::SetTraceId(const opentelemetry::trace::TraceId &trace_id) noexcept
Expand Down Expand Up @@ -275,7 +313,7 @@ void ElasticSearchRecordable::SetAttribute(
nostd::string_view key,
const opentelemetry::common::AttributeValue &value) noexcept
{
WriteKeyValue(key, value, "attributes");
WriteValue(value, key.data());
}

void ElasticSearchRecordable::SetResource(
Expand All @@ -291,7 +329,7 @@ void ElasticSearchRecordable::SetInstrumentationScope(
const opentelemetry::sdk::instrumentationscope::InstrumentationScope
&instrumentation_scope) noexcept
{
json_["name"] = instrumentation_scope.GetName();
json_["log"]["logger"] = instrumentation_scope.GetName();
}

} // namespace logs
Expand Down

1 comment on commit 5e62859

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'OpenTelemetry-cpp api Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite Current: 5e62859 Previous: 2c912d5 Ratio
BM_ProcYieldSpinLockThrashing/4/process_time/real_time 1.4781999588012695 ms/iter 0.6982234486362391 ms/iter 2.12

This comment was automatically generated by workflow using github-action-benchmark.

Please sign in to comment.