Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: [release-3.0.x] fix: Updated Loki Otlp Ingest Configuration #12669

Merged
merged 1 commit into from
Apr 18, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 72 additions & 61 deletions docs/sources/send-data/otel/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,21 @@ For ingesting logs to Loki using the OpenTelemetry Collector, you must use the [

When logs are ingested by Loki using an OpenTelemetry protocol (OTLP) ingestion endpoint, some of the data is stored as [Structured Metadata]({{< relref "../../get-started/labels/structured-metadata" >}}).

You must set `allow_structured_metadata` to `true` within your Loki config file. Otherwise, Loki will reject the log payload as malformed.

```yaml
limits_config:
allow_structured_metadata: true
```
## Configure the OpenTelemetry Collector to write logs into Loki
You need to make the following changes to the [OpenTelemetry Collector config](https://opentelemetry.io/docs/collector/configuration/) to write logs to Loki on its OTLP ingestion endpoint.
```yaml
exporters:
otlphttp:
endpoint: http://<loki-addr>/otlp
endpoint: http://<loki-addr>:3100/otlp
```
And enable it in `service.pipelines`:
Expand All @@ -50,7 +57,7 @@ exporters:
otlphttp:
auth:
authenticator: basicauth/otlp
endpoint: http://<loki-addr>/otlp
endpoint: http://<loki-addr>:3100/otlp
service:
extensions: [basicauth/otlp]
Expand Down Expand Up @@ -118,50 +125,52 @@ It currently only supports changing the storage of Attributes. Here is how the c

```yaml
# OTLP log ingestion configurations
otlp_config:
# Configuration for Resource Attributes to store them as index labels or
# Structured Metadata or drop them altogether
resource_attributes:
# Configure whether to ignore the default list of resource attributes set in
# 'distributor.otlp.default_resource_attributes_as_index_labels' to be
# stored as index labels and only use the given resource attributes config
[ignore_defaults: <boolean>]
[attributes_config: <list of attributes_configs>]
# Configuration for Scope Attributes to store them as Structured Metadata or
# drop them altogether
[scope_attributes: <list of attributes_configs>]
# Configuration for Log Attributes to store them as Structured Metadata or
# drop them altogether
[log_attributes: <list of attributes_configs>]
attributes_config:
# Configures action to take on matching Attributes. It allows one of
# [structured_metadata, drop] for all Attribute types. It additionally allows
# index_label action for Resource Attributes
[action: <string> | default = ""]
# List of attributes to configure how to store them or drop them altogether
[attributes: <list of strings>]
# Regex to choose attributes to configure how to store them or drop them
# altogether
[regex: <Regexp>]
limits_config:
otlp_config:
# Configuration for Resource Attributes to store them as index labels or
# Structured Metadata or drop them altogether
resource_attributes:
# Configure whether to ignore the default list of resource attributes set in
# 'distributor.otlp.default_resource_attributes_as_index_labels' to be
# stored as index labels and only use the given resource attributes config
[ignore_defaults: <boolean>]
[attributes_config: <list of attributes_configs>]
# Configuration for Scope Attributes to store them as Structured Metadata or
# drop them altogether
[scope_attributes: <list of attributes_configs>]
# Configuration for Log Attributes to store them as Structured Metadata or
# drop them altogether
[log_attributes: <list of attributes_configs>]
attributes_config:
# Configures action to take on matching Attributes. It allows one of
# [structured_metadata, drop] for all Attribute types. It additionally allows
# index_label action for Resource Attributes
[action: <string> | default = ""]
# List of attributes to configure how to store them or drop them altogether
[attributes: <list of strings>]
# Regex to choose attributes to configure how to store them or drop them
# altogether
[regex: <Regexp>]
```

Here are some example configs to change the default mapping of OTLP to Loki format:

#### Example 1:

```yaml
otlp_config:
resource_attributes:
attributes_config:
- action: index_label
attributes:
- service.group
limits_config:
otlp_config:
resource_attributes:
attributes_config:
- action: index_label
attributes:
- service.group
```

With the example config, here is how various kinds of Attributes would be stored:
Expand All @@ -172,12 +181,13 @@ With the example config, here is how various kinds of Attributes would be stored
#### Example 2:

```yaml
otlp_config:
resource_attributes:
ignore_defaults: true
attributes_config:
- action: index_label
regex: service.group
limits_config:
otlp_config:
resource_attributes:
ignore_defaults: true
attributes_config:
- action: index_label
regex: service.group
```

With the example config, here is how various kinds of Attributes would be stored:
Expand All @@ -188,25 +198,26 @@ With the example config, here is how various kinds of Attributes would be stored
#### Example 2:

```yaml
otlp_config:
resource_attributes:
attributes_config:
- action: index_label
regex: service.group
scope_attributes:
- action: drop
attributes:
- method.name
log_attributes:
- action: structured_metadata
attributes:
- user.id
- action: drop
regex: .*
limits_config:
otlp_config:
resource_attributes:
attributes_config:
- action: index_label
regex: service.group
scope_attributes:
- action: drop
attributes:
- method.name
log_attributes:
- action: structured_metadata
attributes:
- user.id
- action: drop
regex: .*
```

With the example config, here is how various kinds of Attributes would be stored:
* Store all 17 Resource Attributes mentioned earlier and `service.group` Resource Attribute as index labels.
* Store remaining Resource Attributes as Structured Metadata.
* Drop Scope Attribute named `method.name` and store all other Scope Attributes as Structured Metadata.
* Store Log Attribute named `user.id` as Structured Metadata and drop all other Log Attributes.
* Store Log Attribute named `user.id` as Structured Metadata and drop all other Log Attributes.
Loading