From dc84b17eac8648703f268d0a96bd56a09c532cf5 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Thu, 29 Aug 2024 15:27:10 +0100 Subject: [PATCH 01/39] added main page, added sandbox, moved community plugin to own page --- docs/sources/send-data/fluentbit/_index.md | 319 +++++------------- .../send-data/fluentbit/community-plugin.md | 283 ++++++++++++++++ .../fluentbit/fluentbit-loki-tutorial.md | 268 +++++++++++++++ 3 files changed, 644 insertions(+), 226 deletions(-) create mode 100644 docs/sources/send-data/fluentbit/community-plugin.md create mode 100644 docs/sources/send-data/fluentbit/fluentbit-loki-tutorial.md diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index 2c2845d766d3..4b62f858c164 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -9,274 +9,141 @@ weight: 500 # Fluent Bit client [Fluent Bit](https://fluentbit.io/) is a fast and lightweight logs and metrics processor and forwarder that can be configured with the Grafana Fluent Bit Plugin described here or with the [Fluent-bit Loki output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/loki) to ship logs to Loki. -This plugin has more configuration options compared to the built-in Fluent Bit Loki plugin. -You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. - -{{< youtube id="s43IBSVyTpQ" >}} - -## Usage -### Docker - -You can run a Fluent Bit container with Loki output plugin pre-installed using our [Docker Hub](https://hub.docker.com/r/grafana/fluent-bit-plugin-loki) image: - -```bash -docker run -v /var/log:/var/log \ - -e LOG_PATH="/var/log/*.log" -e LOKI_URL="http://localhost:3100/loki/api/v1/push" \ - grafana/fluent-bit-plugin-loki:latest -``` - -Or, an alternative is to run the fluent-bit container using [Docker Hub](https://hub.docker.com/r/fluent/fluent-bit) image: - -### Docker Container Logs +You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. -To ship logs from Docker containers to Grafana Cloud using Fluent Bit, you can use the Fluent Bit Docker image and configure it to forward logs directly to Grafana Cloud's Loki. Below is a step-by-step guide on setting up Fluent Bit for this purpose. +{{< admonition type="note" >}} +We now recommend using the official Fluent Bit Loki output plugin to send logs to Loki rather than the Grafana Fluent Bit Plugin. The official Fluent Bit Loki output plugin is more feature-rich and actively maintained. -#### Prerequisites +For more information on the Fluent Bit Loki output plugin, see the [Fluent-bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki). -- Docker is installed on your machine. -- You have a Grafana Cloud account with access to Loki. +Below you will find several examples on how to configure Fluent Bit to send logs to Loki using the official Fluent Bit Loki output plugin. +{{< /admonition >}} -#### Configuration +## Configuration -1. Create a Fluent Bit configuration file named `fluent-bit.conf` with the following content, which defines the input from Docker container logs and sets up the output to send logs to your Grafana Cloud Loki instance: +All configuration options for the Fluent Bit Loki output plugin are documented in the [Fluent Bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki#configuration-parameters). - ```ini - [SERVICE] - Flush 1 - Log_Level info +Here is a generic example for connecting Fluent Bit to Loki hosted on Grafana Cloud: - [INPUT] - Name tail - Path /var/lib/docker/containers/*/*.log - Parser docker - Tag docker.* +```conf + [OUTPUT] + Name loki + Match * + Host logs-prod-eu-west-0.grafana.net + port 443 + tls on + tls.verify on + http_user XXX + http_passwd XXX +``` - [OUTPUT] - Name loki - Match * - Host logs-prod-006.grafana.net - Port 443 - TLS On - TLS.Verify On - HTTP_User 478625 - HTTP_Passwd YOUR_GRAFANA_CLOUD_API_KEY - Labels job=fluentbit +Replace `Host`, `http_user`, and `http_passwd` with your Grafana Cloud Loki endpoint and credentials. -### Kubernetes -You can run Fluent Bit as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs. +## Usage examples -To do so you can use the [Fluent Bit helm chart](https://github.com/fluent/helm-charts) with the following `values.yaml` changing the value of `FLUENT_LOKI_URL`: +Here are some examples of how to use Fluent Bit to send logs to Loki. -```yaml -image: - # Here we use the Docker image which has the plugin installed - repository: grafana/fluent-bit-plugin-loki - tag: main-e2ed1c0 - -args: - - "-e" - - "/fluent-bit/bin/out_grafana_loki.so" - - --workdir=/fluent-bit/etc - - --config=/fluent-bit/etc/conf/fluent-bit.conf - -env: - # Note that for security reasons you should fetch the credentials through a Kubernetes Secret https://kubernetes.io/docs/concepts/configuration/secret/ . You may use the envFrom for this. - - name: FLUENT_LOKI_URL - value: https://user:pass@your-loki.endpoint/loki/api/v1/push +### Tail Docker logs -config: - inputs: | - [INPUT] - Name tail - Tag kube.* - Path /var/log/containers/*.log - # Be aware that local clusters like docker-desktop or kind use the docker log format and not the cri (https://docs.fluentbit.io/manual/installation/kubernetes#container-runtime-interface-cri-parser) - multiline.parser docker, cri - Mem_Buf_Limit 5MB - Skip_Long_Lines On - - outputs: | - [Output] - Name grafana-loki - Match kube.* - Url ${FLUENT_LOKI_URL} - Labels {job="fluent-bit"} - LabelKeys level,app # this sets the values for actual Loki streams and the other labels are converted to structured_metadata https://grafana.com/docs/loki//get-started/labels/structured-metadata/ - BatchWait 1 - BatchSize 1001024 - LineFormat json - LogLevel info - AutoKubernetesLabels true -``` +Here is a quick start example to run Fluent Bit in a Docker container, collect Docker logs, and send them to a local Loki instance. ```bash -helm repo add fluent https://fluent.github.io/helm-charts -helm repo update -helm install fluent-bit fluent/fluent-bit -f values.yaml +docker run -v /var/lib/docker/containers:/var/lib/docker/containers fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -i tail -p Path="/var/lib/docker/containers/*/*.log" -p Parser=docker -p Tag="docker.*" -o loki -p host=loki -p port=3100 -p labels="agent=fluend-bit,env=docker" ``` -By default it will collect all containers logs and extract labels from Kubernetes API (`container_name`, `namespace`, etc..). - -If you also want to host your Loki instance inside the cluster install the [official Loki helm chart](https://grafana.com/docs/loki//setup/install/helm/). +In this example, we are using the `tail` input plugin to collect Docker logs and the `loki` output plugin to send logs to Loki. Note it is recommended to use a configuration file to define the input and output plugins. The `-p` flag is used to pass configuration parameters to the plugins. -### AWS Elastic Container Service (ECS) +#### Configuration file (Alternative to command line arguments) -You can use fluent-bit Loki Docker image as a Firelens log router in AWS ECS. -For more information about this see our [AWS documentation]({{< relref "../promtail/cloud/ecs" >}}) +Create a configuration file `fluent-bit.conf` with the following content: -### Local - -First, you need to follow the [instructions](https://github.com/grafana/loki/blob/main/clients/cmd/fluent-bit/README.md) in order to build the plugin dynamic library. +```conf +[INPUT] + Name tail + Path /var/lib/docker/containers/*/*.log + Parser docker + Tag docker.* + +[OUTPUT] + Name loki + Match * + Host loki + Port 3100 + Labels agent=fluend-bit,env=docker +``` -The assuming you have Fluent Bit installed in your `$PATH` you can run the plugin using: +Run Fluent Bit with the configuration file: ```bash -fluent-bit -e /path/to/built/out_grafana_loki.so -c fluent-bit.conf +docker run -v /var/lib/docker/containers:/var/lib/docker/containers -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf ``` -You can also adapt your plugins.conf, removing the need to change the command line options: +### Docker Events -```conf -[PLUGINS] - Path /path/to/built/out_grafana_loki.so -``` - -## Configuration Options - -| Key | Description | Default | -|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------| -| Url | Url of Loki server API endpoint. | http://localhost:3100/loki/api/v1/push | -| TenantID | The tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no `X-Scope-OrgID` header is sent. | "" | -| BatchWait | Time to wait before send a log batch to Loki, full or not. | 1s | -| BatchSize | Log batch size to send a log batch to Loki (unit: Bytes). | 10 KiB (10 * 1024 Bytes) | -| Timeout | Maximum time to wait for Loki server to respond to a request. | 10s | -| MinBackoff | Initial backoff time between retries. | 500ms | -| MaxBackoff | Maximum backoff time between retries. | 5m | -| MaxRetries | Maximum number of retries when sending batches. Setting it to `0` will retry indefinitely. | 10 | -| Labels | labels for API requests. | {job="fluent-bit"} | -| LogLevel | LogLevel for plugin logger. | "info" | -| RemoveKeys | Specify removing keys. | none | -| AutoKubernetesLabels | If set to true, it will add all Kubernetes labels to Loki labels | false | -| LabelKeys | Comma separated list of keys to use as stream labels. All other keys will be placed into the log line. LabelKeys is deactivated when using `LabelMapPath` label mapping configuration. | none | -| LineFormat | Format to use when flattening the record to a log line. Valid values are "json" or "key_value". If set to "json" the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to "key_value", the log line will be each item in the record concatenated together (separated by a single space) in the format =. | json | -| DropSingleKey | If set to true and after extracting label_keys a record only has a single key remaining, the log line sent to Loki will just be the value of the record key. | true | -| LabelMapPath | Path to a json file defining how to transform nested records. | none | -| Buffer | Enable buffering mechanism | false | -| BufferType | Specify the buffering mechanism to use (currently only dque is implemented). | dque | -| DqueDir | Path to the directory for queued logs | /tmp/flb-storage/loki | -| DqueSegmentSize | Segment size in terms of number of records per segment | 500 | -| DqueSync | Whether to fsync each queue change. Specify no fsync with "normal", and fsync with "full". | "normal" | -| DqueName | Queue name, must be uniq per output | dque | - -### Labels - -Labels are used to [query logs]({{< relref "../../query" >}}) `{container_name="nginx", cluster="us-west1"}`, they are usually metadata about the workload producing the log stream (`instance`, `container_name`, `region`, `cluster`, `level`). In Loki labels are indexed consequently you should be cautious when choosing them (high cardinality label values can have performance drastic impact). - -You can use `Labels`, `RemoveKeys` , `LabelKeys` and `LabelMapPath` to how the output plugin will perform labels extraction. - -### AutoKubernetesLabels - -If set to true, it will add all Kubernetes labels to Loki labels automatically and ignore parameters `LabelKeys`, LabelMapPath. - -### LabelMapPath - -When using the `Parser` and `Filter` plugins Fluent Bit can extract and add data to the current record/log data. While Loki labels are key value pair, record data can be nested structures. -You can pass a JSON file that defines how to extract labels from each record. Each json key from the file will be matched with the log record to find label values. Values from the configuration are used as label names. - -Considering the record below : - -```json -{ - "kubernetes": { - "container_name": "promtail", - "pod_name": "promtail-xxx", - "namespace_name": "prod", - "labels" : { - "team": "x-men" - } - }, - "HOSTNAME": "docker-desktop", - "log" : "a log line", - "time": "20190926T152206Z" -} -``` +Here is a quick start example to run Fluent Bit in a Docker container, collect docker events, and send them to a local Loki instance. -and a LabelMap file as follow : - -```json -{ - "kubernetes": { - "container_name": "container", - "pod_name": "pod", - "namespace_name": "namespace", - "labels" : { - "team": "team" - } - } -} +```bash +docker run -v /var/run/docker.sock:/var/run/docker.sock fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -i docker_events -o loki -p host=loki -p port=3100 -p labels="agent=fluend-bit,env=docker" ``` -The labels extracted will be `{team="x-men", container="promtail", pod="promtail-xxx", namespace="prod"}`. - -If you don't want the `kubernetes` and `HOSTNAME` fields to appear in the log line you can use the `RemoveKeys` configuration field. (e.g. `RemoveKeys kubernetes,HOSTNAME`). - -### Buffering +In this example, we are using the `docker_events` input plugin to collect Docker events and the `loki` output plugin to send logs to Loki. Note it is recommended to use a configuration file to define the input and output plugins. The `-p` flag is used to pass configuration parameters to the plugins. -Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. The Loki output plugin can be blocked by the Loki client because of its design: +#### Configuration file (Alternative to command line arguments) -- If the BatchSize is over the limit, the output plugin pauses receiving new records until the pending batch is successfully sent to the server -- If the Loki server is unreachable (retry 429s, 500s and connection-level errors), the output plugin blocks new records until the Loki server is available again, and the pending batch is successfully sent to the server or as long as the maximum number of attempts has been reached within configured back-off mechanism +Create a configuration file `fluent-bit.conf` with the following content: -The blocking state with some of the input plugins is not acceptable, because it can have an undesirable side effect on the part that generates the logs. Fluent Bit implements a buffering mechanism that is based on parallel processing. Therefore, it cannot send logs in order. There are two ways of handling the out-of-order logs: +```conf +[INPUT] + Name docker_events + +[OUTPUT] + Name loki + Match * + Host loki + Port 3100 + Labels agent=fluent-bit,env=docker +``` -- Configure Loki to [accept out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes). +Run Fluent Bit with the configuration file: -- Configure the Loki output plugin to use the buffering mechanism based on [`dque`](https://github.com/joncrlsn/dque), which is compatible with the Loki server strict time ordering: +```bash +docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf +``` - ```properties - [Output] - Name grafana-loki - Match * - Url http://localhost:3100/loki/api/v1/push - Buffer true - DqueSegmentSize 8096 - DqueDir /tmp/flb-storage/buffer - DqueName loki.0 - ``` +### Kubernetes -### Configuration examples +The recommended way to collect logs from Kubernetes with Fluent Bit is to use the helm chart provided by the Fluent Bit project. The helm chart is available at [https://github.com/fluent/helm-charts](https://github.com/fluent/helm-charts). -To configure the Loki output plugin add this section to fluent-bit.conf +Here is an example of how to deploy the Fluent Bit helm chart to collect logs from Kubernetes and send them to Loki: -```properties -[Output] - Name grafana-loki - Match * - Url http://localhost:3100/loki/api/v1/push - BatchWait 1s - BatchSize 30720 - # (30KiB) - Labels {test="fluent-bit-go", lang="Golang"} - RemoveKeys key1,key2 - LabelKeys key3,key4 - LineFormat key_value +1. Add the Fluent Bit helm repository: + +```bash +helm repo add fluent https://fluent.github.io/helm-charts ``` +1. Create a `values.yaml` file with the following content: -```properties -[Output] - Name grafana-loki - Match * - Url http://localhost:3100/loki/api/v1/push - BatchWait 1s - BatchSize 30720 # (30KiB) - AutoKubernetesLabels true - RemoveKeys key1,key2 +```yaml +config: + outputs: | + [OUTPUT] + Name loki + Match * + Host logs-prod-eu-west-0.grafana.net + port 443 + tls on + tls.verify on + http_user XXX + http_passwd XXX + Labels agent=fluend-bit ``` -A full [example configuration file](https://github.com/grafana/loki/blob/main/clients/cmd/fluent-bit/fluent-bit.conf) is also available in the Loki repository. +Note we are only updating the `outputs` section of the Fluent Bit configuration. This is to replace the default output plugin with the Loki output plugin. If you need to update other parts of the Fluent Bit configuration checkout the [Fluent Bit values file reference](https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml). -### Running multiple plugin instances +1. Deploy the Fluent Bit helm chart: -You can run multiple plugin instances in the same fluent-bit process, for example if you want to push to different Loki servers or route logs into different Loki tenant IDs. To do so, add additional `[Output]` sections. +```bash +helm install fluent-bit fluent/fluent-bit -f values.yaml +``` diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md new file mode 100644 index 000000000000..0553177c1779 --- /dev/null +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -0,0 +1,283 @@ +--- +title: Fluent Bit Community Plugin +menuTitle: Fluent Bit Community Plugin +description: Provides instructions for how to install, configure, and use the Fluent Bit Community plugin to send logs to Loki. +aliases: +- ../clients/fluentbit/ +weight: 500 +--- +# Fluent Bit client + +{{< admonition type="note" >}} + +It is now recommended to use the official [Fluent Bit Loki plugin]({{< relref "./_index.md" >}}). The official plugin is more feature-rich and has better support for features such as structured meta data. The community plugin is still available for use, but it is no longer actively maintained. + +{{< /admonition >}} + +[Fluent Bit](https://fluentbit.io/) is a fast and lightweight logs and metrics processor and forwarder that can be configured with the Grafana Fluent Bit Plugin described here or with the [Fluent-bit Loki output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/loki) to ship logs to Loki. +This plugin has more configuration options compared to the built-in Fluent Bit Loki plugin. +You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. + +{{< youtube id="s43IBSVyTpQ" >}} + +## Usage + +### Docker + +You can run a Fluent Bit container with Loki output plugin pre-installed using our [Docker Hub](https://hub.docker.com/r/grafana/fluent-bit-plugin-loki) image: + +```bash +docker run -v /var/log:/var/log \ + -e LOG_PATH="/var/log/*.log" -e LOKI_URL="http://localhost:3100/loki/api/v1/push" \ + grafana/fluent-bit-plugin-loki:latest +``` + +Or, an alternative is to run the fluent-bit container using [Docker Hub](https://hub.docker.com/r/fluent/fluent-bit) image: + +### Docker Container Logs + +To ship logs from Docker containers to Grafana Cloud using Fluent Bit, you can use the Fluent Bit Docker image and configure it to forward logs directly to Grafana Loki. Below is a step-by-step guide on setting up Fluent Bit for this purpose. + +#### Prerequisites + +- Docker is installed on your machine. +- Running instance of Loki OSS. + +#### Configuration + +1. Create a Fluent Bit configuration file named `fluent-bit.conf` with the following content, which defines the input from Docker container logs and sets up the output to send logs to your Grafana Cloud Loki instance: + + ```ini + [SERVICE] + Flush 1 + Log_Level info + + [INPUT] + Name tail + Path /var/lib/docker/containers/*/*.log + Parser docker + Tag docker.* + + [OUTPUT] + Name grafana-loki + Match * + Url http://localhost:3100/loki/api/v1/push + Labels {job="fluentbit"} + +### Kubernetes + +You can run Fluent Bit as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs. + +To do so you can use the [Fluent Bit helm chart](https://github.com/fluent/helm-charts) with the following `values.yaml` changing the value of `FLUENT_LOKI_URL`: + +```yaml +image: + # Here we use the Docker image which has the plugin installed + repository: grafana/fluent-bit-plugin-loki + tag: main-e2ed1c0 + +args: + - "-e" + - "/fluent-bit/bin/out_grafana_loki.so" + - --workdir=/fluent-bit/etc + - --config=/fluent-bit/etc/conf/fluent-bit.conf + +env: + # Note that for security reasons you should fetch the credentials through a Kubernetes Secret https://kubernetes.io/docs/concepts/configuration/secret/ . You may use the envFrom for this. + - name: FLUENT_LOKI_URL + value: https://user:pass@your-loki.endpoint/loki/api/v1/push + +config: + inputs: | + [INPUT] + Name tail + Tag kube.* + Path /var/log/containers/*.log + # Be aware that local clusters like docker-desktop or kind use the docker log format and not the cri (https://docs.fluentbit.io/manual/installation/kubernetes#container-runtime-interface-cri-parser) + multiline.parser docker, cri + Mem_Buf_Limit 5MB + Skip_Long_Lines On + + outputs: | + [Output] + Name grafana-loki + Match kube.* + Url ${FLUENT_LOKI_URL} + Labels {job="fluent-bit"} + LabelKeys level,app # this sets the values for actual Loki streams and the other labels are converted to structured_metadata https://grafana.com/docs/loki//get-started/labels/structured-metadata/ + BatchWait 1 + BatchSize 1001024 + LineFormat json + LogLevel info + AutoKubernetesLabels true +``` + +```bash +helm repo add fluent https://fluent.github.io/helm-charts +helm repo update +helm install fluent-bit fluent/fluent-bit -f values.yaml +``` + +By default it will collect all containers logs and extract labels from Kubernetes API (`container_name`, `namespace`, etc..). + +If you also want to host your Loki instance inside the cluster install the [official Loki helm chart](https://grafana.com/docs/loki//setup/install/helm/). + +### AWS Elastic Container Service (ECS) + +You can use fluent-bit Loki Docker image as a Firelens log router in AWS ECS. +For more information about this see our [AWS documentation]({{< relref "../promtail/cloud/ecs" >}}) + +### Local + +First, you need to follow the [instructions](https://github.com/grafana/loki/blob/main/clients/cmd/fluent-bit/README.md) in order to build the plugin dynamic library. + +The assuming you have Fluent Bit installed in your `$PATH` you can run the plugin using: + +```bash +fluent-bit -e /path/to/built/out_grafana_loki.so -c fluent-bit.conf +``` + +You can also adapt your plugins.conf, removing the need to change the command line options: + +```conf +[PLUGINS] + Path /path/to/built/out_grafana_loki.so +``` + +## Configuration Options + +| Key | Description | Default | +|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------| +| Url | Url of Loki server API endpoint. | http://localhost:3100/loki/api/v1/push | +| TenantID | The tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no `X-Scope-OrgID` header is sent. | "" | +| BatchWait | Time to wait before send a log batch to Loki, full or not. | 1s | +| BatchSize | Log batch size to send a log batch to Loki (unit: Bytes). | 10 KiB (10 * 1024 Bytes) | +| Timeout | Maximum time to wait for Loki server to respond to a request. | 10s | +| MinBackoff | Initial backoff time between retries. | 500ms | +| MaxBackoff | Maximum backoff time between retries. | 5m | +| MaxRetries | Maximum number of retries when sending batches. Setting it to `0` will retry indefinitely. | 10 | +| Labels | labels for API requests. | {job="fluent-bit"} | +| LogLevel | LogLevel for plugin logger. | "info" | +| RemoveKeys | Specify removing keys. | none | +| AutoKubernetesLabels | If set to true, it will add all Kubernetes labels to Loki labels | false | +| LabelKeys | Comma separated list of keys to use as stream labels. All other keys will be placed into the log line. LabelKeys is deactivated when using `LabelMapPath` label mapping configuration. | none | +| LineFormat | Format to use when flattening the record to a log line. Valid values are "json" or "key_value". If set to "json" the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to "key_value", the log line will be each item in the record concatenated together (separated by a single space) in the format =. | json | +| DropSingleKey | If set to true and after extracting label_keys a record only has a single key remaining, the log line sent to Loki will just be the value of the record key. | true | +| LabelMapPath | Path to a json file defining how to transform nested records. | none | +| Buffer | Enable buffering mechanism | false | +| BufferType | Specify the buffering mechanism to use (currently only dque is implemented). | dque | +| DqueDir | Path to the directory for queued logs | /tmp/flb-storage/loki | +| DqueSegmentSize | Segment size in terms of number of records per segment | 500 | +| DqueSync | Whether to fsync each queue change. Specify no fsync with "normal", and fsync with "full". | "normal" | +| DqueName | Queue name, must be uniq per output | dque | + +### Labels + +Labels are used to [query logs]({{< relref "../../query" >}}) `{container_name="nginx", cluster="us-west1"}`, they are usually metadata about the workload producing the log stream (`instance`, `container_name`, `region`, `cluster`, `level`). In Loki labels are indexed consequently you should be cautious when choosing them (high cardinality label values can have performance drastic impact). + +You can use `Labels`, `RemoveKeys` , `LabelKeys` and `LabelMapPath` to how the output plugin will perform labels extraction. + +### AutoKubernetesLabels + +If set to true, it will add all Kubernetes labels to Loki labels automatically and ignore parameters `LabelKeys`, LabelMapPath. + +### LabelMapPath + +When using the `Parser` and `Filter` plugins Fluent Bit can extract and add data to the current record/log data. While Loki labels are key value pair, record data can be nested structures. +You can pass a JSON file that defines how to extract labels from each record. Each json key from the file will be matched with the log record to find label values. Values from the configuration are used as label names. + +Considering the record below : + +```json +{ + "kubernetes": { + "container_name": "promtail", + "pod_name": "promtail-xxx", + "namespace_name": "prod", + "labels" : { + "team": "x-men" + } + }, + "HOSTNAME": "docker-desktop", + "log" : "a log line", + "time": "20190926T152206Z" +} +``` + +and a LabelMap file as follow : + +```json +{ + "kubernetes": { + "container_name": "container", + "pod_name": "pod", + "namespace_name": "namespace", + "labels" : { + "team": "team" + } + } +} +``` + +The labels extracted will be `{team="x-men", container="promtail", pod="promtail-xxx", namespace="prod"}`. + +If you don't want the `kubernetes` and `HOSTNAME` fields to appear in the log line you can use the `RemoveKeys` configuration field. (e.g. `RemoveKeys kubernetes,HOSTNAME`). + +### Buffering + +Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. The Loki output plugin can be blocked by the Loki client because of its design: + +- If the BatchSize is over the limit, the output plugin pauses receiving new records until the pending batch is successfully sent to the server +- If the Loki server is unreachable (retry 429s, 500s and connection-level errors), the output plugin blocks new records until the Loki server is available again, and the pending batch is successfully sent to the server or as long as the maximum number of attempts has been reached within configured back-off mechanism + +The blocking state with some of the input plugins is not acceptable, because it can have an undesirable side effect on the part that generates the logs. Fluent Bit implements a buffering mechanism that is based on parallel processing. Therefore, it cannot send logs in order. There are two ways of handling the out-of-order logs: + +- Configure Loki to [accept out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes). + +- Configure the Loki output plugin to use the buffering mechanism based on [`dque`](https://github.com/joncrlsn/dque), which is compatible with the Loki server strict time ordering: + + ```properties + [Output] + Name grafana-loki + Match * + Url http://localhost:3100/loki/api/v1/push + Buffer true + DqueSegmentSize 8096 + DqueDir /tmp/flb-storage/buffer + DqueName loki.0 + ``` + +### Configuration examples + +To configure the Loki output plugin add this section to fluent-bit.conf + +```properties +[Output] + Name grafana-loki + Match * + Url http://localhost:3100/loki/api/v1/push + BatchWait 1s + BatchSize 30720 + # (30KiB) + Labels {test="fluent-bit-go", lang="Golang"} + RemoveKeys key1,key2 + LabelKeys key3,key4 + LineFormat key_value +``` + +```properties +[Output] + Name grafana-loki + Match * + Url http://localhost:3100/loki/api/v1/push + BatchWait 1s + BatchSize 30720 # (30KiB) + AutoKubernetesLabels true + RemoveKeys key1,key2 +``` + +A full [example configuration file](https://github.com/grafana/loki/blob/main/clients/cmd/fluent-bit/fluent-bit.conf) is also available in the Loki repository. + +### Running multiple plugin instances + +You can run multiple plugin instances in the same fluent-bit process, for example if you want to push to different Loki servers or route logs into different Loki tenant IDs. To do so, add additional `[Output]` sections. diff --git a/docs/sources/send-data/fluentbit/fluentbit-loki-tutorial.md b/docs/sources/send-data/fluentbit/fluentbit-loki-tutorial.md new file mode 100644 index 000000000000..7911fc685754 --- /dev/null +++ b/docs/sources/send-data/fluentbit/fluentbit-loki-tutorial.md @@ -0,0 +1,268 @@ +--- +title: Sending logs to Loki using Fluent Bit tutorial +menuTitle: Sending logs to Loki using Fluent Bit tutorial +description: Sending logs to Loki using Fluent Bit using the official Fluent Bit Loki output plugin. +weight: 250 +killercoda: + title: Sending logs to Loki using Fluent Bit tutorial + description: Sending logs to Loki using Fluent Bit using the official Fluent Bit Loki output plugin. + preprocessing: + substitutions: + - regexp: loki-fundamentals-fluent-bit-1 + replacement: loki-fundamentals_fluent-bit_1 + - regexp: docker compose + replacement: docker-compose + backend: + imageid: ubuntu +--- + + + +# Sending logs to Loki using Fluent Bit tutorial + +In this tutorial, you will learn how to send logs to Loki using Fluent Bit. Fluent Bit is a lightweight and fast log processor and forwarder that can collect, process, and deliver logs to various destinations. We will use the official Fluent Bit Loki output plugin to send logs to Loki. + + + + +## Dependencies + +Before you begin, ensure you have the following to run the demo: + +- Docker +- Docker Compose + +{{< admonition type="tip" >}} +Alternatively, you can try out this example in our interactive learning environment: [Sending logs to Loki using Fluent Bit tutorial](https://killercoda.com/grafana-labs/course/loki/fluentbit-loki-tutorial). + +It's a fully configured environment with all the dependencies already installed. + +![Interactive](/media/docs/loki/loki-ile.svg) + +Provide feedback, report bugs, and raise issues in the [Grafana Killercoda repository](https://github.com/grafana/killercoda). +{{< /admonition >}} + + + +## Scenario + +In this scenario, we have a microservices application called the Carnivourse Greenhouse. This application consists of the following services: + +- **User Service:** Manages user data and authentication for the application. Such as creating users and logging in. +- **Plant Service:** Manages the creation of new plants and updates other services when a new plant is created. +- **Simulation Service:** Generates sensor data for each plant. +- **Websocket Service:** Manages the websocket connections for the application. +- **Bug Service:** A service that when enabled, randomly causes services to fail and generate additional logs. +- **Main App:** The main application that ties all the services together. +- **Database:** A database that stores user and plant data. + +Each service has been instrumented with the fluent bit logging framework to generate logs. If you would like to learn more about how the Carnivorous Greenhouse application was instrumented with Fluent Bit, refer to the [Carnivorous Greenhouse repository](https://github.com/grafana/loki-fundamentals/blob/fluentbit-official/greenhouse/loggingfw.py). + + + + + +## Step 1: Environment setup + +In this step, we will set up our environment by cloning the repository that contains our demo application and spinning up our observability stack using Docker Compose. + +1. To get started, clone the repository that contains our demo application: + + ```bash + git clone -b fluentbit-official https://github.com/grafana/loki-fundamentals.git + ``` + +1. Next we will spin up our observability stack using Docker Compose: + + ```bash + docker compose -f loki-fundamentals/docker-compose.yml up -d + ``` + + This will spin up the following services: + ```console + ✔ Container loki-fundamentals-grafana-1 Started + ✔ Container loki-fundamentals-loki-1 Started + ✔ Container loki-fundamentals-fluent-bit-1 Started + ``` +Once we have finished configuring the Fluent Bit agent and sending logs to Loki, we will be able to view the logs in Grafana. To check if Grafana is up and running, navigate to the following URL: [http://localhost:3000](http://localhost:3000) + + + + +## Step 2: Configure Fluent Bit to send logs to Loki + +To configure Fluent Bit to receive logs from our application, we need to provide a configuration file. This configuration file will define the components and their relationships. We will build the entire observability pipeline within this configuration file. + +### Open your Code Editor and Locate the `fluent-bit.conf` file + +Fluent Bit requires a configuration file to define the components and their relationships. The configuration file is written using Fluent Bit configuration syntax. We will build the entire observability pipeline within this configuration file. To start, we will open the `fluent-bit.conf` file in the code editor: + +{{< docs/ignore >}} +> Note: Killercoda has an inbuilt Code editor which can be accessed via the `Editor` tab. +1. Expand the `loki-fundamentals` directory in the file explorer of the `Editor` tab. +1. Locate the `fluent-bit.conf` file in the top level directory, `loki-fundamentals'. +1. Click on the `fluent-bit.conf` file to open it in the code editor. +{{< /docs/ignore >}} + + +1. Open the `loki-fundamentals` directory in a code editor of your choice. +1. Locate the `fluent-bit.conf` file in the `loki-fundamentals` directory (Top level directory). +1. Click on the `fluent-bit.conf` file to open it in the code editor. + + +You will copy all of the configuration snippets into the `fluent-bit.conf` file. + +### Receiving Fluent Bit protocal logs + +The first step is to configure Fluent Bit to receive logs from the Carnivorous Greenhouse application. Since the application is instrumented with Fluent Bit logging framework, it will send logs using the forward protocal (unique to Fluent Bit). We will use the `forward` input plugin to receive logs from the application. + +Now add the following configuration to the `fluent-bit.conf` file: +```conf +[INPUT] + Name forward + Listen 0.0.0.0 + Port 24224 +``` + +In this configuration: +- `Name`: The name of the input plugin. In this case, we are using the `forward` input plugin. +- `Listen`: The IP address to listen on. In this case, we are listening on all IP addresses. +- `Port`: The port to listen on. In this case, we are listening on port `24224`. + +For more information on the `forward` input plugin, see the [Fluent Bit Forward documentation](https://docs.fluentbit.io/manual/pipeline/inputs/forward). + + + +### Export logs to Loki using the official Loki output plugin + +Lastly, we will configure Fluent Bit to export logs to Loki using the official Loki output plugin. The Loki output plugin allows you to send logs or events to a Loki service. It supports data enrichment with Kubernetes labels, custom label keys and structured metadata. + +Add the following configuration to the `fluent-bit.conf` file: +```conf +[OUTPUT] + name loki + match service.** + host loki + port 3100 + labels agent=fluent-bit + label_map_path /fluent-bit/etc/conf/logmap.json +``` + +In this configuration: +- `Name`: The name of the output plugin. In this case, we are using the `loki` output plugin. +- `Match`: The tag to match. In this case, we are matching all logs with the tag `service.**`. +- `Host`: The hostname of the Loki service. In this case, we are using the hostname `loki`. +- `Port`: The port of the Loki service. In this case, we are using port `3100`. +- `Labels`: Additional labels to add to the logs. In this case, we are adding the label `agent=fluent-bit`. +- `Label_map_path`: The path to the label map file. In this case, we are using the file `logmap.json`. + +For more information on the `loki` output plugin, see the [Fluent Bit Loki documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki). + +#### `logmap.json` file + +The `logmap.json` file is used to map the log fields to the Loki labels. In this tutorial we have pre-filled the `logmap.json` file with the following configuration: +```json +{ +"service": "service_name", +"instance_id": "instance_id" + } +``` +This configuration maps the `service` field to the Loki label `service_name` and the `instance_id` field to the Loki label `instance_id`. + + +### Reload the Fluent Bit configuration + +After adding the configuration to the `fluent-bit.conf` file, you will need to reload the Fluent Bit configuration. To reload the configuration, run the following command: + +```bash +docker restart loki-fundamentals-fluent-bit-1 +``` +To verify that the configuration has been loaded successfully, you can check the Fluent Bit logs by running the following command: + +```bash +docker logs loki-fundamentals-fluent-bit-1 +``` + +## Stuck? Need help? + +If you get stuck or need help creating the configuration, you can copy and replace the entire `config.alloy` using the completed configuration file: + +```bash +cp loki-fundamentals/completed/fluent-bit.conf loki-fundamentals/fluent-bit.conf +docker restart loki-fundamentals-fluent-bit-1 +``` + + + + + +## Step 3: Start the Carnivorous Greenhouse + +In this step, we will start the Carnivorous Greenhouse application. To start the application, run the following command: + +{{< admonition type="note" >}} +This docker-compose file relies on the `loki-fundamentals_loki` docker network. If you have not started the observability stack, you will need to start it first. +{{< /admonition >}} + + +{{< docs/ignore >}} + +> Note: This docker-compose file relies on the `loki-fundamentals_loki` docker network. If you have not started the observability stack, you will need to start it first. + +{{< /docs/ignore >}} + +```bash +docker compose -f loki-fundamentals/greenhouse/docker-compose-micro.yml up -d --build +``` + +This will start the following services: +```bash + ✔ Container greenhouse-db-1 Started + ✔ Container greenhouse-websocket_service-1 Started + ✔ Container greenhouse-bug_service-1 Started + ✔ Container greenhouse-user_service-1 Started + ✔ Container greenhouse-plant_service-1 Started + ✔ Container greenhouse-simulation_service-1 Started + ✔ Container greenhouse-main_app-1 Started +``` + +Once started, you can access the Carnivorous Greenhouse application at [http://localhost:5005](http://localhost:5005). Generate some logs by interacting with the application in the following ways: + +- Create a user +- Log in +- Create a few plants to monitor +- Enable bug mode to activate the bug service. This will cause services to fail and generate additional logs. + +Finally to view the logs in Loki, navigate to the Loki Logs Explore view in Grafana at [http://localhost:3000/a/grafana-lokiexplore-app/explore](http://localhost:3000/a/grafana-lokiexplore-app/explore). + + + + + + +## Summary + +In this tutorial, you learned how to send logs to Loki using Fluent Bit. You configured Fluent Bit to receive logs from the Carnivorous Greenhouse application and export logs to Loki using the official Loki output plugin. Where to next? + +{{< docs/ignore >}} + +### Back to Docs +Head back to where you started from to continue with the Loki documentation: [Loki documentation](https://grafana.com/docs/loki/latest/send-data/alloy) + +{{< /docs/ignore >}} + + +## Further reading + +For more information on Fluent Bit, refer to the following resources: +- [Fluent Bit documentation](https://docs.fluentbit.io/manual/) +- [Other examples of Fluent Bit configurations](https://grafana.com/docs/loki/latest/send-data/fluentbit/) + +## Complete metrics, logs, traces, and profiling example + +If you would like to use a demo that includes Mimir, Loki, Tempo, and Grafana, you can use [Introduction to Metrics, Logs, Traces, and Profiling in Grafana](https://github.com/grafana/intro-to-mlt). `Intro-to-mltp` provides a self-contained environment for learning about Mimir, Loki, Tempo, and Grafana. + +The project includes detailed explanations of each component and annotated configurations for a single-instance deployment. Data from `intro-to-mltp` can also be pushed to Grafana Cloud. + + + From 5bc8efaa205f278c1a14414a833b68d4cfd3f237 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Thu, 29 Aug 2024 15:46:25 +0100 Subject: [PATCH 02/39] Updated titles --- docs/sources/send-data/fluentbit/_index.md | 6 +++--- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index 4b62f858c164..76f59fb3f5c4 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -1,12 +1,12 @@ --- -title: Fluent Bit client -menuTitle: Fluent Bit +title: Fluent Bit Loki output plugin +menuTitle: Fluent Bit Loki output plugin description: Provides instructions for how to install, configure, and use the Fluent Bit client to send logs to Loki. aliases: - ../clients/fluentbit/ weight: 500 --- -# Fluent Bit client +# Fluent Bit Loki output plugin [Fluent Bit](https://fluentbit.io/) is a fast and lightweight logs and metrics processor and forwarder that can be configured with the Grafana Fluent Bit Plugin described here or with the [Fluent-bit Loki output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/loki) to ship logs to Loki. diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 0553177c1779..b70aa218d048 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -6,7 +6,7 @@ aliases: - ../clients/fluentbit/ weight: 500 --- -# Fluent Bit client +# Fluent Bit Community Plugin {{< admonition type="note" >}} From 6b6595fcc986a47649469978e7df2f96c08bc646 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 18 Sep 2024 14:50:47 +0100 Subject: [PATCH 03/39] Added changes --- docs/sources/send-data/fluentbit/_index.md | 22 +++++++++---------- .../send-data/fluentbit/community-plugin.md | 2 +- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index 76f59fb3f5c4..cd58849a4443 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -1,6 +1,6 @@ --- title: Fluent Bit Loki output plugin -menuTitle: Fluent Bit Loki output plugin +menuTitle: Fluent Bit description: Provides instructions for how to install, configure, and use the Fluent Bit client to send logs to Loki. aliases: - ../clients/fluentbit/ @@ -13,7 +13,7 @@ weight: 500 You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. {{< admonition type="note" >}} -We now recommend using the official Fluent Bit Loki output plugin to send logs to Loki rather than the Grafana Fluent Bit Plugin. The official Fluent Bit Loki output plugin is more feature-rich and actively maintained. +We recommend using the official Fluent Bit Loki output plugin to send logs to Loki rather than the Grafana Fluent Bit Plugin. The official Fluent Bit Loki output plugin has native support for Loki, is more feature-rich, and is actively maintained. For more information on the Fluent Bit Loki output plugin, see the [Fluent-bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki). @@ -30,7 +30,7 @@ Here is a generic example for connecting Fluent Bit to Loki hosted on Grafana Cl [OUTPUT] Name loki Match * - Host logs-prod-eu-west-0.grafana.net + Host YourHostname.company.com port 443 tls on tls.verify on @@ -47,7 +47,7 @@ Here are some examples of how to use Fluent Bit to send logs to Loki. ### Tail Docker logs -Here is a quick start example to run Fluent Bit in a Docker container, collect Docker logs, and send them to a local Loki instance. +Here is an example to run Fluent Bit in a Docker container, collect Docker logs, and send them to a local Loki instance. ```bash docker run -v /var/lib/docker/containers:/var/lib/docker/containers fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -i tail -p Path="/var/lib/docker/containers/*/*.log" -p Parser=docker -p Tag="docker.*" -o loki -p host=loki -p port=3100 -p labels="agent=fluend-bit,env=docker" @@ -80,9 +80,9 @@ Run Fluent Bit with the configuration file: docker run -v /var/lib/docker/containers:/var/lib/docker/containers -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf ``` -### Docker Events +### Collect Docker events -Here is a quick start example to run Fluent Bit in a Docker container, collect docker events, and send them to a local Loki instance. +Here is an example to run Fluent Bit in a Docker container, collect docker events, and send them to a local Loki instance. ```bash docker run -v /var/run/docker.sock:/var/run/docker.sock fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -i docker_events -o loki -p host=loki -p port=3100 -p labels="agent=fluend-bit,env=docker" @@ -112,13 +112,13 @@ Run Fluent Bit with the configuration file: docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf ``` -### Kubernetes +### Collect Kubernetes logs -The recommended way to collect logs from Kubernetes with Fluent Bit is to use the helm chart provided by the Fluent Bit project. The helm chart is available at [https://github.com/fluent/helm-charts](https://github.com/fluent/helm-charts). +The recommended way to collect logs from Kubernetes with Fluent Bit is to use the Helm chart provided by the Fluent Bit project. The Helm chart is available at [https://github.com/fluent/helm-charts](https://github.com/fluent/helm-charts). Here is an example of how to deploy the Fluent Bit helm chart to collect logs from Kubernetes and send them to Loki: -1. Add the Fluent Bit helm repository: +1. Add the Fluent Bit Helm repository: ```bash helm repo add fluent https://fluent.github.io/helm-charts @@ -131,7 +131,7 @@ config: [OUTPUT] Name loki Match * - Host logs-prod-eu-west-0.grafana.net + Host YourHost.Company.net port 443 tls on tls.verify on @@ -142,7 +142,7 @@ config: Note we are only updating the `outputs` section of the Fluent Bit configuration. This is to replace the default output plugin with the Loki output plugin. If you need to update other parts of the Fluent Bit configuration checkout the [Fluent Bit values file reference](https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml). -1. Deploy the Fluent Bit helm chart: +1. Deploy the Fluent Bit Helm chart: ```bash helm install fluent-bit fluent/fluent-bit -f values.yaml diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index b70aa218d048..0428e8cc9966 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -10,7 +10,7 @@ weight: 500 {{< admonition type="note" >}} -It is now recommended to use the official [Fluent Bit Loki plugin]({{< relref "./_index.md" >}}). The official plugin is more feature-rich and has better support for features such as structured meta data. The community plugin is still available for use, but it is no longer actively maintained. +We recommend using the official [Fluent Bit Loki plugin]({{< relref "./_index.md" >}}). The official plugin is more feature-rich and has better support for features such as structured meta data. The community plugin is still available for use, but it is no longer actively maintained. {{< /admonition >}} From f78ae464ee876112c62f1945bef6a141b42e71b3 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 18 Sep 2024 14:58:46 +0100 Subject: [PATCH 04/39] Updated admintion in index --- docs/sources/send-data/fluentbit/_index.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index cd58849a4443..0119e9499137 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -13,11 +13,9 @@ weight: 500 You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. {{< admonition type="note" >}} -We recommend using the official Fluent Bit Loki output plugin to send logs to Loki rather than the Grafana Fluent Bit Plugin. The official Fluent Bit Loki output plugin has native support for Loki, is more feature-rich, and is actively maintained. +There are two Fluent Bit plugins for Loki: the officially maintained plugin `loki` described in this document and the `grafana-loki` plugin. We recommend using the `loki` plugin as it's officially maintained by the Fluent Bit project. -For more information on the Fluent Bit Loki output plugin, see the [Fluent-bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki). - -Below you will find several examples on how to configure Fluent Bit to send logs to Loki using the official Fluent Bit Loki output plugin. +For more information, see the [Fluent Bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki). Note that the `grafana-loki` plugin is no longer actively maintained. {{< /admonition >}} ## Configuration From ab80e14b115bc89c6a4d204776bdcc4628a6bbfa Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 18 Sep 2024 15:00:38 +0100 Subject: [PATCH 05/39] fixed wording --- docs/sources/send-data/fluentbit/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index 0119e9499137..a6ab7159b2ee 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -13,7 +13,7 @@ weight: 500 You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. {{< admonition type="note" >}} -There are two Fluent Bit plugins for Loki: the officially maintained plugin `loki` described in this document and the `grafana-loki` plugin. We recommend using the `loki` plugin as it's officially maintained by the Fluent Bit project. +There are two Fluent Bit plugins for Loki: the officially maintained plugin `loki` described within this page and the `grafana-loki` plugin. We recommend using the `loki` plugin as it's officially maintained by the Fluent Bit project. For more information, see the [Fluent Bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki). Note that the `grafana-loki` plugin is no longer actively maintained. {{< /admonition >}} From 2264a06b1796c37cc083e073d9e758840fc997d9 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 18 Sep 2024 15:01:19 +0100 Subject: [PATCH 06/39] updated notice --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 0428e8cc9966..e6e0e1035f8e 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -8,7 +8,7 @@ weight: 500 --- # Fluent Bit Community Plugin -{{< admonition type="note" >}} +{{< admonition type="warning" >}} We recommend using the official [Fluent Bit Loki plugin]({{< relref "./_index.md" >}}). The official plugin is more feature-rich and has better support for features such as structured meta data. The community plugin is still available for use, but it is no longer actively maintained. From 7f9fedc349361734b29f10fc0e8e63076df43556 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Fri, 20 Sep 2024 11:22:46 +0100 Subject: [PATCH 07/39] Update docs/sources/send-data/fluentbit/_index.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index a6ab7159b2ee..6a8a5245a032 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -114,7 +114,7 @@ docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/fluent-bit.con The recommended way to collect logs from Kubernetes with Fluent Bit is to use the Helm chart provided by the Fluent Bit project. The Helm chart is available at [https://github.com/fluent/helm-charts](https://github.com/fluent/helm-charts). -Here is an example of how to deploy the Fluent Bit helm chart to collect logs from Kubernetes and send them to Loki: +Here is an example of how to deploy the Fluent Bit Helm chart to collect logs from Kubernetes and send them to Loki: 1. Add the Fluent Bit Helm repository: From 9fb12a8849950cb02a0a41136c0859ef18c6c822 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 20 Sep 2024 12:06:38 +0100 Subject: [PATCH 08/39] added index and updated language around offical vs community --- docs/sources/send-data/fluentbit/_index.md | 143 ++--------------- .../send-data/fluentbit/community-plugin.md | 6 +- ...utorial.md => fluent-bit-loki-tutorial.md} | 4 +- .../send-data/fluentbit/fluent-bit-plugin.md | 151 ++++++++++++++++++ 4 files changed, 165 insertions(+), 139 deletions(-) rename docs/sources/send-data/fluentbit/{fluentbit-loki-tutorial.md => fluent-bit-loki-tutorial.md} (99%) create mode 100644 docs/sources/send-data/fluentbit/fluent-bit-plugin.md diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index a6ab7159b2ee..dfc0c8b585df 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -1,147 +1,24 @@ --- -title: Fluent Bit Loki output plugin +title: menuTitle: Fluent Bit description: Provides instructions for how to install, configure, and use the Fluent Bit client to send logs to Loki. aliases: - ../clients/fluentbit/ weight: 500 --- -# Fluent Bit Loki output plugin +# Fluent Bit -[Fluent Bit](https://fluentbit.io/) is a fast and lightweight logs and metrics processor and forwarder that can be configured with the Grafana Fluent Bit Plugin described here or with the [Fluent-bit Loki output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/loki) to ship logs to Loki. +[Fluent Bit](https://fluentbit.io/) is a fast, lightweight logs and metrics agent. It is a CNCF graduated sub-project under the umbrella of Fluentd. Fluent Bit is licensed under the terms of the Apache License v2.0. -You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. +When using Fluent Bit to ship logs to Loki, you can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. -{{< admonition type="note" >}} -There are two Fluent Bit plugins for Loki: the officially maintained plugin `loki` described within this page and the `grafana-loki` plugin. We recommend using the `loki` plugin as it's officially maintained by the Fluent Bit project. +There are two Fluent Bit plugins for Loki: -For more information, see the [Fluent Bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki). Note that the `grafana-loki` plugin is no longer actively maintained. -{{< /admonition >}} +1. The integrated `loki` [plugin]({{< relref "./fluent-bit-plugin.md" >}}), which is officially maintained by the Fluent Bit project. +2. The `grafana-loki` [plugin]({{< relref "./community-plugin.md" >}}), an alternative community plugin by Grafana Labs. -## Configuration +We recommend using the `loki` plugin as this provides the most complete feature set and is actively maintained by the Fluent Bit project. -All configuration options for the Fluent Bit Loki output plugin are documented in the [Fluent Bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki#configuration-parameters). +## Tutorial -Here is a generic example for connecting Fluent Bit to Loki hosted on Grafana Cloud: - -```conf - [OUTPUT] - Name loki - Match * - Host YourHostname.company.com - port 443 - tls on - tls.verify on - http_user XXX - http_passwd XXX -``` - -Replace `Host`, `http_user`, and `http_passwd` with your Grafana Cloud Loki endpoint and credentials. - - -## Usage examples - -Here are some examples of how to use Fluent Bit to send logs to Loki. - -### Tail Docker logs - -Here is an example to run Fluent Bit in a Docker container, collect Docker logs, and send them to a local Loki instance. - -```bash -docker run -v /var/lib/docker/containers:/var/lib/docker/containers fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -i tail -p Path="/var/lib/docker/containers/*/*.log" -p Parser=docker -p Tag="docker.*" -o loki -p host=loki -p port=3100 -p labels="agent=fluend-bit,env=docker" -``` - -In this example, we are using the `tail` input plugin to collect Docker logs and the `loki` output plugin to send logs to Loki. Note it is recommended to use a configuration file to define the input and output plugins. The `-p` flag is used to pass configuration parameters to the plugins. - -#### Configuration file (Alternative to command line arguments) - -Create a configuration file `fluent-bit.conf` with the following content: - -```conf -[INPUT] - Name tail - Path /var/lib/docker/containers/*/*.log - Parser docker - Tag docker.* - -[OUTPUT] - Name loki - Match * - Host loki - Port 3100 - Labels agent=fluend-bit,env=docker -``` - -Run Fluent Bit with the configuration file: - -```bash -docker run -v /var/lib/docker/containers:/var/lib/docker/containers -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf -``` - -### Collect Docker events - -Here is an example to run Fluent Bit in a Docker container, collect docker events, and send them to a local Loki instance. - -```bash -docker run -v /var/run/docker.sock:/var/run/docker.sock fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -i docker_events -o loki -p host=loki -p port=3100 -p labels="agent=fluend-bit,env=docker" -``` - -In this example, we are using the `docker_events` input plugin to collect Docker events and the `loki` output plugin to send logs to Loki. Note it is recommended to use a configuration file to define the input and output plugins. The `-p` flag is used to pass configuration parameters to the plugins. - -#### Configuration file (Alternative to command line arguments) - -Create a configuration file `fluent-bit.conf` with the following content: - -```conf -[INPUT] - Name docker_events - -[OUTPUT] - Name loki - Match * - Host loki - Port 3100 - Labels agent=fluent-bit,env=docker -``` - -Run Fluent Bit with the configuration file: - -```bash -docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf -``` - -### Collect Kubernetes logs - -The recommended way to collect logs from Kubernetes with Fluent Bit is to use the Helm chart provided by the Fluent Bit project. The Helm chart is available at [https://github.com/fluent/helm-charts](https://github.com/fluent/helm-charts). - -Here is an example of how to deploy the Fluent Bit helm chart to collect logs from Kubernetes and send them to Loki: - -1. Add the Fluent Bit Helm repository: - -```bash -helm repo add fluent https://fluent.github.io/helm-charts -``` -1. Create a `values.yaml` file with the following content: - -```yaml -config: - outputs: | - [OUTPUT] - Name loki - Match * - Host YourHost.Company.net - port 443 - tls on - tls.verify on - http_user XXX - http_passwd XXX - Labels agent=fluend-bit -``` - -Note we are only updating the `outputs` section of the Fluent Bit configuration. This is to replace the default output plugin with the Loki output plugin. If you need to update other parts of the Fluent Bit configuration checkout the [Fluent Bit values file reference](https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml). - -1. Deploy the Fluent Bit Helm chart: - -```bash -helm install fluent-bit fluent/fluent-bit -f values.yaml -``` +To get started with the `loki` plugin, follow the [Sending logs to Loki using Fluent Bit tutorial]({{< relref "./fluent-bit-loki-tutorial.md" >}}). \ No newline at end of file diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index e6e0e1035f8e..37b04942ef2f 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -10,13 +10,11 @@ weight: 500 {{< admonition type="warning" >}} -We recommend using the official [Fluent Bit Loki plugin]({{< relref "./_index.md" >}}). The official plugin is more feature-rich and has better support for features such as structured meta data. The community plugin is still available for use, but it is no longer actively maintained. +We recommend using the official [Fluent Bit Loki plugin]({{< relref "./fluent-bit-plugin.md" >}}). The official plugin is more feature-rich and has better support for features such as structured metadata. The community plugin is still available for use, but it's no longer actively maintained. {{< /admonition >}} -[Fluent Bit](https://fluentbit.io/) is a fast and lightweight logs and metrics processor and forwarder that can be configured with the Grafana Fluent Bit Plugin described here or with the [Fluent-bit Loki output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/loki) to ship logs to Loki. -This plugin has more configuration options compared to the built-in Fluent Bit Loki plugin. -You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. +The Fluent Bit Community plugin by Grafana Labs (`grafana-loki`) provided an alternative way to send logs to Loki. Although very similar to the [official plugin]({{< relref "./fluent-bit-plugin.md" >}}) there are some differences in the configuration options. This page provides instructions for how to install, configure, and use the Fluent Bit Community plugin to send logs to Loki. Although the plugin is no longer actively maintained, this documentation is still available for reference. {{< youtube id="s43IBSVyTpQ" >}} diff --git a/docs/sources/send-data/fluentbit/fluentbit-loki-tutorial.md b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md similarity index 99% rename from docs/sources/send-data/fluentbit/fluentbit-loki-tutorial.md rename to docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md index 7911fc685754..1b12187bf464 100644 --- a/docs/sources/send-data/fluentbit/fluentbit-loki-tutorial.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md @@ -1,6 +1,6 @@ --- title: Sending logs to Loki using Fluent Bit tutorial -menuTitle: Sending logs to Loki using Fluent Bit tutorial +menuTitle: Fluent Bit tutorial description: Sending logs to Loki using Fluent Bit using the official Fluent Bit Loki output plugin. weight: 250 killercoda: @@ -114,7 +114,7 @@ You will copy all of the configuration snippets into the `fluent-bit.conf` file. ### Receiving Fluent Bit protocal logs -The first step is to configure Fluent Bit to receive logs from the Carnivorous Greenhouse application. Since the application is instrumented with Fluent Bit logging framework, it will send logs using the forward protocal (unique to Fluent Bit). We will use the `forward` input plugin to receive logs from the application. +The first step is to configure Fluent Bit to receive logs from the Carnivorous Greenhouse application. Since the application is instrumented with Fluent Bit logging framework, it will send logs using the forward protocol (unique to Fluent Bit). We will use the `forward` input plugin to receive logs from the application. Now add the following configuration to the `fluent-bit.conf` file: ```conf diff --git a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md new file mode 100644 index 000000000000..c00beb6d30da --- /dev/null +++ b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md @@ -0,0 +1,151 @@ +--- +title: Fluent Bit Loki output plugin +menuTitle: Fluent Bit +description: Provides instructions for how to install, configure, and use the Fluent Bit client to send logs to Loki. +aliases: +- ../clients/fluentbit/ +weight: 500 +--- +# Fluent Bit Loki output plugin + +[Fluent Bit](https://fluentbit.io/) is a fast and lightweight logs and metrics processor and forwarder that can be configured with the [Fluent-bit Loki output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/loki) to ship logs to Loki. + +You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines. + +{{< admonition type="note" >}} +There are two Fluent Bit plugins for Loki: the officially maintained plugin `loki` and the `grafana-loki` plugin. We recommend using the `loki` plugin described within this page as it's officially maintained by the Fluent Bit project. + +For more information, see the [Fluent Bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki). Note that the `grafana-loki` plugin is no longer actively maintained. +{{< /admonition >}} + +## Configuration + +All configuration options for the Fluent Bit Loki output plugin are documented in the [Fluent Bit Loki output plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki#configuration-parameters). + +Here is a generic example for connecting Fluent Bit to Loki hosted on Grafana Cloud: + +```conf + [OUTPUT] + Name loki + Match * + Host YourHostname.company.com + port 443 + tls on + tls.verify on + http_user XXX + http_passwd XXX +``` + +Replace `Host`, `http_user`, and `http_passwd` with your Grafana Cloud Loki endpoint and credentials. + + +## Usage examples + +Here are some examples of how to use Fluent Bit to send logs to Loki. + +### Tail Docker logs + +Here is an example to run Fluent Bit in a Docker container, collect Docker logs, and send them to a local Loki instance. + +```bash +docker run -v /var/lib/docker/containers:/var/lib/docker/containers fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -i tail -p Path="/var/lib/docker/containers/*/*.log" -p Parser=docker -p Tag="docker.*" -o loki -p host=loki -p port=3100 -p labels="agent=fluend-bit,env=docker" +``` + +In this example, we are using the `tail` input plugin to collect Docker logs and the `loki` output plugin to send logs to Loki. Note it is recommended to use a configuration file to define the input and output plugins. The `-p` flag is used to pass configuration parameters to the plugins. + +#### Configuration file (Alternative to command line arguments) + +Create a configuration file `fluent-bit.conf` with the following content: + +```conf +[INPUT] + Name tail + Path /var/lib/docker/containers/*/*.log + Parser docker + Tag docker.* + +[OUTPUT] + Name loki + Match * + Host loki + Port 3100 + Labels agent=fluend-bit,env=docker +``` + +Run Fluent Bit with the configuration file: + +```bash +docker run -v /var/lib/docker/containers:/var/lib/docker/containers -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf +``` + +### Collect Docker events + +Here is an example to run Fluent Bit in a Docker container, collect docker events, and send them to a local Loki instance. + +```bash +docker run -v /var/run/docker.sock:/var/run/docker.sock fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -i docker_events -o loki -p host=loki -p port=3100 -p labels="agent=fluend-bit,env=docker" +``` + +In this example, we are using the `docker_events` input plugin to collect Docker events and the `loki` output plugin to send logs to Loki. Note it is recommended to use a configuration file to define the input and output plugins. The `-p` flag is used to pass configuration parameters to the plugins. + +#### Configuration file (Alternative to command line arguments) + +Create a configuration file `fluent-bit.conf` with the following content: + +```conf +[INPUT] + Name docker_events + +[OUTPUT] + Name loki + Match * + Host loki + Port 3100 + Labels agent=fluent-bit,env=docker +``` + +Run Fluent Bit with the configuration file: + +```bash +docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:latest /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf +``` + +### Collect Kubernetes logs + +The recommended way to collect logs from Kubernetes with Fluent Bit is to use the Helm chart provided by the Fluent Bit project. The Helm chart is available at [https://github.com/fluent/helm-charts](https://github.com/fluent/helm-charts). + +Here is an example of how to deploy the Fluent Bit helm chart to collect logs from Kubernetes and send them to Loki: + +1. Add the Fluent Bit Helm repository: + +```bash +helm repo add fluent https://fluent.github.io/helm-charts +``` +1. Create a `values.yaml` file with the following content: + +```yaml +config: + outputs: | + [OUTPUT] + Name loki + Match * + Host YourHost.Company.net + port 443 + tls on + tls.verify on + http_user XXX + http_passwd XXX + Labels agent=fluend-bit +``` + +Note we are only updating the `outputs` section of the Fluent Bit configuration. This is to replace the default output plugin with the Loki output plugin. If you need to update other parts of the Fluent Bit configuration checkout the [Fluent Bit values file reference](https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml). + +1. Deploy the Fluent Bit Helm chart: + +```bash +helm install fluent-bit fluent/fluent-bit -f values.yaml +``` + +## Where next? + +- [Sending logs to Loki using Fluent Bit tutorial]({{< relref "./fluent-bit-loki-tutorial.md" >}}) \ No newline at end of file From 2808fe3cc3d2534450bc89c1850352cae841b393 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 20 Sep 2024 12:15:05 +0100 Subject: [PATCH 09/39] fixed title --- docs/sources/send-data/fluentbit/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index 29dc0e2c2970..dade804fa1b1 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -1,5 +1,5 @@ --- -title: +title: Fluent Bit menuTitle: Fluent Bit description: Provides instructions for how to install, configure, and use the Fluent Bit client to send logs to Loki. aliases: From c6be58101889b8fb0438a58e387f362229192885 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 1 Oct 2024 16:01:29 +0100 Subject: [PATCH 10/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 37b04942ef2f..53c810a68902 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -64,7 +64,7 @@ To ship logs from Docker containers to Grafana Cloud using Fluent Bit, you can u ### Kubernetes -You can run Fluent Bit as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs. +You can run Fluent Bit as a [daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs. To do so you can use the [Fluent Bit helm chart](https://github.com/fluent/helm-charts) with the following `values.yaml` changing the value of `FLUENT_LOKI_URL`: From e0ed07cfc9dc115c52eea24cc7f80bfdddbf66fc Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 1 Oct 2024 16:01:35 +0100 Subject: [PATCH 11/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 53c810a68902..c74e7d1b323b 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -66,7 +66,7 @@ To ship logs from Docker containers to Grafana Cloud using Fluent Bit, you can u You can run Fluent Bit as a [daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs. -To do so you can use the [Fluent Bit helm chart](https://github.com/fluent/helm-charts) with the following `values.yaml` changing the value of `FLUENT_LOKI_URL`: +To do so you can use the [Fluent Bit Helm chart](https://github.com/fluent/helm-charts) with the following `values.yaml` changing the value of `FLUENT_LOKI_URL`: ```yaml image: From 1c404ed674f74a58c9a6ca9a7c12acff9a62900e Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 1 Oct 2024 16:01:41 +0100 Subject: [PATCH 12/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index c74e7d1b323b..d5037c2a1159 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -116,7 +116,7 @@ helm repo update helm install fluent-bit fluent/fluent-bit -f values.yaml ``` -By default it will collect all containers logs and extract labels from Kubernetes API (`container_name`, `namespace`, etc..). +By default it will collect all containers logs and extract labels from Kubernetes API (`container_name`, `namespace`, etc.). If you also want to host your Loki instance inside the cluster install the [official Loki helm chart](https://grafana.com/docs/loki//setup/install/helm/). From c85bae3b2fa96382889af7738e4ccf2a656fb4a9 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 1 Oct 2024 16:01:47 +0100 Subject: [PATCH 13/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index d5037c2a1159..876bcf3966ba 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -118,7 +118,7 @@ helm install fluent-bit fluent/fluent-bit -f values.yaml By default it will collect all containers logs and extract labels from Kubernetes API (`container_name`, `namespace`, etc.). -If you also want to host your Loki instance inside the cluster install the [official Loki helm chart](https://grafana.com/docs/loki//setup/install/helm/). +If you also want to host your Loki instance inside the cluster install the [official Loki Helm chart](https://grafana.com/docs/loki//setup/install/helm/). ### AWS Elastic Container Service (ECS) From 9ece7a2ffd98e8d368fbcd76e7131af12b972027 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 1 Oct 2024 16:01:54 +0100 Subject: [PATCH 14/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 876bcf3966ba..38994a6dff9b 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -122,7 +122,7 @@ If you also want to host your Loki instance inside the cluster install the [offi ### AWS Elastic Container Service (ECS) -You can use fluent-bit Loki Docker image as a Firelens log router in AWS ECS. +You can use the fluent-bit Loki Docker image as a Firelens log router in AWS ECS. For more information about this see our [AWS documentation]({{< relref "../promtail/cloud/ecs" >}}) ### Local From fbf91c13fb4ae5ce13b4d341b7eb3a8b4d00b3a3 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 1 Oct 2024 16:02:22 +0100 Subject: [PATCH 15/39] Update docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md Co-authored-by: J Stickler --- .../send-data/fluentbit/fluent-bit-loki-tutorial.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md index 1b12187bf464..119a495bc7a0 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md @@ -149,12 +149,12 @@ Add the following configuration to the `fluent-bit.conf` file: ``` In this configuration: -- `Name`: The name of the output plugin. In this case, we are using the `loki` output plugin. -- `Match`: The tag to match. In this case, we are matching all logs with the tag `service.**`. -- `Host`: The hostname of the Loki service. In this case, we are using the hostname `loki`. -- `Port`: The port of the Loki service. In this case, we are using port `3100`. -- `Labels`: Additional labels to add to the logs. In this case, we are adding the label `agent=fluent-bit`. -- `Label_map_path`: The path to the label map file. In this case, we are using the file `logmap.json`. +- `name`: The name of the output plugin. In this case, we are using the `loki` output plugin. +- `match`: The tag to match. In this case, we are matching all logs with the tag `service.**`. +- `host`: The hostname of the Loki service. In this case, we are using the hostname `loki`. +- `port`: The port of the Loki service. In this case, we are using port `3100`. +- `labels`: Additional labels to add to the logs. In this case, we are adding the label `agent=fluent-bit`. +- `label_map_path`: The path to the label map file. In this case, we are using the file `logmap.json`. For more information on the `loki` output plugin, see the [Fluent Bit Loki documentation](https://docs.fluentbit.io/manual/pipeline/outputs/loki). From 2075af8d1644838d1b67934f183a366cd087cdf6 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 1 Oct 2024 16:02:31 +0100 Subject: [PATCH 16/39] Update docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md index 119a495bc7a0..938679a4d1d8 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md @@ -93,7 +93,7 @@ Once we have finished configuring the Fluent Bit agent and sending logs to Loki, To configure Fluent Bit to receive logs from our application, we need to provide a configuration file. This configuration file will define the components and their relationships. We will build the entire observability pipeline within this configuration file. -### Open your Code Editor and Locate the `fluent-bit.conf` file +### Open your code editor and locate the `fluent-bit.conf` file Fluent Bit requires a configuration file to define the components and their relationships. The configuration file is written using Fluent Bit configuration syntax. We will build the entire observability pipeline within this configuration file. To start, we will open the `fluent-bit.conf` file in the code editor: From 74e82abd2253762cbb33dd974dec49c475766fea Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 1 Oct 2024 16:02:44 +0100 Subject: [PATCH 17/39] Update docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md index 938679a4d1d8..736e077c874e 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md @@ -247,7 +247,7 @@ In this tutorial, you learned how to send logs to Loki using Fluent Bit. You con {{< docs/ignore >}} ### Back to Docs -Head back to where you started from to continue with the Loki documentation: [Loki documentation](https://grafana.com/docs/loki/latest/send-data/alloy) +Head back to where you started from to continue with the [Loki documentation](https://grafana.com/docs/loki/latest/send-data/alloy). {{< /docs/ignore >}} From f4607d500878fe1325a3d0569daccfb1af767137 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 1 Oct 2024 16:02:58 +0100 Subject: [PATCH 18/39] Update docs/sources/send-data/fluentbit/fluent-bit-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/fluent-bit-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md index c00beb6d30da..fc86e8077426 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md @@ -138,7 +138,7 @@ config: Labels agent=fluend-bit ``` -Note we are only updating the `outputs` section of the Fluent Bit configuration. This is to replace the default output plugin with the Loki output plugin. If you need to update other parts of the Fluent Bit configuration checkout the [Fluent Bit values file reference](https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml). +Note we are only updating the `outputs` section of the Fluent Bit configuration. This is to replace the default output plugin with the Loki output plugin. If you need to update other parts of the Fluent Bit configuration refer to the [Fluent Bit values file reference](https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml). 1. Deploy the Fluent Bit Helm chart: From 449ae776c5b111618aef4bd6e7ed839f01586f53 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Tue, 1 Oct 2024 16:08:27 +0100 Subject: [PATCH 19/39] added changes to fluent bit example --- .../send-data/fluentbit/fluent-bit-loki-tutorial.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md index 736e077c874e..14960a14692a 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md @@ -228,10 +228,10 @@ This will start the following services: Once started, you can access the Carnivorous Greenhouse application at [http://localhost:5005](http://localhost:5005). Generate some logs by interacting with the application in the following ways: -- Create a user -- Log in -- Create a few plants to monitor -- Enable bug mode to activate the bug service. This will cause services to fail and generate additional logs. +1. Create a user +1. Log in +1. Create a few plants to monitor +1. Enable bug mode to activate the bug service. This will cause services to fail and generate additional logs. Finally to view the logs in Loki, navigate to the Loki Logs Explore view in Grafana at [http://localhost:3000/a/grafana-lokiexplore-app/explore](http://localhost:3000/a/grafana-lokiexplore-app/explore). From 12089f5d41886ea8be2872b10153400e5e29454d Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 7 Oct 2024 09:46:36 +0100 Subject: [PATCH 20/39] Update docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md index 14960a14692a..8abc4e6c68c7 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md @@ -100,7 +100,7 @@ Fluent Bit requires a configuration file to define the components and their rela {{< docs/ignore >}} > Note: Killercoda has an inbuilt Code editor which can be accessed via the `Editor` tab. 1. Expand the `loki-fundamentals` directory in the file explorer of the `Editor` tab. -1. Locate the `fluent-bit.conf` file in the top level directory, `loki-fundamentals'. +1. Locate the `fluent-bit.conf` file in the top level directory, `loki-fundamentals`. 1. Click on the `fluent-bit.conf` file to open it in the code editor. {{< /docs/ignore >}} From 9de2d04d72724efabe1c8b55e68ed77e86781c91 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 7 Oct 2024 09:46:43 +0100 Subject: [PATCH 21/39] Update docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md Co-authored-by: J Stickler --- .../sources/send-data/fluentbit/fluent-bit-loki-tutorial.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md index 8abc4e6c68c7..b76ab4a67a25 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md @@ -228,9 +228,9 @@ This will start the following services: Once started, you can access the Carnivorous Greenhouse application at [http://localhost:5005](http://localhost:5005). Generate some logs by interacting with the application in the following ways: -1. Create a user -1. Log in -1. Create a few plants to monitor +1. Create a user. +1. Log in. +1. Create a few plants to monitor. 1. Enable bug mode to activate the bug service. This will cause services to fail and generate additional logs. Finally to view the logs in Loki, navigate to the Loki Logs Explore view in Grafana at [http://localhost:3000/a/grafana-lokiexplore-app/explore](http://localhost:3000/a/grafana-lokiexplore-app/explore). From 75c8af50ce35f2b0c201674f8cd5873c169059f8 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 7 Oct 2024 09:47:01 +0100 Subject: [PATCH 22/39] Update docs/sources/send-data/fluentbit/_index.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index dade804fa1b1..540bcdb6200b 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -14,7 +14,7 @@ When using Fluent Bit to ship logs to Loki, you can define which log files you w There are two Fluent Bit plugins for Loki: -1. The integrated `loki` [plugin]({{< relref "./fluent-bit-plugin.md" >}}), which is officially maintained by the Fluent Bit project. +1. The integrated `loki` [plugin](https://grafana.com/docs/loki//send-data/fluentbit/fluent-bit-plugin/), which is officially maintained by the Fluent Bit project. 2. The `grafana-loki` [plugin]({{< relref "./community-plugin.md" >}}), an alternative community plugin by Grafana Labs. We recommend using the `loki` plugin as this provides the most complete feature set and is actively maintained by the Fluent Bit project. From 4f070f57a5f25f0379cabee3ba45826c760413f3 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 7 Oct 2024 09:47:21 +0100 Subject: [PATCH 23/39] Update docs/sources/send-data/fluentbit/_index.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index 540bcdb6200b..2cada8fbedb9 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -15,7 +15,7 @@ When using Fluent Bit to ship logs to Loki, you can define which log files you w There are two Fluent Bit plugins for Loki: 1. The integrated `loki` [plugin](https://grafana.com/docs/loki//send-data/fluentbit/fluent-bit-plugin/), which is officially maintained by the Fluent Bit project. -2. The `grafana-loki` [plugin]({{< relref "./community-plugin.md" >}}), an alternative community plugin by Grafana Labs. +2. The `grafana-loki` [plugin](https://grafana.com/docs/loki//send-data/fluentbit/community-plugin/), an alternative community plugin by Grafana Labs. We recommend using the `loki` plugin as this provides the most complete feature set and is actively maintained by the Fluent Bit project. From c77b795ac08ec42016212fbe2328fa2a28bf4cb7 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 7 Oct 2024 09:47:41 +0100 Subject: [PATCH 24/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 38994a6dff9b..b28ea646069e 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -10,7 +10,7 @@ weight: 500 {{< admonition type="warning" >}} -We recommend using the official [Fluent Bit Loki plugin]({{< relref "./fluent-bit-plugin.md" >}}). The official plugin is more feature-rich and has better support for features such as structured metadata. The community plugin is still available for use, but it's no longer actively maintained. +We recommend using the official [Fluent Bit Loki plugin](https://grafana.com/docs/loki//send-data/fluentbit/fluent-bit-plugin/). The official plugin is more feature-rich and has better support for features such as structured metadata. The community plugin is still available for use, but it's no longer actively maintained. {{< /admonition >}} From c7a8da76c429b3e44a40314d7d18a0e10fa98f5b Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 7 Oct 2024 09:47:49 +0100 Subject: [PATCH 25/39] Update docs/sources/send-data/fluentbit/_index.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index 2cada8fbedb9..5a6884efd5db 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -21,4 +21,4 @@ We recommend using the `loki` plugin as this provides the most complete feature ## Tutorial -To get started with the `loki` plugin, follow the [Sending logs to Loki using Fluent Bit tutorial]({{< relref "./fluent-bit-loki-tutorial.md" >}}). +To get started with the `loki` plugin, follow the [Sending logs to Loki using Fluent Bit tutorial](https://grafana.com/docs/loki//send-data/fluentbit/fluent-bit-loki-tutorial/). From 28270a198410156cdaf961521e4aab6825ffc636 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 7 Oct 2024 09:47:56 +0100 Subject: [PATCH 26/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index b28ea646069e..09c8c3a0cdcd 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -123,7 +123,7 @@ If you also want to host your Loki instance inside the cluster install the [offi ### AWS Elastic Container Service (ECS) You can use the fluent-bit Loki Docker image as a Firelens log router in AWS ECS. -For more information about this see our [AWS documentation]({{< relref "../promtail/cloud/ecs" >}}) +For more information about this see our [AWS documentation](https://grafana.com/docs/loki//send-data/promtail/cloud/ecs/). ### Local From 288ee319154f0742db1f3808f85ad395f8468dae Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 7 Oct 2024 09:48:05 +0100 Subject: [PATCH 27/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 09c8c3a0cdcd..4a82f5885ffb 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -14,7 +14,7 @@ We recommend using the official [Fluent Bit Loki plugin](https://grafana.com/doc {{< /admonition >}} -The Fluent Bit Community plugin by Grafana Labs (`grafana-loki`) provided an alternative way to send logs to Loki. Although very similar to the [official plugin]({{< relref "./fluent-bit-plugin.md" >}}) there are some differences in the configuration options. This page provides instructions for how to install, configure, and use the Fluent Bit Community plugin to send logs to Loki. Although the plugin is no longer actively maintained, this documentation is still available for reference. +The Fluent Bit Community plugin by Grafana Labs (`grafana-loki`) provided an alternative way to send logs to Loki. Although very similar to the [official plugin](https://grafana.com/docs/loki//send-data/fluentbit/fluent-bit-plugin/) there are some differences in the configuration options. This page provides instructions for how to install, configure, and use the Fluent Bit Community plugin to send logs to Loki. Although the plugin is no longer actively maintained, this documentation is still available for reference. {{< youtube id="s43IBSVyTpQ" >}} From b85b82313548b7db755055824b49eebb4bed7303 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 7 Oct 2024 09:48:15 +0100 Subject: [PATCH 28/39] Update docs/sources/send-data/fluentbit/fluent-bit-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/fluent-bit-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md index fc86e8077426..64112fe39832 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md @@ -148,4 +148,4 @@ helm install fluent-bit fluent/fluent-bit -f values.yaml ## Where next? -- [Sending logs to Loki using Fluent Bit tutorial]({{< relref "./fluent-bit-loki-tutorial.md" >}}) \ No newline at end of file +- [Sending logs to Loki using Fluent Bit tutorial](https://grafana.com/docs/loki//send-data/fluentbit/fluent-bit-loki-tutorial/) \ No newline at end of file From 3ccb18fa7d22a6743d766cd0f25e3caefb3d23b5 Mon Sep 17 00:00:00 2001 From: J Stickler Date: Tue, 8 Oct 2024 15:19:42 -0400 Subject: [PATCH 29/39] Apply suggestions from code review Copyedits --- .../send-data/fluentbit/community-plugin.md | 32 +++++++++---------- .../fluentbit/fluent-bit-loki-tutorial.md | 6 ++-- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 4a82f5885ffb..71f82803d61b 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -129,7 +129,7 @@ For more information about this see our [AWS documentation](https://grafana.com/ First, you need to follow the [instructions](https://github.com/grafana/loki/blob/main/clients/cmd/fluent-bit/README.md) in order to build the plugin dynamic library. -The assuming you have Fluent Bit installed in your `$PATH` you can run the plugin using: +Assuming you have Fluent Bit installed in your `$PATH` you can run the plugin using: ```bash fluent-bit -e /path/to/built/out_grafana_loki.so -c fluent-bit.conf @@ -154,35 +154,35 @@ You can also adapt your plugins.conf, removing the need to change the command li | MinBackoff | Initial backoff time between retries. | 500ms | | MaxBackoff | Maximum backoff time between retries. | 5m | | MaxRetries | Maximum number of retries when sending batches. Setting it to `0` will retry indefinitely. | 10 | -| Labels | labels for API requests. | {job="fluent-bit"} | -| LogLevel | LogLevel for plugin logger. | "info" | +| Labels | Labels for API requests. | {job="fluent-bit"} | +| LogLevel | LogLevel for plugin logger. | `info` | | RemoveKeys | Specify removing keys. | none | -| AutoKubernetesLabels | If set to true, it will add all Kubernetes labels to Loki labels | false | +| AutoKubernetesLabels | If set to `true`, it will add all Kubernetes labels to Loki labels. | false | | LabelKeys | Comma separated list of keys to use as stream labels. All other keys will be placed into the log line. LabelKeys is deactivated when using `LabelMapPath` label mapping configuration. | none | -| LineFormat | Format to use when flattening the record to a log line. Valid values are "json" or "key_value". If set to "json" the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to "key_value", the log line will be each item in the record concatenated together (separated by a single space) in the format =. | json | +| LineFormat | Format to use when flattening the record to a log line. Valid values are `json` or `key_value`. If set to `json` the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to `key_value`, the log line will be each item in the record concatenated together (separated by a single space) in the format <key>=<value>. | json | | DropSingleKey | If set to true and after extracting label_keys a record only has a single key remaining, the log line sent to Loki will just be the value of the record key. | true | | LabelMapPath | Path to a json file defining how to transform nested records. | none | -| Buffer | Enable buffering mechanism | false | -| BufferType | Specify the buffering mechanism to use (currently only dque is implemented). | dque | -| DqueDir | Path to the directory for queued logs | /tmp/flb-storage/loki | -| DqueSegmentSize | Segment size in terms of number of records per segment | 500 | -| DqueSync | Whether to fsync each queue change. Specify no fsync with "normal", and fsync with "full". | "normal" | -| DqueName | Queue name, must be uniq per output | dque | +| Buffer | Enable buffering mechanism. | false | +| BufferType | Specify the buffering mechanism to use (currently only `dque` is implemented). | dque | +| DqueDir | Path to the directory for queued logs. | /tmp/flb-storage/loki | +| DqueSegmentSize | Segment size in terms of number of records per segment. | 500 | +| DqueSync | Whether to fsync each queue change. Specify no fsync with `normal`, and fsync with `full`. | `normal` | +| DqueName | Queue name, must be unique per output. | dque | ### Labels -Labels are used to [query logs]({{< relref "../../query" >}}) `{container_name="nginx", cluster="us-west1"}`, they are usually metadata about the workload producing the log stream (`instance`, `container_name`, `region`, `cluster`, `level`). In Loki labels are indexed consequently you should be cautious when choosing them (high cardinality label values can have performance drastic impact). +Labels are used to [query logs](https://grafana.com/docs/loki//query/) `{container_name="nginx", cluster="us-west1"}`, they are usually metadata about the workload producing the log stream (`instance`, `container_name`, `region`, `cluster`, `level`). In Loki labels are indexed, so you should be cautious when choosing them. High cardinality label values can have drastic impact on performance. You can use `Labels`, `RemoveKeys` , `LabelKeys` and `LabelMapPath` to how the output plugin will perform labels extraction. ### AutoKubernetesLabels -If set to true, it will add all Kubernetes labels to Loki labels automatically and ignore parameters `LabelKeys`, LabelMapPath. +If set to `true`, `AutoKubernetesLabels` will add all Kubernetes labels to Loki labels automatically and ignore parameters `LabelKeys`, `LabelMapPath`. ### LabelMapPath -When using the `Parser` and `Filter` plugins Fluent Bit can extract and add data to the current record/log data. While Loki labels are key value pair, record data can be nested structures. -You can pass a JSON file that defines how to extract labels from each record. Each json key from the file will be matched with the log record to find label values. Values from the configuration are used as label names. +When using the `Parser` and `Filter` plugins Fluent Bit can extract and add data to the current record/log data. While Loki labels are key value pairs, record data can be nested structures. +You can pass a JSON file that defines how to extract labels from each record. Each JSON key from the file will be matched with the log record to find label values. Values from the configuration are used as label names. Considering the record below : @@ -219,7 +219,7 @@ and a LabelMap file as follow : The labels extracted will be `{team="x-men", container="promtail", pod="promtail-xxx", namespace="prod"}`. -If you don't want the `kubernetes` and `HOSTNAME` fields to appear in the log line you can use the `RemoveKeys` configuration field. (e.g. `RemoveKeys kubernetes,HOSTNAME`). +If you don't want the `kubernetes` and `HOSTNAME` fields to appear in the log line you can use the `RemoveKeys` configuration field. (for example, `RemoveKeys kubernetes,HOSTNAME`). ### Buffering diff --git a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md index b76ab4a67a25..c5564ca3f881 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-loki-tutorial.md @@ -46,7 +46,7 @@ Provide feedback, report bugs, and raise issues in the [Grafana Killercoda repos ## Scenario -In this scenario, we have a microservices application called the Carnivourse Greenhouse. This application consists of the following services: +In this scenario, we have a microservices application called the Carnivorous Greenhouse. This application consists of the following services: - **User Service:** Manages user data and authentication for the application. Such as creating users and logging in. - **Plant Service:** Manages the creation of new plants and updates other services when a new plant is created. @@ -56,7 +56,7 @@ In this scenario, we have a microservices application called the Carnivourse Gre - **Main App:** The main application that ties all the services together. - **Database:** A database that stores user and plant data. -Each service has been instrumented with the fluent bit logging framework to generate logs. If you would like to learn more about how the Carnivorous Greenhouse application was instrumented with Fluent Bit, refer to the [Carnivorous Greenhouse repository](https://github.com/grafana/loki-fundamentals/blob/fluentbit-official/greenhouse/loggingfw.py). +Each service has been instrumented with the Fluent Bit logging framework to generate logs. If you would like to learn more about how the Carnivorous Greenhouse application was instrumented with Fluent Bit, refer to the [Carnivorous Greenhouse repository](https://github.com/grafana/loki-fundamentals/blob/fluentbit-official/greenhouse/loggingfw.py). @@ -135,7 +135,7 @@ For more information on the `forward` input plugin, see the [Fluent Bit Forward ### Export logs to Loki using the official Loki output plugin -Lastly, we will configure Fluent Bit to export logs to Loki using the official Loki output plugin. The Loki output plugin allows you to send logs or events to a Loki service. It supports data enrichment with Kubernetes labels, custom label keys and structured metadata. +Lastly, we will configure Fluent Bit to export logs to Loki using the official Loki output plugin. The Loki output plugin allows you to send logs or events to a Loki service. It supports data enrichment with Kubernetes labels, custom label keys, and structured metadata. Add the following configuration to the `fluent-bit.conf` file: ```conf From 2868bde0da58223b608cd179191bbcaa0d91ae01 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:35:45 +0100 Subject: [PATCH 30/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 71f82803d61b..fff9d23fa1e0 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -1,5 +1,5 @@ --- -title: Fluent Bit Community Plugin +title: Fluent Bit community plugin menuTitle: Fluent Bit Community Plugin description: Provides instructions for how to install, configure, and use the Fluent Bit Community plugin to send logs to Loki. aliases: From 41f29ac08fd84a3183168c2fce56da5b2d564c89 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:36:00 +0100 Subject: [PATCH 31/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index fff9d23fa1e0..bac47fa9882d 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -6,7 +6,7 @@ aliases: - ../clients/fluentbit/ weight: 500 --- -# Fluent Bit Community Plugin +# Fluent Bit community plugin {{< admonition type="warning" >}} From 69170d718cad284fb28c12baa0b870e975775666 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:36:13 +0100 Subject: [PATCH 32/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index bac47fa9882d..467087fc853e 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -14,7 +14,7 @@ We recommend using the official [Fluent Bit Loki plugin](https://grafana.com/doc {{< /admonition >}} -The Fluent Bit Community plugin by Grafana Labs (`grafana-loki`) provided an alternative way to send logs to Loki. Although very similar to the [official plugin](https://grafana.com/docs/loki//send-data/fluentbit/fluent-bit-plugin/) there are some differences in the configuration options. This page provides instructions for how to install, configure, and use the Fluent Bit Community plugin to send logs to Loki. Although the plugin is no longer actively maintained, this documentation is still available for reference. +The Fluent Bit community plugin by Grafana Labs (`grafana-loki`) provided an alternative way to send logs to Loki. Although very similar to the [official plugin](https://grafana.com/docs/loki//send-data/fluentbit/fluent-bit-plugin/) there are some differences in the configuration options. This page provides instructions for how to install, configure, and use the Fluent Bit community plugin to send logs to Loki. Although the plugin is no longer actively maintained, this documentation is still available for reference. {{< youtube id="s43IBSVyTpQ" >}} From e165d5cbfcbda6b32979e8476a5b5b49a3f29ac5 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:36:26 +0100 Subject: [PATCH 33/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index 467087fc853e..c5c9928aacce 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -171,7 +171,7 @@ You can also adapt your plugins.conf, removing the need to change the command li ### Labels -Labels are used to [query logs](https://grafana.com/docs/loki//query/) `{container_name="nginx", cluster="us-west1"}`, they are usually metadata about the workload producing the log stream (`instance`, `container_name`, `region`, `cluster`, `level`). In Loki labels are indexed, so you should be cautious when choosing them. High cardinality label values can have drastic impact on performance. +Labels, for example `{container_name="nginx", cluster="us-west1"}`, are used to [query logs](https://grafana.com/docs/loki//query/). Labels are usually metadata about the workload producing the log stream (`instance`, `container_name`, `region`, `cluster`, `level`). In Loki labels are indexed, so you should be cautious when choosing them. High cardinality label values can have drastic impact on query performance. You can use `Labels`, `RemoveKeys` , `LabelKeys` and `LabelMapPath` to how the output plugin will perform labels extraction. From 4c4cec9f31a7e4d822db489b85d0cfe4b4f403cd Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:36:33 +0100 Subject: [PATCH 34/39] Update docs/sources/send-data/fluentbit/community-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/community-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/community-plugin.md b/docs/sources/send-data/fluentbit/community-plugin.md index c5c9928aacce..e89c6524ae1f 100644 --- a/docs/sources/send-data/fluentbit/community-plugin.md +++ b/docs/sources/send-data/fluentbit/community-plugin.md @@ -202,7 +202,7 @@ Considering the record below : } ``` -and a LabelMap file as follow : +and a LabelMap file as follows : ```json { From 0d3e3091b72df60ca7e396b3c841a4db05089256 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:36:46 +0100 Subject: [PATCH 35/39] Update docs/sources/send-data/fluentbit/fluent-bit-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/fluent-bit-plugin.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md index 64112fe39832..255b6d990c3b 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md @@ -118,9 +118,8 @@ Here is an example of how to deploy the Fluent Bit helm chart to collect logs fr 1. Add the Fluent Bit Helm repository: -```bash -helm repo add fluent https://fluent.github.io/helm-charts -``` + ```bash + helm repo add fluent https://fluent.github.io/helm-charts 1. Create a `values.yaml` file with the following content: ```yaml From 6f2231647dc6d8b4706a98b470a210a4b24d63fd Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:37:07 +0100 Subject: [PATCH 36/39] Update docs/sources/send-data/fluentbit/fluent-bit-plugin.md Co-authored-by: J Stickler --- .../send-data/fluentbit/fluent-bit-plugin.md | 27 +++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md index 255b6d990c3b..9221539932b6 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md @@ -122,20 +122,19 @@ Here is an example of how to deploy the Fluent Bit helm chart to collect logs fr helm repo add fluent https://fluent.github.io/helm-charts 1. Create a `values.yaml` file with the following content: -```yaml -config: - outputs: | - [OUTPUT] - Name loki - Match * - Host YourHost.Company.net - port 443 - tls on - tls.verify on - http_user XXX - http_passwd XXX - Labels agent=fluend-bit -``` + ```yaml + config: + outputs: | + [OUTPUT] + Name loki + Match * + Host YourHost.Company.net + port 443 + tls on + tls.verify on + http_user XXX + http_passwd XXX + Labels agent=fluend-bit Note we are only updating the `outputs` section of the Fluent Bit configuration. This is to replace the default output plugin with the Loki output plugin. If you need to update other parts of the Fluent Bit configuration refer to the [Fluent Bit values file reference](https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml). From bb796eb59bb34a9dd25bcbe180faa4104cd5b57f Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:37:21 +0100 Subject: [PATCH 37/39] Update docs/sources/send-data/fluentbit/fluent-bit-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/fluent-bit-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md index 9221539932b6..2cbdabb9194a 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md @@ -136,7 +136,7 @@ Here is an example of how to deploy the Fluent Bit helm chart to collect logs fr http_passwd XXX Labels agent=fluend-bit -Note we are only updating the `outputs` section of the Fluent Bit configuration. This is to replace the default output plugin with the Loki output plugin. If you need to update other parts of the Fluent Bit configuration refer to the [Fluent Bit values file reference](https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml). + Note we are only updating the `outputs` section of the Fluent Bit configuration. This is to replace the default output plugin with the Loki output plugin. If you need to update other parts of the Fluent Bit configuration refer to the [Fluent Bit values file reference](https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml). 1. Deploy the Fluent Bit Helm chart: From 7468b3c6564cf1e38e6318cc22d8069453a6bf25 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:37:37 +0100 Subject: [PATCH 38/39] Update docs/sources/send-data/fluentbit/fluent-bit-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/fluent-bit-plugin.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md index 2cbdabb9194a..714198627efa 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md @@ -140,9 +140,8 @@ Here is an example of how to deploy the Fluent Bit helm chart to collect logs fr 1. Deploy the Fluent Bit Helm chart: -```bash -helm install fluent-bit fluent/fluent-bit -f values.yaml -``` + ```bash + helm install fluent-bit fluent/fluent-bit -f values.yaml ## Where next? From 602ca54f06a8c943c54fa8f8c5c7e0801f56e305 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 9 Oct 2024 11:37:47 +0100 Subject: [PATCH 39/39] Update docs/sources/send-data/fluentbit/fluent-bit-plugin.md Co-authored-by: J Stickler --- docs/sources/send-data/fluentbit/fluent-bit-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md index 714198627efa..13f31994ab4e 100644 --- a/docs/sources/send-data/fluentbit/fluent-bit-plugin.md +++ b/docs/sources/send-data/fluentbit/fluent-bit-plugin.md @@ -143,6 +143,6 @@ Here is an example of how to deploy the Fluent Bit helm chart to collect logs fr ```bash helm install fluent-bit fluent/fluent-bit -f values.yaml -## Where next? +## Next steps - [Sending logs to Loki using Fluent Bit tutorial](https://grafana.com/docs/loki//send-data/fluentbit/fluent-bit-loki-tutorial/) \ No newline at end of file