User-facing documentation for Logstash on ECK #6743
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Initial Commit of ECK for Logstash Initial commit of a simple operator. The first operator will create: * A Service exposing the logstash metrics API, so it can be used for monitoring purposes, with the option to expose further services if necessary * Secrets holding logstash.yml * A StatefulSet deploying the logstash pods * Pods with default readiness probe The sample logstash yml file as located in config/samples/logstash/logstash_svc.yaml will create: ``` ✗ kubectl tree logstash logstash-sample NAMESPACE NAME READY REASON AGE default Logstash/logstash-sample - 7m3s default ├─Secret/logstash-sample-ls-config - 7m3s default ├─Service/logstash-sample-ls-api - 7m3s default │ └─EndpointSlice/logstash-sample-ls-api-4dl2x - 7m3s default ├─Service/logstash-sample-ls-beats - 7m3s default │ └─EndpointSlice/logstash-sample-ls-beats-rw8sp - 7m3s default ├─Service/logstash-sample-ls-default - 7m3s default │ └─EndpointSlice/logstash-sample-ls-default-6vmhp - 7m3s default └─StatefulSet/logstash-sample-ls - 7m3s default ├─ControllerRevision/logstash-sample-ls-86448964 - 7m3s default ├─Pod/logstash-sample-ls-0 True 7m3s default ├─Pod/logstash-sample-ls-1 True 7m3s default └─Pod/logstash-sample-ls-2 True 7m3s``` And shows status: ``` ✗ kubectl get logstash logstash-sample NAME AVAILABLE EXPECTED AGE VERSION logstash-sample 2 2 2m35s 8.6.1 ``` Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co> Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> Co-authored-by: Michael Morello <michael.morello@gmail.com>
Following the prior art, this commit integrates stack monitoring for Logstash by adding metricsbeat and filebeat as sidecars to deliver metrics and log to monitoring es cluster There is a sample resource in config/samples/logstash/logstash_stackmonitor.yaml to deploy and test. Co-authored-by: Rob Bavey <rob.bavey@elastic.co> Co-authored-by: Michael Morello <michael.morello@gmail.com>
This PR allows configuration of Logstash pipeline.yml in CRD. pipeline.yml is an array of map of string/interface{} defining multiple pipelines. `.` in between the key is treated as string. Pipelines can be set from either inline in Pipelines or secret referencing in PipelinesRef.
e2e test: TestPipelineConfigRefLogstash, TestPipelineConfigLogstash
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Michael Morello <michael.morello@gmail.com>
This commit adds extra unit tests for the Logstash Controller, including tests for config, controller, pods and services
…pod restart (elastic#6674) This commit adds the ability to reload logstash pipelines when the pipeline changes, without triggering a full restart of the pod, leveraging Logstash's ability to watch pipeline definitions, and reload automatically if a change is discovered. A logstash config directory typically includes logstash.yml, pipelines.yml, jvm.options and log4j2.properties required to run logstash - while logstash can store the contents of a pipeline definition in any location, the pipelines.yml definition file, which states where these definition files are must be in the same config directory as the other setup files. To enables us to have a mixture of copied and generated files in this config directory, an initContainer is used, with a small script to prepare the config directory. The script copies the contents of the /usr/share/logstash/config into the a shared config volume Additionally softlinks are created into this volume from the pipelines.yml and logstash.yml secrets created by the logstash operator. Additionally, we now do not include changes to pipelines in the configuration hash that triggers a pod reload, but instead write the config.reload.automatic: true setting in to logstash.yml Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>
This commit adds information about Logstash to the set of ECK telemetry Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>
Fix Logstash service e2e tests This test was failing, because the readiness probe was hitting the port defined by the service rather than the default port. This commit updates the e2e test to set the config to serve the logstash api from the same port that the service specifies. Co-authored-by: Michael Morello <michael.morello@gmail.com>
This commit allows configuration of `elasticsearchRefs` in CRD to associate Elasticsearch clusters to Logstash. The connectivity info/ credentials are passed via environment variable to Logstash container for referencing in Logstash plugins: logstash-output-elasticsearch, logstash-input-elasticsearch and logstash-filter-elasticsearch. Environment variables taking `$clusterName_ES_` as a prefix are available. Co-authored-by: Michael Morello <michael.morello@gmail.com> Co-authored-by: Rob Bavey <rob.bavey@elastic.co> Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
…ic#6731) * Fix the sample tests for logstash service and stack monitoring The stack monitoring tests were failing as the builder was not creating the monitoring elasticsearch references. This was added, as well as a minor refactoring to split the builder into adding metrics and logging monitoring separately. This commit also calculates the the monitoring assocation counts correctly, as previously the metrics association was double counted. This commit also fixes the logstash service sample end to end test - the pipeline id in the yaml definition did not match that in the test. Co-authored-by: Michael Morello <michael.morello@gmail.com>
Co-authored-by: Peter Brachwitz <peter.brachwitz@elastic.co>
Still a work in progress - will need elasticsearchRef to work fully
…nts are mentioned
{logstash} should be {ls}
|
@pebrc @thbkrkr @barkbay @alaudazzi @karenzone @mashhurs @kaisecheng This is ready for another round of reviews. Thanks! |
kaisecheng
reviewed
May 3, 2023
docs/orchestrating-elastic-stack-applications/logstash.asciidoc
Outdated
Show resolved
Hide resolved
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
kaisecheng
reviewed
May 3, 2023
| count: 3 | ||
| config: | ||
| # This setting has performance implications. See the README for more details. | ||
| node.store.allow_mmap: false |
There was a problem hiding this comment.
optional: it will be great to also see elasticsearch and kibana metrics in stack monitoring. This also makes sure Logstash metrics show in UI
kaisecheng
reviewed
May 4, 2023
docs/orchestrating-elastic-stack-applications/logstash.asciidoc
Outdated
Show resolved
Hide resolved
kaisecheng
reviewed
May 4, 2023
docs/orchestrating-elastic-stack-applications/logstash.asciidoc
Outdated
Show resolved
Hide resolved
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
… into robbavey-logstash_doc
karenzone
approved these changes
May 8, 2023
|
@alaudazzi @pebrc @thbkrkr @barkbay - this docs PR has been approved on the Logstash side by our tech writer @karenzone and is ready for final review from the cloud team. Thanks! |
Yes, release related label added (it will help me to [check/take care about] backports). |
💚 All backports created successfully
Questions ?Please refer to the Backport tool documentation |
pebrc
pushed a commit
to pebrc/cloud-on-k8s
that referenced
this pull request
May 10, 2023
Documentation for the technical preview of Logstash support in ECK (cherry picked from commit 298157f)
pebrc
added a commit
that referenced
this pull request
May 10, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes documentation for Logstash on ECK.
Includes:
DOC PREVIEW: https://cloud-on-k8s_6743.docs-preview.app.elstc.co/guide/en/cloud-on-k8s/master/k8s-logstash.html