Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document Logstash connection to external Elasticsearch #6895

Merged
merged 7 commits into from
Jun 22, 2023
Merged

Document Logstash connection to external Elasticsearch #6895

Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
802df0b
Logstash add port to Elasticsearch URL
kaisecheng Jun 8, 2023
181d31f
add doc
kaisecheng Jun 11, 2023
47bd790
lint
kaisecheng Jun 12, 2023
4e9d013
Update pkg/controller/logstash/env_test.go
kaisecheng Jun 20, 2023
5c18ce6
add note to port
kaisecheng Jun 20, 2023
ed8064a
Revert "Logstash add port to Elasticsearch URL"
kaisecheng Jun 20, 2023
30de0e2
Update docs/orchestrating-elastic-stack-applications/logstash.asciidoc
kaisecheng Jun 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 46 additions & 2 deletions docs/orchestrating-elastic-stack-applications/logstash.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,10 +288,10 @@ NOTE: Logstash persistent queues (PQs) and dead letter queues (DLQs) are not cur
[id="{p}-logstash-pipelines-es"]
=== Using Elasticsearch in Logstash pipelines

The `spec.elasticsearchRefs` section provides a mechanism to help configure Logstash to estabish a secured connection to one or more managed Elasticsearch clusters. By default, each `elasticsearchRef` will target all nodes in its referenced Elasticsearch cluster. If you want to direct traffic to specific nodes of your Elasticsearch cluster, refer to <<{p}-traffic-splitting>> for more information and examples.
The `spec.elasticsearchRefs` section provides a mechanism to help configure Logstash to establish a secured connection to one or more ECK managed Elasticsearch clusters. By default, each `elasticsearchRef` will target all nodes in its referenced Elasticsearch cluster. If you want to direct traffic to specific nodes of your Elasticsearch cluster, refer to <<{p}-traffic-splitting>> for more information and examples.

When you use `elasticsearchRefs` in a Logstash pipeline, the Logstash operator creates the necessary resources from the associated Elasticsearch cluster, and provides environment variables to allow these resources to be accessed from the pipeline configuration.
Environment variables are replaced at runtime with the appropriate values.```
Environment variables are replaced at runtime with the appropriate values.
The environment variables have a fixed naming convention:

* `NORMALIZED_CLUSTERNAME_ES_HOSTS`
Expand Down Expand Up @@ -367,6 +367,50 @@ spec:
<4> Elasticsearch output definitions - use the environment variables created by the Logstash operator when specifying an `ElasticsearchRef`. Note the use of "normalized" versions of the `clusterName` in the environment variables used to populate the relevant fields.


[id="{p}-logstash-external-es"]
==== Connect to an external Elasticsearch cluster

Logstash can connect to external Elasticsearch cluster that is not managed by ECK.
You can reference a Secret instead of an Elasticsearch cluster in the `elasticsearchRefs` section through the `secretName` attribute:

[source,yaml,subs="attributes,callouts"]
----
apiVersion: v1
kind: Secret
metadata:
name: external-es-ref
stringData:
url: https://abcd-42.xyz.elastic-cloud.com:443 <1>
username: logstash_user <2>
password: REDACTED <3>
ca.crt: REDACTED <4>
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
name: quickstart
spec:
version: {version}
count: 1
elasticsearchRefs:
- clusterName: prod-es
secretName: external-es-ref <5>
monitoring:
metrics:
elasticsearchRefs:
- secretName: external-es-ref <5>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming a user specifies a URL without explicit port in the secret, then the changes in this PR append 443 by default for the environment. However Beats also assume 9200 by default and while the Logstash side is fixed, the monitoring will still not work until the user fixes the url field in the secret to include a port.

logs:
elasticsearchRefs:
- secretName: external-es-ref <5>
----

<1> The URL to reach the Elasticsearch cluster.
<2> The username of the user to be authenticated to the Elasticsearch cluster.
<3> The password of the user to be authenticated to the Elasticsearch cluster.
<4> The CA certificate in PEM format to secure communication to the Elasticsearch cluster (optional).
<5> The `secretName` and `name` attributes are mutually exclusive, you have to choose one or the other.

NOTE: Please always specify the port in URL when connect to external Elasticsearch Cluster.
kaisecheng marked this conversation as resolved.
Show resolved Hide resolved

[id="{p}-logstash-expose-services"]
=== Expose services
Expand Down