Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enabled by default SSL for Logstash API #6971

Closed
thbkrkr opened this issue Jul 3, 2023 · 0 comments · Fixed by #7408
Closed

Enabled by default SSL for Logstash API #6971

thbkrkr opened this issue Jul 3, 2023 · 0 comments · Fixed by #7408
Labels
>enhancement Enhancement of existing functionality :logstash

Comments

@thbkrkr
Copy link
Contributor

thbkrkr commented Jul 3, 2023

We should enabled by default SSL for Logstash API and make it possible to configure it through spec.http like it's possible for other stack apps (see k8s-tls-certificates doc).

spec:
  http:
  
    service:
      spec:
        type: LoadBalancer
    
    tls:  
      selfSignedCertificate:
        disabled: true

      selfSignedCertificate:
        subjectAltNames:
          - ip: 160.46.176.15
          - dns: hulk.example.com
      
      certificate:
        secretName: my-cert

There is currently a TODO related to this:

cloud-on-k8s/pkg/controller/logstash/pod.go

Lines 104 to 110 in 3a2ce8d

// TODO integrate with api.ssl.enabled
// if params.Logstash.Spec.HTTP.TLS.Enabled() {
// httpVol := certificates.HTTPCertSecretVolume(logstashv1alpha1.Namer, params.Logstash.Name)
// builder.
// WithVolumes(httpVol.Volume()).
// WithVolumeMounts(httpVol.VolumeMount())
// }

@thbkrkr thbkrkr added >enhancement Enhancement of existing functionality :logstash labels Jul 3, 2023
@kaisecheng kaisecheng mentioned this issue Jan 9, 2024
Merged
pebrc added a commit that referenced this issue Jan 23, 2024
@kaisecheng @robbavey
@pebrc @barkbay
Logstash adds TLS support to API server (#7408)
849ce1e 
This PR adds TLS/ HTTPS and basic authentication integration to Logstash
[API
server](https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html#monitoring-api-security).
The minimum support version changes from `8.6.0` to `8.12.0`.

Sample logstash.yml
```
api.ssl.enabled: "true"
api.ssl.keystore.path: "/path/to/keystore.p12"
api.ssl.keystore.password: "${SSL_KEYSTORE_PASSWORD}"
api.auth.type: basic
api.auth.basic.username: "${API_USERNAME}"
api.auth.basic.password: "${API_PASSWORD}"
```

HTTPS is on by default meaning `api.ssl.enabled`,
`api.ssl.keystore.path` and `api.ssl.keystore.password` is set in config
`logstash.yml`. The API server (puma jruby) only supports HTTPS with p12
keystore and java keystore. Therefore,
[InitContainer](https://github.com/elastic/cloud-on-k8s/pull/7408/files#diff-000e81cb01c6f6b546ab205bc72599d2cc662ddcb8c5df9106eb7a2dd316c25aR38)
needs to covert CA and TLS certs to the format puma accepts. If
`api.ssl.enabled` set to true and the API Service is set to
[disable](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-tls-certificates.html#k8s-disable-tls)
TLS `tls.selfSignedCertificate.disabled`, reconcile config
[fails](https://github.com/elastic/cloud-on-k8s/pull/7408/files#diff-f2238a0d916b12187fca471853c77565a5d549079202cfe69199cd31b0139525R140).
If API Service is set to disable and `api.ssl.enabled` is unset, server
will disable TLS.

Logstash resolves `${VAR}` from
[ENV](https://www.elastic.co/guide/en/logstash/current/environment-variables.html)
and
[Keystore](https://www.elastic.co/guide/en/logstash/current/keystore.html).
When the same key is declared in both places, keystore takes the
precedence. As Logstash allows setting HTTP basic authentication with
`api.auth.type`, `api.auth.basic.username` and `api.auth.basic.password`
in `logstash.yml`, this PR has integrated ReadinessProbe and Stack
Monitoring by passing the resolved value of username password. The value
of the variable comes from the following
[sources](https://github.com/elastic/cloud-on-k8s/pull/7408/files#diff-f2238a0d916b12187fca471853c77565a5d549079202cfe69199cd31b0139525R202-R255)
in the order of priority: Env, Env from ConfigMap, Env from Secret,
Keystore from Secure Settings . The later sources take precedence.

Sample config
```yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: monitoring
spec:
  version: 8.12.0
  nodeSets:
    - name: default
      count: 1
      config:
        node.store.allow_mmap: false
---
apiVersion: v1
kind: Secret
metadata:
  name: logstash-secure-settings
stringData:
  API_USERNAME: batman
  API_PASSWORD: i_am_rich
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-sample
spec:
  count: 1
  version: 8.12.0
  config:
    api.auth.type: basic
    api.auth.basic.username: "${API_USERNAME}"
    api.auth.basic.password: "${API_PASSWORD"
  secureSettings:
    - secretName: logstash-secure-settings
  monitoring:
    metrics:
      elasticsearchRefs:
        - name: monitoring
    logs:
      elasticsearchRefs:
        - name: monitoring
  pipelines:
    - pipeline.id: main
      pipeline.workers: 2
      config.string: |
        input { exec { command => 'uptime' interval => 10 } } 
        output { 
          stdout {}
        }
---
```

The sample config creates following resources
```yaml
NAMESPACE  NAME                                                          READY  REASON  AGE
default    Logstash/logstash-sample                                      -              11m
default    ├─Secret/logstash-sample-default-monitoring-beat-ls-mon-user  -              11m
default    ├─Secret/logstash-sample-ls-config                            -              11m
default    ├─Secret/logstash-sample-ls-http-ca-internal                  -              11m
default    ├─Secret/logstash-sample-ls-http-certs-internal               -              11m
default    ├─Secret/logstash-sample-ls-monitoring-default-monitoring-ca  -              11m
default    ├─Secret/logstash-sample-ls-monitoring-filebeat-config        -              11m
default    ├─Secret/logstash-sample-ls-monitoring-metricbeat-config      -              11m
default    ├─Secret/logstash-sample-ls-pipeline                          -              11m
default    ├─Service/logstash-sample-ls-api                              -              11m
default    │ └─EndpointSlice/logstash-sample-ls-api-nh5w6                -              11m
default    └─StatefulSet/logstash-sample-ls                              -              11m
default      ├─ControllerRevision/logstash-sample-ls-5f77b6b9ff          -              11m
default      └─Pod/logstash-sample-ls-0                                  True           11m
```

In the past, Secret/logstash-sample-ls-config only stored the
`logstash.yml` content. Now it stores the resolved value of
api.ssl.keystore.password under the Secret key `API_KEYSTORE_PASS` for
not exposing the password in plain text in initConfigContainer


e2e test
- TestLogstashStackMonitoring
- TestLogstashResolvingDollarVariableInStackMonitoring

fix: #6971,
elastic/ingest-dev#1591

---------

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
Co-authored-by: Michael Morello <michael.morello@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>enhancement Enhancement of existing functionality :logstash
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add TLS and basic authentication integration to Logstash API server kaisecheng/cloud-on-k8s
1 participant
@thbkrkr