Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add TLS and basic authentication integration to Logstash API server #7408

Merged
merged 46 commits into from
Jan 23, 2024

Conversation

kaisecheng
Copy link
Contributor

@kaisecheng kaisecheng commented Dec 20, 2023

This PR adds TLS/ HTTPS and basic authentication integration to Logstash API server. The minimum support version changes from 8.6.0 to 8.12.0.

Sample logstash.yml

api.ssl.enabled: "true"
api.ssl.keystore.path: "/path/to/keystore.p12"
api.ssl.keystore.password: "${SSL_KEYSTORE_PASSWORD}"
api.auth.type: basic
api.auth.basic.username: "${API_USERNAME}"
api.auth.basic.password: "${API_PASSWORD}"

HTTPS is on by default meaning api.ssl.enabled, api.ssl.keystore.path and api.ssl.keystore.password is set in config logstash.yml. The API server (puma jruby) only supports HTTPS with p12 keystore and java keystore. Therefore, InitContainer needs to covert CA and TLS certs to the format puma accepts. If api.ssl.enabled set to true and the API Service is set to disable TLS tls.selfSignedCertificate.disabled, reconcile config fails. If API Service is set to disable and api.ssl.enabled is unset, server will disable TLS.

Logstash resolves ${VAR} from ENV and Keystore. When the same key is declared in both places, keystore takes the precedence. As Logstash allows setting HTTP basic authentication with api.auth.type, api.auth.basic.username and api.auth.basic.password in logstash.yml, this PR has integrated ReadinessProbe and Stack Monitoring by passing the resolved value of username password. The value of the variable comes from the following sources in the order of priority: Env, Env from ConfigMap, Env from Secret, Keystore from Secure Settings . The later sources take precedence.

Sample config

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: monitoring
spec:
  version: 8.12.0
  nodeSets:
    - name: default
      count: 1
      config:
        node.store.allow_mmap: false
---
apiVersion: v1
kind: Secret
metadata:
  name: logstash-secure-settings
stringData:
  API_USERNAME: batman
  API_PASSWORD: i_am_rich
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-sample
spec:
  count: 1
  version: 8.12.0
  config:
    api.auth.type: basic
    api.auth.basic.username: "${API_USERNAME}"
    api.auth.basic.password: "${API_PASSWORD"
  secureSettings:
    - secretName: logstash-secure-settings
  monitoring:
    metrics:
      elasticsearchRefs:
        - name: monitoring
    logs:
      elasticsearchRefs:
        - name: monitoring
  pipelines:
    - pipeline.id: main
      pipeline.workers: 2
      config.string: |
        input { exec { command => 'uptime' interval => 10 } } 
        output { 
          stdout {}
        }
---

The sample config creates following resources

NAMESPACE  NAME                                                          READY  REASON  AGE
default    Logstash/logstash-sample                                      -              11m
default    ├─Secret/logstash-sample-default-monitoring-beat-ls-mon-user  -              11m
default    ├─Secret/logstash-sample-ls-config                            -              11m
default    ├─Secret/logstash-sample-ls-http-ca-internal                  -              11m
default    ├─Secret/logstash-sample-ls-http-certs-internal               -              11m
default    ├─Secret/logstash-sample-ls-monitoring-default-monitoring-ca  -              11m
default    ├─Secret/logstash-sample-ls-monitoring-filebeat-config        -              11m
default    ├─Secret/logstash-sample-ls-monitoring-metricbeat-config      -              11m
default    ├─Secret/logstash-sample-ls-pipeline                          -              11m
default    ├─Service/logstash-sample-ls-api                              -              11m
default    │ └─EndpointSlice/logstash-sample-ls-api-nh5w6                -              11m
default    └─StatefulSet/logstash-sample-ls                              -              11m
default      ├─ControllerRevision/logstash-sample-ls-5f77b6b9ff          -              11m
default      └─Pod/logstash-sample-ls-0                                  True           11m

In the past, Secret/logstash-sample-ls-config only stored the logstash.yml content. Now it stores the resolved value of api.ssl.keystore.password under the Secret key API_KEYSTORE_PASS for not exposing the password in plain text in initConfigContainer

e2e test

  • TestLogstashStackMonitoring
  • TestLogstashResolvingDollarVariableInStackMonitoring

fix: #6971, https://github.com/elastic/ingest-dev/issues/1591

@botelastic botelastic bot added the triage label Dec 20, 2023
@kaisecheng kaisecheng force-pushed the logstash_support_https_api branch from ecd1518 to 4395ddf Compare December 20, 2023 22:37
@thbkrkr thbkrkr added the >enhancement Enhancement of existing functionality label Dec 21, 2023
@botelastic botelastic bot removed the triage label Dec 21, 2023
…support_https_api

# Conflicts:
#	config/samples/logstash/logstash_stackmonitor.yaml
@kaisecheng kaisecheng requested a review from robbavey December 21, 2023 12:31
@kaisecheng kaisecheng marked this pull request as ready for review January 2, 2024 13:05
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a comment on testing.

Also - what is the effort to make the api.* settings support setting with secrets and ENV variables, rather than putting the password in clear in the Config object?

Running through the example and seeing:

Spec:
  Config:
    api.auth.basic.password:  i_am_rich
    api.auth.basic.username:  batman
    api.auth.type:            basic

when running k describe logstashes.logstash.k8s.elastic.co logstash-sample is something we should fix before we GA this feature

Name: LogstashAPIServiceName,
TLS: commonv1.TLSOptions{
SelfSignedCertificate: &commonv1.SelfSignedCertificate{
Disabled: false,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test passes regardless of whether the value here has the TLS block, or whether selfSignedCertificate.Disabled value is true or false, as only the useTLS flag is relevant here.

I think we are missing a layer of testing, of how the useTLS setting is derived from the TLSOptions

@kaisecheng
Copy link
Contributor Author

@robbavey Thanks for testing this PR.
I am making api.* related configs to support ${VAR} from ENV and keystore that defined in Logstash Spec. The key value pairs of keystore are only taken from Spec.SecureSettings. When the same key defined in both places, keystore takes precedence. As Logstash supports ${VAR:default_value}, operator also needs to support in the same way

Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on what is up so far.

…ure settings) and env

- integrate the resolved value to stack monitoring, readinessProbe
- store the API keystore password in config secret for initConfigContainer
@kaisecheng kaisecheng requested a review from robbavey January 5, 2024 22:13
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still working through it, but adding my thoughts as I go

@@ -699,6 +700,97 @@ spec:
The name of the container in the Pod template must be `logstash`.


[id="{p}-logstash-http-config"]
== HTTP Configuration
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to qualify that this is related to the Monitoring API/API service only, and tie it back to the use of stack monitoring, maybe something like "Securing Monitoring Endpoint" with a discussion that we use HTTPS by default, but this can be changed following the instructions here


logstashv1alpha1 "github.com/elastic/cloud-on-k8s/v2/pkg/apis/logstash/v1alpha1"
"github.com/elastic/cloud-on-k8s/v2/pkg/apis/logstash/v1alpha1"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason we lost the logstash disambiguation here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert it.

elasticsearchRef:
name: "elasticsearch-sample"
services:
- name: api
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should add a note that the service name has to be api

// and Keystore from SecureSettings. If the same key defined in both places, keystore takes the precedence.
func getKeystoreEnvKeyValue(params Params) (map[string]string, error) {
data := make(map[string]string)
c := getLogstashContainer(params.Logstash.Spec.PodTemplate.Spec.Containers)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return value from getLogstashContainer needs to be checked, and we should add a warning if no container is found -
this can be nil if no explicit podTemplate.spec.containers block is set, which will cause the operator to crash hard when subsequently attempting to read env values from it

// VAR is the variable name that expect to be defined in Keystore or Env
//
// The variable name can consist of digit, underscores and letters
// The default value is optional, ${VAR:} and ${VAR} are valid.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we need to support the default case, and if we can just stick with the standard ${VAR}? If we do support the default case, we should log that we are using the default

As far as supporting default values, I think that allowing defaults will only work when there are already ENV variables set in the podTemplate container definition as it stands, so the use case for supporting defaults is potentially narrow, and potentially a source of confusion.

Also, I'm not sure if allowing defaults is typical in Kubernetes - I don't believe that envFrom supports it's use

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Supporting default value is mainly for keeping consistent of the feature that Logstash provide. The default value is used when the ${VAR} is missing in env and keystore. It allows Logstash starts without raising error.

apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-sample
spec:
  count: 1
  version: 8.11.1
  config:
    api.auth.type: basic
    api.auth.basic.username: "${API_USERNAME:batman}"
    api.auth.basic.password: "${API_PASSWORD:i_am_rich}"

Supporting default is not a thing for Kubernetes, but for Stack Monitoring and ReadinessProbe query monitoring endpoint.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that makes sense, but I'd still like to see a log if we do revert to default values (not logging the default values...)

But I'm still on the fence whether we need the complexity of adding support for default values for ENV variables for this feature only, as I suspect adding default values for usernames and passwords is not going to be a massively used feature, and may be something we can document our way around.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree having default value for username password is a small use case and not a good practice. This makes me think why Logstash supports this feature in logstash.yml at the first place. For pipelines config, this is a handy features for business logic. However, for logstash config, all settings are important, so, if I expect a ${VAR} and it is missing, I only want to see error and stop Logstash.

Probably, the default value is just a fallback setup to disable basic authentication and SSL in testing phase.

api.ssl.enabled: "${SSL_ENABLED:false}"
api.auth.basic.username: "${API_USERNAME:}"
api.auth.basic.password: "${API_PASSWORD:}"

I think it is acceptable to remove the feature. Do you see any downside to keep it? I am open to remove the feature.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, for business logic in pipelines, it's very useful, but it feels a little less so in this case, and silently falling back to defaults for passwords is something I worry about.

How much effort is it to remove? And how much would it simplify?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove it then. I will need to update doc, test cases and the config.go

// and Keystore from SecureSettings. If the same key defined in both places, keystore takes the precedence.
func getKeystoreEnvKeyValue(params Params) (map[string]string, error) {
data := make(map[string]string)
c := getLogstashContainer(params.Logstash.Spec.PodTemplate.Spec.Containers)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The return value from getLogstashContainer should be checked - the value will be nil with a CRD definition not containing an explicit container definition in the podTemplate spec, which will cause a crash in subsequent checks of the Env and EnvFrom, etc below.

<1> Store the username and password in a Secret.
<2> Map the username and password to the environment variables of the Pod.
<3> At Logstash startup, `${API_USERNAME}` and `${API_PASSWORD}` are replaced by the value of environment variables. Check links:https://www.elastic.co/guide/en/logstash/current/environment-variables.html[using environment variables] for more details.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a reference somewhere to using the logstash keystore?

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
@kaisecheng
Copy link
Contributor Author

Is there any version restriction?

@barkbay Are you getting the docker image released by Elastic? openssl command is there since 7.17.0 and I can see it is in your testing version 8.11.1

Running openssl version in the container returns OpenSSL 1.1.1f 31 Mar 2020

docker run --rm -it --entrypoint openssl docker.elastic.co/logstash/logstash:8.11.1 version

Copy link
Contributor

@barkbay barkbay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@barkbay Are you getting the docker image released by Elastic? openssl command is there since 7.17.0 and I can see it is in your testing version 8.11.1

Not in the UBI images until version 8.12.0 IIUC:

❌ docker.elastic.co/logstash/logstash-ubi8:8.11.1

docker run --rm --entrypoint openssl  docker.elastic.co/logstash/logstash-ubi8:8.11.1 version
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "openssl": executable file not found in $PATH: unknown.

✅ docker.elastic.co/logstash/logstash:8.11.1

docker run --rm --entrypoint openssl  docker.elastic.co/logstash/logstash:8.11.1 version 
OpenSSL 1.1.1f  31 Mar 2020

✅ docker.elastic.co/logstash/logstash-ubi:8.12.0

docker run --rm --entrypoint openssl  docker.elastic.co/logstash/logstash-ubi:8.12.0 version 
OpenSSL 1.1.1k  FIPS 25 Mar 2021

Which means that this feature is likely to break Logstash deployments for users on OpenShift until 8.12.0.

@kaisecheng
Copy link
Contributor Author

kaisecheng commented Jan 18, 2024

@barkbay You are right and thanks for testing it! Sadly, this is the case. I would update the minimum version to 8.12.0.

Copy link
Collaborator

@pebrc pebrc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, we probably need a big note in the release notes that we are restricting the version to 8.12 especially for early adopters who are already running Logstash in earlier versions.

if b.MutatedFrom != nil {
builderCopy.MutatedFrom = b.MutatedFrom.DeepCopy()
}
builderCopy.GlobalCA = b.GlobalCA
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are not copying the APIServer struct?

test/e2e/logstash/stack_monitoring_test.go Outdated Show resolved Hide resolved
pkg/controller/logstash/stackmon/sidecar_test.go Outdated Show resolved Hide resolved
Comment on lines 65 to 67
if version.MustParse(test.Ctx().ElasticStackVersion).LT(logstashv1alpha1.MinStackMonVersion) {
t.SkipNow()
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is enforce by the log stash builder already

Copy link
Contributor

@barkbay barkbay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I noticed this error message while running TestLogstashResolvingDollarVariableInStackMonitoring:

k logs -n e2e-mercury pod/test-ls-mon-dollar-5hq9-ls-0 elastic-internal-init-keystore
+ keystore_initialized_flag=/usr/share/logstash/config/elastic-internal-init-keystore.ok
+ [[ -f /usr/share/logstash/config/elastic-internal-init-keystore.ok ]]
+ echo 'Initializing keystore.'
Initializing keystore.
+ echo y
+ /usr/share/logstash/bin/logstash-keystore create
Using bundled JDK: /usr/share/logstash/jdk
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, exception=LogStash::ConfigurationError, message=>Cannot evaluate `${API_PASSWORD}`. Replacement variable `API_PASSWORD` is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties

WARNING: The keystore password is not set. Please set the environment variable `LOGSTASH_KEYSTORE_PASS`. Failure to do so will result in reduced security. Continue without password protection on the keystore? [y/N] [2024-01-22T08:27:35,934][INFO ][org.logstash.secret.store.backend.JavaKeyStore] Created Logstash keystore at /usr/share/logstash/config/logstash.keystore
Created Logstash keystore at /usr/share/logstash/config/logstash.keystore
...

The e2e test is eventually ✅ , I was wondering if it's relevant.

kaisecheng and others added 5 commits January 22, 2024 11:50
Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
- fix tests for minimum version
@kaisecheng
Copy link
Contributor Author

@barkbay The error message is potentially an item to be improved in Logstash side. The keystore command notices an unresolved ${VAR} which should have printed warning instead of error. The keystore is created successfully with correct value, so the test case should be fine.

Copy link
Contributor

@barkbay barkbay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

pkg/controller/logstash/config.go Outdated Show resolved Hide resolved
Comment on lines 176 to 185

// getUnresolvedVars matches pattern ${VAR} against the configuration value in logstash.yml, such as api.auth.basic.username: ${API_USERNAME}
// and gives a map of string and string pointer, for example: {"API_USERNAME" : &config.Username}
// The keys in the map represent variable names that require further resolution, retrieving the values from either the Keystore or Environment variables.
// The variable name can consist of digit, underscores and letters.
func getUnresolvedVars(config *configs.APIServer) map[string]*string {
data := make(map[string]*string)

pattern := `^\${([a-zA-Z0-9_]+)}$`
regex := regexp.MustCompile(pattern)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// getUnresolvedVars matches pattern ${VAR} against the configuration value in logstash.yml, such as api.auth.basic.username: ${API_USERNAME}
// and gives a map of string and string pointer, for example: {"API_USERNAME" : &config.Username}
// The keys in the map represent variable names that require further resolution, retrieving the values from either the Keystore or Environment variables.
// The variable name can consist of digit, underscores and letters.
func getUnresolvedVars(config *configs.APIServer) map[string]*string {
data := make(map[string]*string)
pattern := `^\${([a-zA-Z0-9_]+)}$`
regex := regexp.MustCompile(pattern)
const (
varPattern = `^\${([a-zA-Z0-9_]+)}$`
)
var (
varRegex = regexp.MustCompile(varPattern)
)
// getUnresolvedVars matches pattern ${VAR} against the configuration value in logstash.yml, such as api.auth.basic.username: ${API_USERNAME}
// and gives a map of string and string pointer, for example: {"API_USERNAME" : &config.Username}
// The keys in the map represent variable names that require further resolution, retrieving the values from either the Keystore or Environment variables.
// The variable name can consist of digit, underscores and letters.
func getUnresolvedVars(config *configs.APIServer) map[string]*string {
data := make(map[string]*string)


for _, envFrom := range c.EnvFrom {
// from ConfigMap
if envFrom.ConfigMapRef != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC the referenced ConfigMap or Secret are not watched? I was wondering if we should do something if they are updated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes they are not watched. I think it is okay to let user manages the pod restart to get latest value

}

func (server APIServer) UseTLS() bool {
switch server.SSLEnabled {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we could also use https://pkg.go.dev/strconv#ParseBool

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty string need to convert to true, but ParseBool() gives false

kaisecheng and others added 3 commits January 23, 2024 13:56
Co-authored-by: Michael Morello <michael.morello@gmail.com>
…support_https_api

# Conflicts:
#	test/e2e/test/logstash/http_client.go
@kaisecheng
Copy link
Contributor Author

@pebrc This is ready to merge. Thank you!

@pebrc pebrc merged commit 849ce1e into elastic:main Jan 23, 2024
6 checks passed
@thbkrkr thbkrkr changed the title Logstash adds TLS support to API server Add TLS and basic authentication integration to Logstash API server Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement Enhancement of existing functionality :logstash v2.12.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enabled by default SSL for Logstash API
5 participants