Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Logstash secure settings from Kubernetes Secrets using keystore #7024

Merged
merged 15 commits into from
Aug 4, 2023

Conversation

kaisecheng
Copy link
Contributor

@kaisecheng kaisecheng commented Jul 14, 2023

This commit adds support for keystore to Logstash operator.

The key values in keystore are available to Logstash pipelines as environment variables, which can resolve by ${KEY} notation. The keystore can be password protected by setting an environment variable called LOGSTASH_KEYSTORE_PASS. The password is expected to be declared in the main container in env.

Sample config with customized keystore password
apiVersion: v1
kind: Secret
metadata:
  name: logstash-keystore-pass
stringData:
  LOGSTASH_KEYSTORE_PASS: changed
---
apiVersion: v1
kind: Secret
metadata:
  name: logstash-secure-settings
stringData:
  HELLO: Hallo
  WORLD: Welt
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-sample
spec:
  version: 8.8.0
  count: 1
  pipelines:
    - pipeline.id: main
      config.string: |-
        input { exec { command => 'uptime' interval => 10 } }
        filter {
          if ("${HELLO:}" != "") {
            mutate { add_tag => ["awesome"] }
          }
        }
  secureSettings:
    - secretName: logstash-secure-settings
  podTemplate:
    spec:
      containers:
        - name: logstash
          env:
            - name: LOGSTASH_KEYSTORE_PASS
              valueFrom:
                secretKeyRef:
                  name: logstash-keystore-pass
                  key: LOGSTASH_KEYSTORE_PASS

A known issue is that the keystore command logstash-keystore is very slow in proportion to the number of key values to add. In my local machine, adding 10 keys needs 6 minutes to start Logstsah.

Adding or updating key values in keystore triggers pod rotation, while deleting a key does not.

This commit adds e2e tests TestLogstashKeystoreWithoutPassword and TestLogstashKeystoreWithPassword

A follow-up issue for improving the keystore command performance

@botelastic botelastic bot added the triage label Jul 14, 2023
@kaisecheng kaisecheng requested a review from robbavey July 14, 2023 17:32
@thbkrkr thbkrkr added the >enhancement Enhancement of existing functionality label Jul 17, 2023
@botelastic botelastic bot removed the triage label Jul 17, 2023
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question about whether we need to set a default password for the keystore.

},
}

DefaultKeystorePass = corev1.EnvVar{Name: KeystorePassKey, Value: "changeit"}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to set a default value? The keystore should would work without a password (even though we don't recommend it), and I'm not sure what this buys us.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the password in env, logstash-keystore prompts user to input [y/N] to continue.
So, we need to either give [N] to stdin or set the default password

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to use yes to send the input to stdin, so you could change the keystoreCommand to

keystoreCommand = "yes | /usr/share/logstash/bin/logstash-keystore"

to pipe y in

or

keystoreCommand = "yes n | /usr/share/logstash/bin/logstash-keystore"

to pipe n in.

I'm not sure it matters a huge amount, but it might be easier to document if we don't set a default that doesn't really do anything

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh! I am such a fool :(
will change the command

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not at all!

And thank you!

@kaisecheng kaisecheng requested a review from robbavey July 19, 2023 14:53
[id="{p}-logstash-keystore"]
=== Setting keystore

You can specify sensitive settings with Kubernetes secrets. ECK automatically injects these settings into the keystore on each Logstash before it starts Logstash. The ingestion process could extend the startup time of Logstash pod significantly.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add this as a technical preview NOTE, rather than in the main text, and add that the startup time increases linearly with the number of variables added.
Something like:

NOTE: For the technical preview, the use of settings in the Logstash keystore may impact startup time for Logstash Pods. Startup time will increase linearly for each entry added to the keystore, and this could extend startup time significantly.

},
}

DefaultKeystorePass = corev1.EnvVar{Name: KeystorePassKey, Value: "changeit"}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to use yes to send the input to stdin, so you could change the keystoreCommand to

keystoreCommand = "yes | /usr/share/logstash/bin/logstash-keystore"

to pipe y in

or

keystoreCommand = "yes n | /usr/share/logstash/bin/logstash-keystore"

to pipe n in.

I'm not sure it matters a huge amount, but it might be easier to document if we don't set a default that doesn't really do anything

Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM - couple of suggestions on doc.

pkg/controller/logstash/keystore.go Outdated Show resolved Hide resolved
kaisecheng and others added 4 commits July 19, 2023 22:38
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
@kaisecheng
Copy link
Contributor Author

@pebrc @thbkrkr @barkbay This is ready for review

Copy link
Collaborator

@pebrc pebrc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM I guess another alternative to allowing batch adding to the keystore would be to optimise the startup time of the keystore commend with JVM techniques like CDS or Graal's native image.

kaisecheng and others added 3 commits August 3, 2023 20:01
Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
…keystore

# Conflicts:
#	docs/orchestrating-elastic-stack-applications/logstash.asciidoc
@kaisecheng
Copy link
Contributor Author

@pebrc Could you merge the PR? I don't have the permission to do so.

@pebrc pebrc merged commit cce77a3 into elastic:main Aug 4, 2023
@rhr323 rhr323 changed the title Add Logstash keystore Keystore support for Logstash Oct 25, 2023
@thbkrkr thbkrkr changed the title Keystore support for Logstash Secure settings support for Logstash from Kubernetes Secrets using keystore Oct 26, 2023
@thbkrkr thbkrkr changed the title Secure settings support for Logstash from Kubernetes Secrets using keystore Support for Logstash secure settings from Kubernetes Secrets using keystore Oct 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement Enhancement of existing functionality v2.10.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants