Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Logstash keystore performance #7642

Merged
merged 8 commits into from
Mar 22, 2024

Conversation

kaisecheng
Copy link
Contributor

@kaisecheng kaisecheng commented Mar 19, 2024

Fixes: #7027

Logstash has performance issue when init container creates keystore with a couple of keys.
Since Logstash 8.12.0, logstash-keystore command supports adding multiple keys in one operation.
This PR uses the command to add keys in a batch. The minimum version of Logstash currently running in ECK is 8.12.0

The steps to create keystore in Logstash

# create keystore without password
echo 'y' | bin/logstash-keystore create 
# add keys one and two
bin/logstash-keystore add one two
Enter value for one: # input ASCII and non-empty string, and then press <enter>
Added 'one' to the Logstash keystore.
Enter value for two:
Added 'two' to the Logstash keystore.

The following resources should start approximately in a minute

apiVersion: v1
kind: Secret
metadata:
  name: logstash-secure-settings
stringData:
  HELLO: HALLO
  A: "1"
  B: "2"
  C: "3"
  D: "4"
  E: "5"
  F: "6"
  G: "7"
  H: "8"
  I: "9"
  J: "10"
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-sample
spec:
  version: 8.12.0
  count: 1
  pipelines:
    - pipeline.id: main
      config.string: |-
        input { exec { command => 'uptime' interval => 10 } }
        filter {
          if ("${HELLO:}" != "") {
            mutate { add_tag => ["${A}", "${B}", "${C}", "${D}", "${E}", "${F}", "${G}", "${H}", "${I}", "${J}"] }
          }
        }
  secureSettings:
    - secretName: logstash-secure-settings

@botelastic botelastic bot added the triage label Mar 19, 2024
@barkbay barkbay added the >enhancement Enhancement of existing functionality label Mar 19, 2024
@botelastic botelastic bot removed the triage label Mar 19, 2024
@botelastic botelastic bot removed the triage label Mar 19, 2024
@kaisecheng kaisecheng requested a review from robbavey March 19, 2024 12:29
@kaisecheng kaisecheng marked this pull request as ready for review March 19, 2024 12:35
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works as advertised, just a couple of naming nits

@@ -27,6 +27,8 @@ type InitContainerParameters struct {
KeystoreAddCommand string
// Keystore create command
KeystoreCreateCommand string
// ContainerCommand is the bash script to run in container
ContainerCommand string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ContainerCommand string
CustomScript string

This may be easier to follow, if we call this CustomScript, and comment to state that this is a script to run that overrides the default Keystore script?

initContainersParameters = keystore.InitContainerParameters{
KeystoreCreateCommand: keystoreCommand + " create",
KeystoreAddCommand: keystoreCommand + ` add "$key" < "$filename"`,
KeystoreCreateCommand: "echo 'y' | /usr/share/logstash/bin/logstash-keystore create",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency, we might want to think about using both KeystoreCreateCommand and KeystoreAddCommand here, and in the script, or just calling out to logstash-keystore directly in the script.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added the command for consistency

@kaisecheng kaisecheng requested a review from robbavey March 19, 2024 19:29
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM other than small nit about using the same executable path consistently

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
@kaisecheng
Copy link
Contributor Author

@pebrc @barkbay this is ready for ECK team to review

@kaisecheng kaisecheng requested review from pebrc and barkbay March 19, 2024 22:03
Copy link
Collaborator

@pebrc pebrc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kaisecheng
Copy link
Contributor Author

@pebrc thanks for the review. This PR is ready to merge

@pebrc
Copy link
Collaborator

pebrc commented Mar 22, 2024

@robbavey can you approve to override your previous "change requested"?

Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@robbavey
Copy link
Member

Approved

@pebrc pebrc merged commit 45f1132 into elastic:main Mar 22, 2024
5 checks passed
pebrc pushed a commit that referenced this pull request Mar 25, 2024
The commit fixes TestLogstashKeystoreWithoutPassword .
The Logstash pipeline gave false positive in #7642.

    Keystore variable cannot do string comparison in conditional statement
    test criteria (pipelines.main.plugins.filters.0.events.out == 1) gives flaky result as the filters position are not guaranteed

Relates: #7642
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement Enhancement of existing functionality :logstash v2.13.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve Logstash keystore command performance
4 participants