Introduce Elasticsearch config and additional secret mounts to stack config policy #7233

kvalliyurnatt · 2023-10-13T21:49:15Z

This PR introduces two new fields to the stack config policy, under Elasticsearch

Config: untyped yaml settings that will go into elasticsearch.yml
SecretMounts: Additional secrets that need to be mounted into the Elasticsearch Pod.

These changes are part of an effort to allow users to configure Authentication policies through the Stack configuration.

This PR does the following.

StackConfigPolicy controller:

Reads the Elasticsearch Config and SecretMounts from the StackConfigPolicy CR
Validates that the secrets mentioned in the SecretMounts actually exist
Marshals the Config and SecretMounts and creates a new secret in the Elasticsearch cluster namespace
Replicates the Secrets in the SecretMounts to the Elasticsearch cluster namespace, so that the secrets can actually be mounted into Elasticsearch Pod

Elasticsearch controller:

Reads the secret create by the StackConfigPolicy controller which contains the Elasticsearch config and secret mounts
Merges the Elasticsearch config to the elasticsearch.yml that the controller creates
Adds the secrets from secretMounts as volume mounts to the Elasticsearch Pods.

Tasks pending:

Clean up code and remove redundant code
Add unit tests
Test complex scenarios with different Elasticsearch cluster configurations

pkg/controller/stackconfigpolicy/controller.go

pkg/controller/stackconfigpolicy/elasticsearch_config_secret_mounts_settings.go

pkg/controller/stackconfigpolicy/controller.go

pkg/apis/elasticsearch/v1/name.go

pkg/controller/stackconfigpolicy/elasticsearch_config_secret_mounts_settings.go

pkg/controller/elasticsearch/nodespec/resources.go

Co-authored-by: Michael Morello <michael.morello@gmail.com>

pkg/apis/stackconfigpolicy/v1alpha1/stackconfigpolicy_types.go

pkg/apis/elasticsearch/v1/name.go

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

pkg/controller/stackconfigpolicy/controller.go

pkg/controller/stackconfigpolicy/elasticsearch_config_settings.go

pkg/controller/stackconfigpolicy/controller.go

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

pkg/apis/stackconfigpolicy/v1alpha1/stackconfigpolicy_types.go

pkg/apis/elasticsearch/v1/name.go

pkg/controller/common/labels/labels.go

pkg/controller/stackconfigpolicy/controller.go

pkg/controller/common/labels/labels.go

pkg/controller/elasticsearch/nodespec/policy_config.go

pkg/apis/stackconfigpolicy/v1alpha1/webhook.go

pkg/controller/stackconfigpolicy/elasticsearch_config_settings.go

barkbay · 2023-11-08T09:18:28Z

pkg/controller/stackconfigpolicy/elasticsearch_config_settings.go

+}
+
+// reconcileSecretMounts creates the secrets in SecretMounts to the respective Elasticsearch namespace where they should be mounted to.
+func reconcileSecretMounts(ctx context.Context, c k8s.Client, es esv1.Elasticsearch, policy *policyv1alpha1.StackConfigPolicy) error {


I'm not sure to understand if Secrets which are no longer referenced in a policy are deleted.

currently if a secret is removed from a stack config policy, that secret gets left behind(it is deleted only when the stack config policy itself is deleted or the elasticsearch cluster is). I was going to address this in a separate PR, as even today we don't reset a secret that is no longer referenced in the stackconfig policy I believe.

Co-authored-by: Michael Morello <michael.morello@gmail.com>

pebrc

LGTM just a few nits.

pkg/controller/stackconfigpolicy/elasticsearch_config_settings.go

pkg/apis/elasticsearch/v1/name.go

pkg/apis/stackconfigpolicy/v1alpha1/stackconfigpolicy_types.go

pkg/controller/elasticsearch/nodespec/policy_config_test.go

pkg/controller/elasticsearch/settings/merged_config_test.go

pkg/controller/stackconfigpolicy/controller_test.go

Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

kvalliyurnatt · 2023-11-14T15:54:42Z

buildkite test this -f p=gke

…config policy (elastic#7233) Introduce two new fields in StackConfigPolicy, ElasticsearchConfig and SecretMounts. The ElasticsearchConfig field can be used to add configurations for the Elasticsearch clusters managed by the StackConfigurationPolicy, these configurations typically go into the elasticsearch.yml file.SecretMounts are used add additional secrets that need to be mounted into the Elasticsearch pods. --------- Co-authored-by: Michael Morello <michael.morello@gmail.com> Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com> Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

kvalliyurnatt added 4 commits October 12, 2023 20:10

initial commit to enable elasticsearch config in stack config

5fce6b6

add ownerships to secrets and make sure secrets are cleaned up

7deebe0

update status only after settings are applied

3e51476

refactor some code

27011af

kvalliyurnatt added >feature Adds or discusses adding a feature to the product v2.10.0 labels Oct 13, 2023

kvalliyurnatt requested review from thbkrkr, pebrc, barkbay, naemono and rhr323 October 13, 2023 21:49

barkbay changed the title ~~[WIP] Intorduce Elasticsearch config and additional secret mounts to stack config policy~~ [WIP] Introduce Elasticsearch config and additional secret mounts to stack config policy Oct 16, 2023

barkbay reviewed Oct 16, 2023

View reviewed changes

rhr323 added v2.11.0 and removed v2.10.0 labels Oct 23, 2023

kvalliyurnatt and others added 9 commits October 23, 2023 11:28

Apply suggestions from code review

8fade13

Co-authored-by: Michael Morello <michael.morello@gmail.com>

clean up some unused code

553830a

change secret name to indicate it contains policy secret

e76cb41

consolidate redundant code

30606d3

add unit test for secret mounts

07daecb

lint changes

a990794

add api doc changes

2ced2d4

fix unit test

c852a52

adding unit tests for config reconcile

42f77ca

kvalliyurnatt requested a review from barkbay October 24, 2023 21:57

adding additional unit tests

54d23d8

kvalliyurnatt changed the title ~~[WIP] Introduce Elasticsearch config and additional secret mounts to stack config policy~~ Introduce Elasticsearch config and additional secret mounts to stack config policy Oct 24, 2023

kvalliyurnatt marked this pull request as ready for review October 24, 2023 23:23

thbkrkr reviewed Oct 27, 2023

View reviewed changes

pkg/apis/stackconfigpolicy/v1alpha1/stackconfigpolicy_types.go Outdated Show resolved Hide resolved

pkg/apis/elasticsearch/v1/name.go Outdated Show resolved Hide resolved

pkg/apis/elasticsearch/v1/name.go Outdated Show resolved Hide resolved

Apply suggestions from code review

780f78b

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

thbkrkr reviewed Nov 7, 2023

View reviewed changes

pkg/controller/stackconfigpolicy/controller.go Outdated Show resolved Hide resolved

pkg/controller/stackconfigpolicy/elasticsearch_config_settings.go Outdated Show resolved Hide resolved

thbkrkr reviewed Nov 7, 2023

View reviewed changes

pkg/controller/stackconfigpolicy/controller.go Outdated Show resolved Hide resolved

kvalliyurnatt and others added 2 commits November 7, 2023 18:20

Apply suggestions from code review

b68d215

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

rename the secrets using hash of secret names

bf1f8e4

kvalliyurnatt requested review from pebrc and thbkrkr November 8, 2023 00:34

kvalliyurnatt added 2 commits November 7, 2023 19:36

run make generate

2215ea1

fix linting issues

fdc2ea3

barkbay reviewed Nov 8, 2023

View reviewed changes

kvalliyurnatt and others added 2 commits November 8, 2023 09:24

Apply suggestions from code review

46edfc8

Co-authored-by: Michael Morello <michael.morello@gmail.com>

add additional unit tests

258a175

kvalliyurnatt requested a review from barkbay November 9, 2023 00:32

kvalliyurnatt added 7 commits November 8, 2023 19:38

update generate code

be06eca

removing TODO comment about backwards compatability

2118384

running make generate

0c8c7c8

fixing a linter issue

99c62f0

rearranfing the if else chain

306755f

fix some mroe linting issues

cda836b

fix linting issues

ac942a2

pebrc reviewed Nov 12, 2023

View reviewed changes

kvalliyurnatt and others added 3 commits November 13, 2023 09:48

Apply suggestions from code review

1cf8bba

Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

address review comments

dab7c40

fix linting issues

2e47f52

kvalliyurnatt requested a review from pebrc November 13, 2023 22:33

pebrc approved these changes Nov 14, 2023

View reviewed changes

kvalliyurnatt enabled auto-merge (squash) November 14, 2023 22:15

kvalliyurnatt merged commit 20f0467 into elastic:main Nov 15, 2023
5 checks passed

kvalliyurnatt mentioned this pull request Jan 3, 2024

Release notes and highlights for 2.11.0 release #7428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce Elasticsearch config and additional secret mounts to stack config policy #7233

Introduce Elasticsearch config and additional secret mounts to stack config policy #7233

kvalliyurnatt commented Oct 13, 2023 •

edited

Loading

barkbay Nov 8, 2023

kvalliyurnatt Nov 8, 2023

pebrc left a comment

kvalliyurnatt commented Nov 14, 2023

Introduce Elasticsearch config and additional secret mounts to stack config policy #7233

Introduce Elasticsearch config and additional secret mounts to stack config policy #7233

Conversation

kvalliyurnatt commented Oct 13, 2023 • edited Loading

barkbay Nov 8, 2023

Choose a reason for hiding this comment

kvalliyurnatt Nov 8, 2023

Choose a reason for hiding this comment

pebrc left a comment

Choose a reason for hiding this comment

kvalliyurnatt commented Nov 14, 2023

kvalliyurnatt commented Oct 13, 2023 •

edited

Loading