feat: Make DataStore Schema (prefix) Configurable #554

SimonKienzler · 2024-08-21T14:22:15Z

Fixes #548

As agreed in the aforementioned discussion, this PR introduces a new TenantControlPlaneSpec field called DataStoreSchema. The field name could also be different, if desired. It allows users to control the database name or etcd prefix. If the field is unset during creation, the defaulting webhook sets the field to <TCPNamespace>_<TCPName>, which is the current behavior. The field is immutable.

Some remarks:

I think I found bugs in the current behavior of the defaulting webhook:
- it never appended to the operations in the OnCreate function, therefore the replicas were never defaulted.
- the OnUpdate function also had some early returns, so same problem there.
Instead of using a webhook, I propose to use kubebuilder markers with CEL to ensure that the DataStore and DataStoreSchema fields can't be unset after creation. This makes the OnUpdate function a NoOp.
Added some test cases for the defaulting webhook.
Updated the doc string for the DataStore field. According to the docs, datastore migrations are in fact possible.

netlify · 2024-08-21T14:22:37Z

✅ Deploy Preview for kamaji-documentation ready!

Name	Link
🔨 Latest commit	`bbadc70`
🔍 Latest deploy log	https://app.netlify.com/sites/kamaji-documentation/deploys/66f3dbd0e0c7300008adf163
😎 Deploy Preview	https://deploy-preview-554--kamaji-documentation.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

prometherion · 2024-08-22T07:02:04Z

internal/webhook/handlers/tcp_defaults.go

@@ -49,21 +43,21 @@ func (t TenantControlPlaneDefaults) OnDelete(runtime.Object) AdmissionResponse {
 }

 func (t TenantControlPlaneDefaults) OnUpdate(object runtime.Object, oldObject runtime.Object) AdmissionResponse {


Suggested change

func (t TenantControlPlaneDefaults) OnUpdate(object runtime.Object, oldObject runtime.Object) AdmissionResponse {

func (t TenantControlPlaneDefaults) OnUpdate(runtime.Object, runtime.Object) AdmissionResponse {

prometherion · 2024-08-22T07:18:53Z

api/v1alpha1/tenantcontrolplane_types.go

@@ -249,12 +249,20 @@ type AddonsSpec struct {
 }

 // TenantControlPlaneSpec defines the desired state of TenantControlPlane.
+// +kubebuilder:validation:XValidation:rule="!has(oldSelf.dataStore) || has(self.dataStore)", message="unsetting the dataStore is not supported"


A CEL policy here would create a breaking change since it requires Kubernetes >= 1.25, and Kamaji has a loose requirement of +1.22 (https://kamaji.clastix.io/reference/versioning/)

@bsctl WDYT here?

About version, frankly I don’t care. 1.22 is a very old release and we can move to 1.25 as baseline for support. If CEL usage provides benefits over webhook and if we want push in this direction for the future, I think it is a good idea, but I would avoid a mix and match of different techniques unless it provides clear benefits, just for sake of maintenance.

From my experience, it's convenient to handle as many validation/defaulting tasks as possible using the kubebuilder annotations, i.e. letting the K8s API Server handle that for you. Except for the things that need a more sophisticated logic, then resort to webhooks

prometherion · 2024-08-22T07:21:42Z

it never appended to the operations in the OnCreate function, therefore the replicas were never defaulted.

Are you referring to changes introduced with 071e679? It seems fine, wondering why we never had a report, maybe everyone's hardly setting values.

SimonKienzler · 2024-08-22T07:51:47Z

Regarding the failing E2E test: The Helm Chart (charts/kamaji) is used for installing Kamaji in the KinD cluster. Is there a make target or script to update the Helm Chart, which I'm missing?

SimonKienzler · 2024-08-23T06:11:34Z

Another question that came to my mind: What would be the preferred way of handling existing TCP CRDs? Should the controller handle this silently (adopting the spec from the status)?

prometherion · 2024-08-23T06:26:07Z

Reading from the status is the way to preserve consistency since it's our state machine declaration, and saved us from many issues.

Users will need to patch their TCP with the newer version of the CRD just to be sure to have a consistent status.

prometherion · 2024-08-23T10:18:52Z

Is there a make target or script to update the Helm Chart, which I'm missing?

It depends on your dev environment, I'm used to developing with kind and installing Kamaji using the local Helm Chart, changing the ImagePullPolicy of the manager Deployment to Never and overriding the image by uploading it to the Kind instances.

Debugging from your IDE is much more complicated since it requires some hacking of the webhooks with tunnelling, as well as having Kamaji with a TLS certificate: I'm used to ngrok, and happy to provide further help, let's move to DM on the Kubernetes Slack workspace.

This also includes converting OnUpdate() to a no-op, as the existence and immutability of the fields are already checked by the API server, thanks to kubebuilder markers. The webhook ensures that fields like dataStore, dataStoreSchema are defaulted during creation (if unset), and the CEL expressions prohibit unsetting them during update.

SimonKienzler · 2024-09-02T13:19:38Z

Okay, with the changes from #565 incorporated, the E2E test suite passed. Nice 🎉

Regarding the adoption of existing TCP resources, this is less complicated than I thought. As the status will always be the same pattern as what the webhook will default to (<namespace>_<name>), ~~there's nothing special to do in terms of "adopting" resources created with the previous CRD version. We just need to make sure the webhook will just fill in the spec field on updates.~~ we can adopt the dataStoreName from the status on first reconcilation after a controller update. So, open points are:

I will add a check to the controller if the Status.Storage.Setup.Schema field is already set, and if yes, adopt it to the spec.
Would you like me to contribute specific E2E test scenarios regarding the dataStoreSchema field?
Regarding the discussion of 1.22 vs 1.25 as a minimum requirement, I understood that you'd be fine with Kamaji requiring Kubernetes 1.25. I will add a commit to this PR documenting this, unless you tell me otherwise.
Anything else you want me to do before this change can be approved?

internal/resources/datastore/datastore_storage_config.go

prometherion · 2024-09-03T18:18:33Z

I will add a check to the controller if the Status.Storage.Setup.Schema field is already set, and if yes, adopt it to the spec.

Just reviewed and suggested some prioritizations.

Would you like me to contribute specific E2E test scenarios regarding the dataStoreSchema field?

That's enough, thanks.

Regarding the discussion of 1.22 vs 1.25 as a minimum requirement, I understood that you'd be fine with Kamaji requiring Kubernetes 1.25. I will add a commit to this PR documenting this, unless you tell me otherwise.

Let's implement the CEL language as you suggested, I'll open a PR once we have it merged to document this change.

Anything else you want me to do before this change can be approved?

LGTM!

Co-authored-by: Dario Tranchitella <dario@tranchitella.eu>

prometherion · 2024-09-23T13:29:55Z

charts/kamaji/crds/kamaji.clastix.io_tenantcontrolplanes.yaml

@@ -6415,8 +6415,18 @@ spec:
                  description: |-
                    DataStore allows to specify a DataStore that should be used to store the Kubernetes data for the given Tenant Control Plane.
                    This parameter is optional and acts as an override over the default one which is used by the Kamaji Operator.
-                    Migration from a different DataStore to another one is not yet supported and the reconciliation will be blocked.
+                    Migration from a different DataStore to another one is supported, see: https://kamaji.clastix.io/guides/datastore-migration/


This is incorrect; rather, this sentence should explain we do not offer migration between Datastore backed by different drivers.

prometherion reviewed Aug 22, 2024

View reviewed changes

SimonKienzler added 10 commits September 2, 2024 11:03

feat: Add DataStoreSchema field to TCP spec

58810ad

feat: Read DB_SCHEMA from TCP spec field

1ed6767

feat: Default DataStoreSchema in webhook

54b4e8c

fix: Catch unsetting the dataStore via CEL

6fa6862

test: Add tests for defaulting webhook

dbe5fef

fix: typo

cf32e27

fix: Linter issues

5f8dbda

fix: make apidoc

12779f7

Update TCP CRD in charts folder

d02c580

SimonKienzler force-pushed the disc-548/configurable-schema branch from 4539724 to d02c580 Compare September 2, 2024 09:20

SimonKienzler added 2 commits September 2, 2024 11:45

fix: Don't run E2E tests during make test

edd6a3e

fix: Use proper metav1 import name

8a0d469

feat: Handle updates of TCPs without dataStoreSchema (+ tests)

007be8e

prometherion reviewed Sep 3, 2024

View reviewed changes

internal/resources/datastore/datastore_storage_config.go Outdated Show resolved Hide resolved

fix: Prioritize Status over Spec

be15eb7

Co-authored-by: Dario Tranchitella <dario@tranchitella.eu>

prometherion reviewed Sep 23, 2024

View reviewed changes

SimonKienzler added 2 commits September 24, 2024 16:00

Update goDoc on DataStore field

912c0c0

make apidoc

bbadc70

prometherion merged commit 8b71843 into clastix:master Oct 2, 2024
10 checks passed

lbrunOVH mentioned this pull request Oct 21, 2024

feat: add support for the datastore schema in kamajicontrolplane clastix/cluster-api-control-plane-provider-kamaji#135

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Make DataStore Schema (prefix) Configurable #554

feat: Make DataStore Schema (prefix) Configurable #554

SimonKienzler commented Aug 21, 2024

netlify bot commented Aug 21, 2024 •

edited

Loading

prometherion Aug 22, 2024

prometherion Aug 22, 2024

bsctl Aug 22, 2024 •

edited

Loading

SimonKienzler Aug 22, 2024

prometherion commented Aug 22, 2024

SimonKienzler commented Aug 22, 2024

SimonKienzler commented Aug 23, 2024

prometherion commented Aug 23, 2024

prometherion commented Aug 23, 2024

SimonKienzler commented Sep 2, 2024 •

edited

Loading

prometherion commented Sep 3, 2024

prometherion Sep 23, 2024

		@@ -49,21 +43,21 @@ func (t TenantControlPlaneDefaults) OnDelete(runtime.Object) AdmissionResponse {
		}

		func (t TenantControlPlaneDefaults) OnUpdate(object runtime.Object, oldObject runtime.Object) AdmissionResponse {

feat: Make DataStore Schema (prefix) Configurable #554

feat: Make DataStore Schema (prefix) Configurable #554

Conversation

SimonKienzler commented Aug 21, 2024

netlify bot commented Aug 21, 2024 • edited Loading

✅ Deploy Preview for kamaji-documentation ready!

prometherion Aug 22, 2024

Choose a reason for hiding this comment

prometherion Aug 22, 2024

Choose a reason for hiding this comment

bsctl Aug 22, 2024 • edited Loading

Choose a reason for hiding this comment

SimonKienzler Aug 22, 2024

Choose a reason for hiding this comment

prometherion commented Aug 22, 2024

SimonKienzler commented Aug 22, 2024

SimonKienzler commented Aug 23, 2024

prometherion commented Aug 23, 2024

prometherion commented Aug 23, 2024

SimonKienzler commented Sep 2, 2024 • edited Loading

prometherion commented Sep 3, 2024

prometherion Sep 23, 2024

Choose a reason for hiding this comment

netlify bot commented Aug 21, 2024 •

edited

Loading

bsctl Aug 22, 2024 •

edited

Loading

SimonKienzler commented Sep 2, 2024 •

edited

Loading