Document problems with setters / parameters, and alternatives

[bgrant0607]

This is a reminder for me to write a doc about this.

I covered over-parameterization in the rationale doc and elsewhere (https://github.com/kubernetes/design-proposals-archive/blob/8da1442ea29adccea40693357d04727127e045ed/architecture/declarative-application-management.md#parameterization-pitfalls).

Setters have kind of been an attractive nuisance, similar to vars in kustomize. I would recommend against the use of setters entirely, or at least try to minimize their use.

Setters are just template parameters with different syntax. Unlike templating tools, however, when a few simple parameters are not sufficient, in kpt it's unclear what to do next in order to solve the problem.

Setters should not be thought of as interfaces to packages. They are ad hoc transformers of the last resort. Packages aren't encapsulated and don't have interfaces. They're unencapsulated bundles of resources. Packages may contain abstractions, defined by Kubernetes resource types, but don't have abstractions. The same concerns apply to other mechanisms, such as abstractions explicitly defined by resource types or implicitly defined via functions. Such mechanisms should not be tightly coupled to specific packages.

Setters need to be added to packages largely manually, which is error-prone. They imply human editing of text representations of configuration rather than tools and interface surfaces generating configuration and performing transformations to it. Parameters are not an API, nor a resource definition. They are a human interface.

Making it easier to create packages is a goal.

Setters are an obstacle to operating on packages with the package orchestrator, to editing them with the UI, to automating transformations, and to creating blueprints automatically, such as using exported resources.

We should strive to require no special markup of blueprints at all, much as kustomize transforms unmodified base resources. Needing to be aware of inline comments makes functions much harder to write. Carrying human-authored comments through, similar to compiler debug info, is fine and desirable.

The functions don't exist today that will be necessary to make this feasible, but we should start building in that direction rather than continuing to focus on what is, at best, a workaround. If a package contains setters, that’s probably because we didn’t create the right functions, someone had to identify string substitution targets by hand, and other users will need to do that for every package containing the parameterized resource types.

As much as possible, we (the community) should build transformer functions that at least know what their targets are, as with kustomize's built-in transformers, and possibly what where the source values should come from. Well defined inputs would create "APIs" that could be used to chain functions together more easily. Some functions will require resource-type-specific logic, and some horizontal domain-specific logic. This is logic that would otherwise be reimplemented in every "as code" style package.

We are trying to create a new approach in the spectrum between ClickOps and Programming, where using and even creating blueprints can be done by ClickOps, and programming is required less frequently than with Infrastructure as Code. When programming is required, we need SDK / language options to satisfy both the middle (relatively easy but low reuse) and right (harder but high reuse) of the spectrum.

Some "functions" may be embedded in what we think of as user interface surfaces, such as GUIs, CLIs, terminal consoles, mobile apps, ChatOps agents, etc. For instance, the Backstage UI plugin operates on the ResourceList as a function would.

Separating the code from the data enables us to apply the full toolbox of standard UX principles, techniques, and tools to the user experience.

[droot]

Thanks for writing this. Setters have kind of been an attractive nuisance -- it captures the sentiment so well.

[bgrant0607]

For posterity, another setter-like approach, but which uses annotations: https://knot8.io/, https://github.com/mkmik/knot8. Comments are problematic, but not the main problem, so I may want to reference this in the writeup.

[bgrant0607]

One difference with knot8 is that it could be viewed as a mechanism for defining field-path shortcuts for fields that need to be frequently changed, and there's no extra source of truth introduced. It also stores the upstream in an annotation, similar to client-side apply, for easy upgrades.

[bgrant0607]

ytt uses comments as markup even more heavily than other tools, including for embedded code and imports as well as to mark data values to change.
https://carvel.dev/ytt/docs/v0.41.0/how-it-works/

This approach makes it infeasible to operate on as a data format, other than using the built-in templating mechanism.

[yuwenma]

Very looking forward to this. I'd like to refer to this issue when answering the user questions regarding how to use setters for XXX.