Implement derive(CELSchema) macro for generating cel validation on CRDs

[Danil-Grigorev]

Motivation

Related to #1367

CRDs allow to declare server-side validation rules using CEL. This functionality is supported via #[schemars(schema_with = "<schemagen-wrapper>")], but requires defining a method with handling logic, which may be error-prone.

Since kube owns CRD generation code, the idea is to simplify this process for added validation rules and achieve more declarative approach, similar to the kubebuilder library. This approach will be compatible with kopium generation based on existing CRD structures, already using CEL expressions.

Solution

Allow for a more native handling of CEL validation rules on the CRDs via a field macro.

#[derive(…, CELSchema)]
#[kube(
  rule = Rule::new("self.metadata.name == 'singleton'"),
…
)]
pub struct FooSpec {
    // Field with CEL validation
    #[serde(default)]
    #[cel_validate(
         rule = Rule::new("self != 'illegal'").message("string cannot be illegal").reason(Reason::FieldValueForbidden),
         rule = Rule::new("self != 'not legal'").reason(Reason::FieldValueInvalid),
     )]
    cel_validated: Option<String>,

This PR is a followup on #1621 which addresses some of the concerns.

1. Implemented by generating a JsonSchema, which allows further additions to the schema later, struct level validation rules and is not affected by schemars version.
2. Based on Implement CEL validation proc macro for generated CRDs #1621 (comment) - using methods from the kube::core and invoking from kube::derive.
3. Generally simplified the code to make it easier to maintain (Implement CEL validation proc macro for generated CRDs #1621 (comment))

Other things tried (TLDR)

Visitor trait:

It is possible to generate a new Visitor implementation per each validation rule. But the problem with this approach is that the generation happens for Validator derive on the structure, while the CustomResource derive is responsible for populating additional visitors for crd(). There is no one specific method which can collect all visitors under one chain, invoked from schemars. This likely requires every individual field in each struct to implement the Validated trait, involving creation of a shemars/serde type of logic.

Then the schemars Schema has no indicators for the source structure in the schema within Visitor, so there is no way (without generating schemars(title = “FooSpec”) as a metadata) to match the added visitor on the processed object param to make modifications. It is possible to add preserve_order feature to schemars and “search” for the property of the structure, as long as the source struct name is mapped to the Schema content.

With generating JsonSchema visitor extensions for more complex scenarios are possible to explore in the future.

Generating schemars attributes

While it is a viable option, such thing is not possible with derive macro, and has to use proc_macro instead, This approach is additionally hiding the updates of derive attributes under the hood, which feels unintuitive, as it performs updates to the macro markers, meant to generate code. Explored in #1621

[codecov]

Codecov Report

Attention: Patch coverage is 76.92308% with 24 lines in your changes missing coverage. Please review.

Project coverage is 75.8%. Comparing base (0424cb4) to head (35c39fe).

Files with missing lines	Patch %	Lines
kube-core/src/cel.rs	56.8%	16 Missing ⚠️
kube-derive/src/cel_schema.rs	89.1%	6 Missing ⚠️
kube-derive/src/lib.rs	0.0%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #1649     +/-   ##
=======================================
+ Coverage   75.8%   75.8%   +0.1%     
=======================================
  Files         82      84      +2     
  Lines       7513    7613    +100     
=======================================
+ Hits        5693    5770     +77     
- Misses      1820    1843     +23     

Files with missing lines	Coverage Δ
kube-derive/src/custom_resource.rs	83.5% <100.0%> (+1.0%)	⬆️
kube-derive/tests/crd_schema_test.rs	96.9% <ø> (ø)
kube/src/lib.rs	88.5% <ø> (ø)
kube-derive/src/lib.rs	0.0% <0.0%> (ø)
kube-derive/src/cel_schema.rs	89.1% <89.1%> (ø)
kube-core/src/cel.rs	56.8% <56.8%> (ø)

[clux]

some comments and questions. i think this is a pretty cool approach.

[clux]

Interestingly, I see these ones in the generated docs under https://github.com/kube-rs/k8s-pb/blob/ce4261fb52266f05cd7a06dbb8f4c0fcaa41c06a/k8s-pb/src/apiextensions_apiserver/pkg/apis/apiextensions/v1/mod.rs#L736 but because of bad go enum usage it's just a doc comment :(

[Danil-Grigorev]

Would not pretend I saw it being generated, but yeah, missing enum is better to have :). Maybe worth adding From/Into conversion for ensuring compatibility.

[clux]

note to self; this may perhaps be churned through a cel crate against an object to perform validation client side. should investigate this later.

[clux]

I think this approach makes sense and is a nice way to opt into injecting validations. Have added some comments for code organisation (file is getting big) and for naming, but ultimately am happy with this!

[clux]

Thanks a lot for this! It's a lot more readable and understandable now as it's factored out. Very minor set of comments, and a few questions for my own sanity. Only want one doc field added.

[clux]

Minor lack: this is a new #[kube( attribute and should have a documentation entry in lib.rs in kube-derive. Something like:

/// ## `#[kube(rule = Rule)]`
/// Inject a top level CEL validation rule for the top level generated struct.
///
/// This attribute is for resources deriving [`CELSchema`] instead of [`JsonSchema`].

[clux]

These lines look identical to the ones in 104-109, could be its own fn with documentation as to why we are filtering out.

[clux]

AFAIKT, there's no real drawbacks to this filtering because it only affects the generated, private Validated struct that serve as a proxy for schema generation. Is my intuition right?

[clux]

So basically, this creates a validated wrapper struct for EVERY field that uses #[cel_validate], and these structs do not clash because they are inlined in a private scope for the purposes of using merge_properties only?