Skip to content

Commit

Permalink
Separate direct and hierarchical policy types
Browse files Browse the repository at this point in the history 
Signed-off-by: Nick Young <nick@isovalent.com>
  • Loading branch information
@youngnick
youngnick committed Feb 27, 2023
1 parent 3f07085 commit 780fb75
Showing 1 changed file with 137 additions and 87 deletions.
224 changes: 137 additions & 87 deletions site-src/geps/gep-713.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,35 +5,43 @@

## TLDR

This GEP aims to standardize attachment of one Kubernetes object to another in
Gateway API by establishing a pattern which defines how _metaresources_ such as
`Policy` API types can have their relevant effects applied to network traffic.
_Metaresources_ are a type of object that _augments_ the behavior of an object
in a standard way. ReferenceGrant is an example of this general type of metaresource.

"Policy Attachment" is a specific type of _metaresource_ that controls how policy
can flow _across_ multiple objects in a hierarchy.
Individual policy APIs (e.g. `TimeoutPolicy`, `RetryPolicy`, etc) must include a
common `TargetRef` field in their specification to identify how and where to
apply that policy, along with a consistent set of behaviors around defaults and
overrides.

This will be important for providing a consistent experience across
implementations of the API, even for configuration details that may not be fully
portable.
This GEP aims to standardize terminology and process around using one Kubernetes
object modify the functions of one or more other objects.

Generally, a Kubernetes object that that _augments_ the behavior of an object
in a standard way is called a _Metaresource_ are a type of object. ReferenceGrant
is an example of this general type of metaresource, but it is far from the only
one.

This document proposes controlling the creation of configuration in the underlying
Gateway data plane using two types of Policy Attachment.
A "Policy Attachment" is a specific type of _metaresource_ that can affect specific
settings across either one object (this is "Direct Policy Attachment"), or objects
in a hierarchy (this is "Hierarchical Policy Attachment").

Individual policy APIs must
- be their own CRDs (e.g. `TimeoutPolicy`, `RetryPolicy` etc),
- can be included in the Gateway API API group and installation or be defined by
implementations
- and must include a common `TargetRef` struct in their specification to identify how and where to
apply that policy, along with a field that indicates if the reference is `Direct`
or `Hierarchical`. (Shorter name suggestions welcomed)

For Hierarchical Policies, this GEP also describes a set of expected behaviors
for how settings can flow across a defined hierarchy.


## Goals

* Clearly communicate what metaresources are, and that policy resources are a
specific _type_ of metaresource
* Establish a pattern for metaresources which will be used for any metaresources
or policies included in the Gateway API spec
* Establish a pattern for metaresources and policy attachment which should be
used for any implementation specific policies used with Gateway API resources
* Establish a pattern for Policy objects which will be used for any policies
included in the Gateway API spec
* Establish a pattern for Policy attachment, whether Direct or Hierarchical,
which must be used for any implementation specific policies used with
Gateway API resources
* Provide a way to distinguish between required and default values for all
policy API implementations
* Enable policy attachment at all relevant scopes, including Gateways, Routes,
Backends
Backends, along with how values should flow across a hierarchy if necessary
* Ensure the policy attachment specification is generic and forward thinking
enough that it could be easily adapted to other grouping mechanisms like
Namespaces in the future
Expand All @@ -54,8 +62,9 @@ When designing Gateway API, one of the things we’ve found is that we often nee
able change the behavior of objects without being able to make changes to the spec
of those objects. Sometimes, this is because we can’t change the spec of the object
to hold the information we need ( ReferenceGrant, from GEP-709, affecting Secrets
and Services is an example), and sometimes it’s because we want the behavior change
to flow across multiple objects (this is what Policy Attachment is for).
and Services is an example, as is Direct Policy Attachment), and sometimes it’s
because we want the behavior change to flow across multiple objects
(this is what Hierarchical Policy Attachment is for).

To put this another way, sometimes we need ways to be able to affect how an object
is interpreted in the API, without representing the description of those effects
Expand All @@ -64,81 +73,110 @@ inside the spec of the object.
This document describes the ways we design objects to meet these two use cases,
and why you might choose one or the other.

We use the term “metaresource” to describe both of these use cases, since the
object is in both cases a superset of its settings and the settings of a wrapped
object (or objects). “Meta” here is used in its Greek sense of “more comprehensive”
We use the term “metaresource” to describe the class of objects that _only_ augment
the behavior of another Kubernetes object, regardless of what they are targeting.

“Meta” here is used in its Greek sense of “more comprehensive”
or “transcending”, and “resource” rather than “object” because “metaresource”
is more pronounceable than “metaobject”. Additionally, a single word is better
than a phrase like “wrapper object” or “wrapper resource” overall, although both
of those terms are effectively synonymous with “metaresource”.

All ReferenceGrants are metaresources, and all objects that use Policy Attachment
are metaresources, but metaresources can be more than just those two things.

### Targeted metaresources

ReferenceGrant is an example of a targeted metaresource. It’s tightly bound to a
particular Kind within a single namespace, and only modifies the behavior of the
objects that match its binding (by allowing references to them from outside their
namespace).

We’ve been discussing the idea of having a new object that holds extra capabilities
for a backend Service (something like BackendCapabilities or similar, name TBD),
as a way to implement GEP-1282 (Backend Properties). A new resource here that
referenced a single service and had extra properties that we can’t put into the
Service object is also an example of a targeted metaresource. The intent is to
augment the behavior of the bound Service object with the structured data in the
Spec of BackendCapabilities, since it’s not easy to make changes to a GA upstream
object like Service. (And arguably, Service is already a pretty overloaded
resource, so there will need to be good justification for any changes).

With these two examples in mind, here are some guidelines for when to consider
using a targeted metaresource:
* The number or scope of objects to be modified is limited or singular. Targeted
metaresources should target one specific object (preferred), or a tightly-scoped
set of objects (like ReferenceGrant).
A "Policy Attachment" is a metaresource that affects the fields in existing objects
(like Gateway or Routes), or influences the configuration that's generated in an
underlying data plane.

"Direct Policy Attachment" is when a Policy object references a single object _only_,
and only modifies the fields of or the configuration associated with that object.

"Hierarchical Policy Attachment" is when a Policy object references a single object
_and any child objects of that object_ (according to some defined hierarchy), and
modifies fields of the child objects, or configuration associated with the child
objects.

### Direct Policy Attachment

A Direct Policy Attachment is tightly bound to one or more instances of a particular
Kind within a single namespace, and only modifies the behavior of the object or
objects that match its binding.

As an example, one use case that Gateway API currently does not support is how
to configure details of the TLS required to connect to a backend (in other words,
if the process running inside the backend workload expects TLS, not that some
automated infrastructure layer is provisioning TLS as in the Mesh case).

A hypothetical TLSConnectionPolicy that targets a Service could be used for this,
using the functionality of the Service as describing a set of endpoints. (It
should be also noted this is not the only way to solve this problem, just an
example to illustrate Direct Policy Attachment.)

The TLSConnectionPolicy would look something like this:

```yaml
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSConnectionPolicy
metadata:
name: tlsport8443
namespace: foo
spec:
targetRef: # This struct is defined as part of Gateway API
type: Direct
group: "" # Empty string means core - this is a standard convention
kind: Service
name: fooService
tls:
certificateAuthorityRefs:
- name: CAcert
port: 8443

```

All this does is tell an implementation, that for connecting to port `8443` on the
Service `fooService`, it should assume that the connection is TLS, and expect the
service's certificate to be validated by the chain in the `CAcert` Secret.

Importantly, this would apply to _every_ usage of that Service across any HTTPRoutes
in that namespace, which could be useful for a Service that is reused in a lot of
HTTPRoutes.

With this two examples in mind, here are some guidelines for when to consider
using Direct Policy Attachment:
* The number or scope of objects to be modified is limited or singular. Direct
Policy Attachments should target one specific object (preferred), or a tightly-scoped
set of objects (like all Services in a namespace).
* The modifications to be made to the objects don’t have any transitive information -
that is, the modifications only affect the single object that the targeted
metaresource is bound to, and don’t have ramifications that flow beyond that
object. ReferenceGrant and BackendCapabilities are good examples here because
they don’t expect the metaresource to modify the behavior of more than one
object. (Even in the ReferenceGrant case, the ability to set a Kind rather
than an individual object is syntactic sugar for having to prevent needing to
have lots of ReferenceGrants if you have lots of wrapped objects).
object.
* In terms of status, it should be reasonably easy for a user to understand that
everything is working - basically, as long as the targeted object exists, and
the modifications are valid, the metaresource is valid, and this should be
straightforward to communicate in one or two Conditions. Note that at the time
of writing, this is *not* completed.
* Be extremely careful in using metaresources to target things in other namespaces.
In general, this is a big security problem, unless the referent resources have
some way to decline. Most of the time, it’s better to have the resource target
something in its own namespace (that is, the targeted resource describes what
should have its behavior modified), with other, similar but distinct fields
then describe what to target for further modifications. An example:
* A metaresource that constrains client behavior when reaching out to other
services should use a targetRef-like field to choose the clients, and have
another field that sets what the external targets of the constrained behavior
are. (The external targets may conceivably be “all external targets”).
In general, metaresources that modify the behavior of things outside their own
namespace should be avoided unless it uses a handshake of some sort, where the
things outside the namespace can opt–out of the behavior. (Notably, this is the
design that we used for ReferenceGrant).)

### Policy Attachment: It's all about the defaults and overrides

Because a Policy is a metaresource, it targets some other resource and _augments_
its behavior.

But why have this distinct from other types of metaresource? Because Policy
resources are designed to have a way for settings to flow down a hierarchy.
* Direct Policy Attachment _should_ only be used to target objects in the same
namespace as the Policy object. Allowing cross-namespace references brings in
significant security concerns, and/or difficulties about merging cross-namespace
policy objects. Notably, Mesh use cases may need to do something like this for
consumer policies, but in general, Policy objects that modify the behavior of
things outside their own namespace should be avoided unless it uses a handshake
of some sort, where the things outside the namespace can opt–out of the behavior.
(Notably, this is the design that we used for ReferenceGrant).)

### Hierarchical Policy Attachment: It's all about the defaults and overrides

Because a Hierarchical Policy is a metaresource, it targets some other resource
and _augments_ its behavior.

But why have this distinct from other types of metaresource? Because Hierarchical
Policy resources are designed to have a way for settings to flow down a hierarchy.

Defaults set the default value for something, and can be overridden by the
“lower” objects (like a connection timeout default policy on a Gateway being
overridable inside a HTTPRoute), and Overrides cannot be overridden by “lower”
objects (like setting a maximum client timeout to some non-infinite value at the
Gateway level to stop HTTPRoute owners from leaking connections over time).

Here are some guidelines for when to consider using a Policy object:
Here are some guidelines for when to consider using a Hierarchical Policy object:
* The settings or configuration are bound to one containing object, but affect
other objects attached to that one (for example, affecting HTTPRoutes attached
to a single Gateway, or all HTTPRoutes in a GatewayClass).
Expand All @@ -153,7 +191,6 @@ Here are some guidelines for when to consider using a Policy object:
is not, and needs to be carefully designed to avoid fanout apiserver load.
(This is not built at all in the current design either).


## API

This approach is building on concepts from all of the alternatives discussed
Expand All @@ -162,16 +199,18 @@ but also borrows some concepts from the [ServicePolicy
proposal](https://github.com/kubernetes-sigs/gateway-api/issues/611).

### Policy Attachment for Ingress
Attaching policy to Gateway resources for ingress use cases is relatively
straightforward. A policy can reference the resource it wants to apply to.
Attaching a Directly Attached Policy to Gateway resources for ingress use cases
is relatively straightforward. A policy can reference the resource it wants to
apply to.

Access is granted with RBAC - anyone that has access to create a RetryPolicy in
a given namespace can attach it to any resource within that namespace.

![Simple Ingress Example](images/713-ingress-simple.png)

To build on that example, it’s possible to attach policies to more resources.
Each policy applies to the referenced resource and everything below it in terms
of hierarchy. Although this example is likely more complex than many real world
A Hierarchical Policy can attach to a parent resource, and then each policy
applies to the referenced resource and everything below it in terms of hierarchy.
Although this example is likely more complex than many real world
use cases, it helps demonstrate how policy attachment can work across
namespaces.

Expand Down Expand Up @@ -216,6 +255,13 @@ The `targetRef` field MUST have the following structure:
```go
// PolicyTargetReference identifies an API object to apply policy to.
type PolicyTargetReference struct {
// Type is the type of the policy attachment.
// This can be either Direct or Hierarchical.
//
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=253
Type string `json:"type"`

// Group is the group of the target resource.
//
// +kubebuilder:validation:MinLength=1
Expand Down Expand Up @@ -340,6 +386,7 @@ spec:
includeProtocol: true
includeQueryString: true
targetRef:
type: hierarchical
kind: Gateway
name: example
---
Expand All @@ -350,6 +397,7 @@ spec:
cachePolicy:
includeQueryString: false
targetRef:
type: direct
kind: HTTPRoute
name: example
```
Expand Down Expand Up @@ -393,6 +441,7 @@ spec:
default:
maxRetries: 5
targetRef:
type: direct
group: networking.acme.io
kind: ExternalService
name: foo.com
Expand All @@ -405,6 +454,7 @@ the same resource _and_ have an identical field specified with different values,
precedence MUST be determined in order of the following criteria, continuing on
ties:
* Hierarchical Policies override Direct Policies.
* The oldest Policy based on creation timestamp. For example, a Policy with a
creation timestamp of "2021-07-15 01:02:03" is given precedence over a Policy
with a creation timestamp of "2021-07-15 01:02:04".
Expand Down

0 comments on commit 780fb75

Please sign in to comment.