Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Traffic Route L7 proposal #2013

Merged
merged 3 commits into from
Jun 8, 2021
Merged

Traffic Route L7 proposal #2013

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6ecb880
traffic route l7 proposal
jakubdyszkiewicz May 20, 2021
2374202
comment updates
jakubdyszkiewicz May 24, 2021
ec6c802
changes to modify section
jakubdyszkiewicz May 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 182 additions & 0 deletions docs/proposals/L7-traffic-route.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
# L7 Traffic Routing

## Context

Kuma provides traffic management with `TrafficRoute` policy based on Kuma tags.
We can split the traffic to different destinations, we can redirect the traffic etc.

## Requirements

Provide a way to manage the traffic based on HTTP headers, path and method.

## Configuration

```yaml
type: TrafficRoute
name: route-all-default
mesh: default
sources:
- match:
kuma.io/service: web
destinations:
- match:
kuma.io/service: backend
conf:
http: # a new http section which is a list
- match: # match is required and need to have at least one sub-element
method: GET
path: # one of either prefix, exact or regex will be allowed
prefix: /users
jakubdyszkiewicz marked this conversation as resolved.
Show resolved Hide resolved
exact: /users/user-1
regex:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest a match-type field:

path:
    type: exact
    value: /users/

This is consistent with Kubernetes API conventions and allows valid match types to be automatically validated by API annotations. It also makes validation check unambigious.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, there is a feature of protobuf called "oneof" and providing a couple of options and picking one of them is a convention of Envoy. We use this already in many places. Even loadBalancer in TrafficRoute

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were they also migrating from "oneof" convention?
Validation on API annotations? I don't think I'm familiar with this, can you tell me more? We have our own unified validators for both Kubernetes and Universal.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The kubernetes CRD generator support validation that will propagate checks into the openapi spec (see kubebuilder docs).

I'm confide about the protobuf reference. Is this a protobuf API? If so, shouldn't it be in proto3 rather than in yaml?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kuma API is described in protobuf and then translated between JSON / YAML (but also sent between control planes in multizone deployment as serialized protobuf). In proposals, we tend to use YAML examples so it's easier to read, but you might be right that maybe we should also include proto schemas.

Protobuf also has a builtin validation functionality but we found this to not be flexible enough (descriptive messages) for our needs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see. I'd probably design a YAML-oriented API differently than a protobuf-oriented API :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, Kubernetes CRD validation doesn't support unions, or one-of. There are generator annotations for it, but they don't do anything 😂

headers:
some-header: # one of either prefix, exact or regex will be allowed
exact: some-value
prefix: some-
regex
Copy link
Contributor

@jpeach jpeach May 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gateway API just moved away from this pattern, to a style that is more consistent with Kubernetes API guidelines:

headers:
- name: some-header
  value: some-value
  matchType: exact

Similar reasons as for path matches. This structure is also consistent with the structure you have in the modify section.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the kubernetes rationale for avoiding maps, https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#lists-of-named-subobjects-preferred-over-maps

modify: # optional section
path: # either rewritePrefix or regex
rewritePrefix: /not-users # validation that it can be used only if there is prefix in match
regex: # (example to change the path from "/service/foo/v1/api" to "/v1/api/instance/foo")
pattern: "^/service/([^/]+)(/.*)$"
substitution: "\2/instance/\1"
host: # either value or fromPath
value: "XYZ"
fromPath: # (example to extract "envoyproxy.io" host header from "/envoyproxy.io/some/path" path)
pattern: "^/(.+)/.+$"
substitution: "\1"
requestHeaders:
add:
- name: x-custom-header
value: xyz
append: true # if true then if there is x-custom-header already, it will append xyz
remove:
- name: x-something
responseHeaders:
add:
- name: x-custom-header
value: xyz
append: true
remove:
- name: x-something
destination: # required either split or a destination. Destination is a syntax sugar over split to one destination with weight 100
kuma.io/service: usr_svc_6379
version: '1.0'
split:
- weight: 90
destination:
kuma.io/service: usr_svc_6379
version: '1.0'
- weight: 10
destination:
kuma.io/service: usr_svc_6379
version: '2.0'
destination: # if we have a http section and we don't match anything, then this rule is applied. It's again destination or split
kuma.io/service: usr_svc_6379
version: '1.0'
split:
- weight: 100
destination:
kuma.io/service: usr_svc_6379
version: '1.0'
```

### More examples

1) Canary deployment

**Use case:** when we release something on prod, but want to check functionality before serving it to all the users

When any service is trying to consume `backend` with header `canary: "true"` the traffic is redirected to 1.1 backend, otherwise the traffic is passed to a 1.0 backend.

```yaml
type: TrafficRoute
name: canary
mesh: default
sources:
- match:
kuma.io/service: '*'
destinations:
- match:
kuma.io/service: backend
conf:
http:
- match:
headers:
canary:
exact: "true"
destination:
kuma.io/service: backend
version: '1.1'
destination:
kuma.io/service: backend
version: '1.0'
```

2) Redirecting traffic to a different service

**Use case:** Extracting a new `offers` microservice from `backend`, we can gradually shift the traffic with the `TrafficRoute`

```yaml
type: TrafficRoute
name: reroute
mesh: default
sources:
- match:
kuma.io/service: '*'
destinations:
- match:
kuma.io/service: backend
conf:
http:
- match:
path:
prefix: "/offers"
destination:
kuma.io/service: offers
destination:
kuma.io/service: backend
```

3) Generic canary

With the "traditional" `split` there is support for `*` in the tags. We should also support this use case in `http`.
So this definition is similar to 1) but more generic.

```yaml
type: TrafficRoute
name: canary
mesh: default
sources:
- match:
kuma.io/service: '*'
destinations:
- match:
kuma.io/service: '*'
conf:
http:
- match:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this match object?

headers:
canary:
exact: "true"
split:
- destination:
kuma.io/service: '*'
version: canary
destination:
kuma.io/service: '*'
```

## Other notes

* Split will require weight

## Matching

`TrafficRoute` will preserve same behavior of matching, meaning that we will only take into account `sources` and `destinations`
for matching and then take `http` section from it.

## Protocol

If we have `http` section, it will be only applied on HTTP traffic.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assume that "HTTP" in this context means, http, https, http/2 and so on?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https yes, but only when mTLS is provided by Kuma. Otherwise, we cannot decrypt the traffic

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok you confused me with mtls :) why is mtls required here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imagine there is a client and server with sidecars

A <-> A's sidecar <-> B's sidecar <-> B

If A itself does not encrypt the traffic then even with mTLS A's sidecar can redirect the traffic because it can read the request metadata (it happens before encrypting the traffic)

However, if A encrypts the traffic with TLS itself (for example old service when migrating to mesh), then A's sidecar cannot decrypt the traffic and we cannot route it by matching path, headers, etc.

If service is marked as TCP, then we won't match and the default `destination` is applied.