Clarification on precedence matching semantics with HTTPRoutes with intersecting hostnames

[arkodg]

Lets take this example where two HTTPRoutes are attached to the same gateway

• HTTPRoute foo-route serves foo.example.com and matches on /foo
• HTTPRoute baz-route serves *.example.com and matches on /baz

What happens when the request has Host: foo.example.com && Path: /baz

• Is the match unsuccessful because it should match foo-route / foo.example.com hostname but cannot find a path match
  OR
• is the match successful because it matches baz-route / *.example.com and finds a match for the /baz path

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: example-gateway
spec:
  gatewayClassName: example-gateway-class
  listeners:
  - name: http
    protocol: HTTP
    port: 80
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: foo-route
spec:
  parentRefs:
  - name: example-gateway
  hostnames:
  - "foo.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /foo
    backendRefs:
    - name: foo-svc
      port: 8080
 ---
 apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: baz-route
spec:
  parentRefs:
  - name: example-gateway
  hostnames:
  - "*.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /baz
    backendRefs:
    - name: baz-svc
      port: 8080     

Here's the relevant part from the HTTPRoute
https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.HTTPRoute

In the event that multiple HTTPRoutes specify intersecting hostnames (e.g. overlapping wildcard matching and exact matching hostnames), precedence must be given to rules from the HTTPRoute with the largest number of:

Characters in a matching non-wildcard hostname.
Characters in a matching hostname.
If ties exist across multiple Routes, the matching precedence rules for HTTPRouteMatches takes over.

[rainest]

This broadens to whether or not inbound request matches can cascade from a more to less specific Listener (where, at present, Hostname is the only match criteria that supports wildcards). Existing language in the spec has:

The incoming hostname MUST be matched using the Hostname field for each Listener in order of most to least specific. That is, exact matches must be processed before wildcard matches.

but does not have explicit language about attempting to continue match finding on less specific Listeners once all attached routes on a more specific Listener have been exhausted.

If we do not allow cascades, and users wish to match both foo.example.com/bar and bar.example.com/bar with foo.example.com and *.example.com Listeners, they would need to create multiple HTTPRoutes with that path, one for each hostname. This requires knowledge of the spec and, if these are effectively the same route (both point to a single upstream service that serves foo.example.com and bar.example.com equally), could cause issues with session affinity and load balancing.

---

No cascade likely has consequences for Gateway collapsing. If you start with GatewayA, which only has a foo.example.com Listener, and GatewayB, which only has a *.example.com Listener, collapsing would mask a bar.example.com/foo route on the latter without cascade matching. This would likely be difficult to handle since the routes are separate resources, and you can't easily rely on knowing the complete set when deciding whether or not to collapse. If you do try, you could hit a scenario where simply adding a route results in two previously collapsed Gateways separating, which arguably breaks the expected separation of roles (application developers arguably should not be able to make changes that affect cluster operator-managed resources' implementation)

---

Kong's implementation cannot easily handle a no-cascade rule. It builds a list matches from the complete set of all routes across all hostnames, sorts them according to specificity, and walks down the list until it finds a match. A route for *.example.com/bar would have lower specificity than a foo.example.com/bar route, but the router will match foo.example.com/bar to the former if the latter does not exist.

We could probably work around this by creating a foo.example.com/ prefix match rule with a proxy-served 404 if there is a foo.example.com Listener and no such user-specified prefix rule. I don't know if this would have any adverse effects but cannot think of any off the top of my head.

That workaround may not work well if the instance also ingests Ingress resources, though handling both those and Gateway API resources is probably ill-advised "here be dragons" territory where compliance with either spec goes a bit awry anyway.

[youngnick]

The more I think about it, the more I think that, if we're going to allow merging (or collapsing) of Gateways, it should only be through an explicit mechanism like that described in #1863, and we should deprecate and/or remove this kind of automatic merge, precisely because of how easy it is to find edge cases that get weird. That's probably a larger topic than this discussion can contain though.

Also, @arkodg, that would be okay as long as the set of rules were distinct in the two HTTPRoutes. This is what the part about the matching precedence rules for HTTPRouteMatches takes over means - although arguably that should say something more like

---

If HTTPRoutes match exactly on hostname, the concatenated list of HTTPRouteRules must be checked for distinct entries, where "distinct" means that they differ on:

• HTTPRouteMatches
• HTTPRouteFilters
• or both

If only the HTTPBackendRefs are distinct, and HTTPRouteMatches and HTTPRouteFilters are identical means that two HTTPRouteRules conflict and the usual conflict resolution rules should be used.

Note that if the either HTTPRouteMatches or HTTPRouteFilters is empty, then they are by definition identical across HTTPRouteRules."

If the HTTPRouteRules are distinct across all matching HTTPRoutes, then all the HTTPRoutes must be accepted for processing.

---

Some users are already using hostname in HTTPRoute as another way of selecting which Listener to bind to, so I don't think we can walk that back now.

The other thing that this raises though is: In the event some rules are distinct, and some are not, do only the distinct rules apply? Well, that depends on how we handle partially-valid Route objects (per #1696). As with that discussion, I believe the answer should be "yes", but this requires solving #1696 as well.

[arkodg]

@youngnick this is my understanding based on the above comments and discussion from the meeting, is this right ?

• if two HTTPRoutess have the identical hostname(s) set, the implementation merges their rules (one phase matching / fall through)
• but if two HTTPRoutes have intersecting hostname(s) such as foo.example.com & *.example.com, the implementation should ensure, the most specific hostname is picked, and only rules associated with it are matched ? (2 phase matching / no fallthrough here)

[youngnick]

That sound mostly right, although I'd argue that in both cases it's still 2 phase matching - we're basically always matching hostname first, then other rules. It's just that there are different rules for exact hostname match vs wildcard hostname match.

Note that if all the hostnames are *.example.com, the rules will be merged as in the first example.

As I said earlier, this is all implicit currently, it seems everyone working on this earlier shared the same mental model, and so the idea it could be done differently never came up.

[arkodg]

thanks for the clarification, this is super useful, does this discussion warrant

1. API doc string changes
2. conformance tests to check for correctness in routes with same and overlapping hostnames

[youngnick]

Yes to both, I think - I think it's probably important to document the implicit assumptions here that:

• Traffic always chooses a listener before doing any other processing
• hostname field on HTTPRoute is a way of having HTTPRoutes influence which Listener they attach to, so generally using hostname does not change that behavior.
• In the case where the Listener has a wildcard and the HTTPRoutes have exact names, then a request that matches an exact name cannot be rerouted to the wildcard if it doesn't match.

The doc updates on #2288 are a bit relevant here as well, I think.