Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contour Ingress & Gateway #468

Merged
merged 82 commits into from
Jun 21, 2024
Merged

Contour Ingress & Gateway #468

merged 82 commits into from
Jun 21, 2024

Conversation

louiseschmidtgen
Copy link
Contributor

@louiseschmidtgen louiseschmidtgen commented Jun 4, 2024

Contour Ingress & Gateway

Ingress

When we k8s enable ingress we get a static deployment of contour:

pod/ck-ingress-contour-contour-5df8788b9d-rbq89   1/1     Running   0          4m56s
pod/ck-ingress-contour-envoy-jll5b                2/2     Running   0          4m56s

And we get this IngressClass:

sudo k8s kubectl get ingressclass
NAME         CONTROLLER                                                    PARAMETERS   AGE
ck-ingress   projectcontour.io/projectcontour/ck-ingress-contour-contour   <none>       108s

When you disable ingress, those will get removed.

When you k8s set default-tls-secret=bananas you will get a TLSCertificateDelegation resource (from the mini /ck-ingress-tls chart) which you can use to point to your bananas secret. See here

sudo k8s set ingress.default-tls-secret=bananas
sudo k8s kubectl get TLSCertificateDelegation ck-ingress-tls-delegation -o yaml -n projectcontour-root
apiVersion: projectcontour.io/v1
kind: TLSCertificateDelegation
metadata:
<shortened>
spec:
  delegations:
  - secretName: bananas
    targetNamespaces:
    - '*'

Now for k8s set enable-proxy-protocol=true this sets a config arg for the static deployment of contour. Contour gets restarted and voila there is an extra Arg:

sudo k8s kubectl describe pod ck-ingress-contour-contour -n projectcontour | grep proxy
      --use-proxy-protocol

Gateway

Now when you do k8s enable gateway you get this dynamic gateway provisioner.

projectcontour     contour-gateway-provisioner-6dbcd4fbb-pd6kc   1/1     Running   0          8m51s

This helm chart lives in ck-gateway. I also added a gateway class to the chart with which people can create their gateway resource.

Additionally, another chart ck-gateway-contour installs:

sudo k8s kubectl get gatewayclass
NAME         CONTROLLER                             ACCEPTED   AGE
ck-gateway   projectcontour.io/gateway-controller   True       108m

You can k8s disable gateway and it will remove all gateway resources.

Shared CRDS

Gateway and Ingress share some common contour CRDs. These are moved to their own chart which lives in ck-contour-common.

sudo k8s kubectl get crd | grep projectcontour
contourconfigurations.projectcontour.io               2024-06-12T08:47:49Z
contourdeployments.projectcontour.io                  2024-06-12T08:47:49Z
extensionservices.projectcontour.io                   2024-06-12T08:47:49Z
httpproxies.projectcontour.io                         2024-06-12T08:47:49Z
tlscertificatedelegations.projectcontour.io           2024-06-12T08:47:49Z

Changes Cilium

Deploy a second gatewayclass named ck-gateway.

sudo k8s kubectl get gatewayclass
NAME         CONTROLLER                     ACCEPTED   AGE
cilium       io.cilium/gateway-controller   True       3m52s
ck-gateway   io.cilium/gateway-controller   True       3m53s

List of images (Moonray):

sudo k8s kubectl get node -o template='{{ range .items }}{{ .metadata.name }}{{":"}}{{ range .status.images }}{{ "\n- " }}{{ index .names 1 }}{{ end }}{{"\n"}}{{ end }}'
louise-inspiron-5580:
- docker.io/calico/node:v3.28.0
- docker.io/calico/cni:v3.28.0
- ghcr.io/canonical/rawfile-localpv:0.8.0-ck5
- docker.io/bitnami/envoy:1.28.2-debian-12-r0
- docker.io/envoyproxy/envoy:v1.29.2
- docker.io/bitnami/contour:1.28.2-debian-12-r4
- docker.io/calico/apiserver:v3.28.0
- docker.io/calico/kube-controllers:v3.28.0
- docker.io/calico/typha:v3.28.0
- k8s.gcr.io/sig-storage/csi-provisioner:v3.4.1
- k8s.gcr.io/sig-storage/csi-resizer:v1.7.0
- k8s.gcr.io/sig-storage/csi-snapshotter:v6.2.1
- ghcr.io/canonical/metrics-server:0.7.0-ck0
- quay.io/tigera/operator:v1.34.0
- ghcr.io/canonical/coredns:1.11.1-ck4
- ghcr.io/projectcontour/contour:v1.28.2
- docker.io/calico/node-driver-registrar:v3.28.0
- k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.10.0
- docker.io/calico/csi:v3.28.0
- docker.io/calico/pod2daemon-flexvol:v3.28.0
- ghcr.io/canonical/k8s-snap/pause:3.10

@louiseschmidtgen louiseschmidtgen changed the title WIP Contour WIP Contour Ingress & Gateway Jun 10, 2024
Copy link
Contributor

@eaudetcobello eaudetcobello left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Two comments and then lgtm

src/k8s/pkg/k8sd/features/contour/ingress.go Outdated Show resolved Hide resolved
src/k8s/pkg/k8sd/features/contour/gateway.go Outdated Show resolved Hide resolved
louiseschmidtgen and others added 3 commits June 19, 2024 14:33
Co-authored-by: eaudetcobello <155978570+eaudetcobello@users.noreply.github.com>
Copy link
Contributor

@neoaggelos neoaggelos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there! Nice work here @louiseschmidtgen !

build-scripts/hack/util.py Outdated Show resolved Hide resolved
@louiseschmidtgen
Copy link
Contributor Author

Ah interesting the api-group seems to be simply projectcontour.io:

10:09:48 louise@louise-Inspiron-5580 k8s-snap ±|KU-864/contour ✗|→ sudo k8s kubectl api-resources --api-group=projectcontour.io
NAME                        SHORTNAMES                           APIVERSION                   NAMESPACED   KIND
contourconfigurations       contourconfig                        projectcontour.io/v1alpha1   true         ContourConfiguration
contourdeployments          contourdeploy                        projectcontour.io/v1alpha1   true         ContourDeployment
extensionservices           extensionservice,extensionservices   projectcontour.io/v1alpha1   true         ExtensionService
httpproxies                 proxy,proxies                        projectcontour.io/v1         true         HTTPProxy
tlscertificatedelegations   tlscerts                             projectcontour.io/v1         true         TLSCertificateDelegation
10:10:02 louise@louise-Inspiron-5580 k8s-snap ±|KU-864/contour ✗|→ sudo k8s kubectl api-resources --api-group=projectcontour.io/v1
NAME   SHORTNAMES   APIVERSION   NAMESPACED   KIND
10:10:17 louise@louise-Inspiron-5580 k8s-snap ±|KU-864/contour ✗|→ sudo k8s kubectl api-resources --api-group=projectcontour.io/v1alpha1 
NAME   SHORTNAMES   APIVERSION   NAMESPACED   KIND

Copy link
Contributor

@neoaggelos neoaggelos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved due to pressure of time, but we need to revisit the tests, as they are a possible source of flakes in the future.

build-scripts/hack/util.py Outdated Show resolved Hide resolved
build-scripts/hack/util.py Outdated Show resolved Hide resolved
Comment on lines +41 to 45
# Get gateway node port
gateway_http_port = None
util.stubbornly(retries=5, delay_s=2).on(session_instance).until(
lambda p: "cilium-gateway-my-gateway" in p.stdout.decode()
lambda p: "my-gateway" in p.stdout.decode()
).exec(["k8s", "kubectl", "get", "service", "-o", "json"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this code here can be dangerous. we repeatedly get the list of services until my-gateway appears on the text. this is right, as the service might take some time to appear (perhaps the 5 retries with 2 seconds backoff is also very short).

However, we then retrieve them again to parse the service info and retrieve the nodePort. Note that this nodePort might not be available upon service creation, so there is a race here: If the test retrieves the service before the ports are configured, then our test will flake.

Instead, we should do all of the logic inside the util.stubbornly. Currently, it's using a lambda, but we should probably write this as a proper function instead, then do the retries until we get the nodePort we were after.

If we don't get the nodePort in time, that's OK, the timeout here will tell us about the failure

Comment on lines +18 to +19
lambda p: "ingress" in p.stdout.decode()
).exec(["k8s", "kubectl", "get", "service", "-A", "-o", "json"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, we fetch services until we see some arbitrary output, then fetch them again. this should all be done inside the util.stubbornly here

@louiseschmidtgen louiseschmidtgen merged commit 78f406c into main Jun 21, 2024
17 checks passed
@louiseschmidtgen louiseschmidtgen deleted the KU-864/contour branch June 21, 2024 13:32
louiseschmidtgen added a commit that referenced this pull request Jul 4, 2024
* put a little note in docs on disabling network

* first draft of contour ingress and gateway

* put contour chart

* minor fixes

* put some fakes

* more wip

* helm client create ns for contour

* gateway

* testy yamls

* gateway provisioner

* ck-gateway

* more gateway

* gateway provider manifest to chart

* remove that

* add mini tls delegation chart

* some cleanup

* set use-proxy-protocol

* yamling, put shared crds in their own chart

* forgot this yaml

* some more comments

* gateway class renamed ck-gateway, some renaming

* testing adjustments & cleanup

* fix ingress class name

* fix cilium ingress class

* try improve tests

* lint

* lint2

* lint

* fix test

* change contour chart version

* update component contour

* update ingress test

* little typo

* ingressClassName patch

* typo

* cleanup cilium ingress class

* update dir

* cleanup

* make contour charts updatable

* fix gateway yamls

* comments

* lost braket,

* beautify

* comments

* move tlscertdelegation

* lint

* undo move enabled config

* update charts

* match on label my-gateway

* improve label check

* linter

* make ingress check more precise

* update helm pull for contour

* wait for contour common crds

* add missing returns

* change name in api resources await

* change chart helm pull

* missed a little name change

* comment update

Co-authored-by: eaudetcobello <155978570+eaudetcobello@users.noreply.github.com>

* update resources for group version

* rewrite wait for crds for two groups

* update wait for crds

* the rest of my comment

* maybe now I understand api group conventions

* Revert "maybe now I understand api group conventions"

This reverts commit 854f205.

* correct api-resource name

* update chart helm pull

* remove resource.group check optional param

* cleanup

* make it two loops

* add images

* comments

---------

Co-authored-by: eaudetcobello <155978570+eaudetcobello@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants