Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Review Ingress. #1995

Merged
merged 17 commits into from
Nov 28, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
e15c05a
Configuration: Prepend `nginx` to annotations.
Gacko Nov 23, 2023
e0c992f
Configuration: Remove `nginx.ingress.kubernetes.io/add-base-url`.
Gacko Nov 23, 2023
3411022
Configuration: Remove `nginx.ingress.kubernetes.io/session-cookie-hash`.
Gacko Nov 23, 2023
9326cac
Configuration: Bump `last_review_date`.
Gacko Nov 23, 2023
69cd4c8
Multiple Ingress NGINX Controllers: Remove deprecated `controller.ing…
Gacko Nov 23, 2023
6c96daf
Multiple Ingress NGINX Controllers: Remove wildcard from `controller.…
Gacko Nov 23, 2023
005d5d1
Multiple Ingress NGINX Controllers: Migrate `configmap` to `controlle…
Gacko Nov 23, 2023
96b8322
Multiple Ingress NGINX Controllers: Improve wording & grammar.
Gacko Nov 28, 2023
5b93cde
Multiple Ingress NGINX Controllers: Bump `last_review_date`.
Gacko Nov 23, 2023
c5614a7
Service Type LoadBalancer: Make `service.beta.kubernetes.io/aws-load-…
Gacko Nov 23, 2023
84273bc
Service Type LoadBalancer: Make `service.beta.kubernetes.io/aws-load-…
Gacko Nov 23, 2023
e1539cc
Service Type LoadBalancer: Move `Other AWS ELB configuration options`.
Gacko Nov 23, 2023
97d9c11
Service Type LoadBalancer: Document Martian Packets when using intern…
Gacko Nov 23, 2023
1403dc8
Service Type LoadBalancer: Document Health Checks failing when using …
Gacko Nov 23, 2023
e4f0f03
Service Type LoadBalancer: Document Security Group configuration on i…
Gacko Nov 23, 2023
027f101
Service Type LoadBalancer: Bump `last_review_date`.
Gacko Nov 23, 2023
8f85aa2
Service Type LoadBalancer: Add exception for Health Checks via PROXY …
Gacko Nov 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ user_questions:
- How can I use Ingress NGINX Controller as a Web Application Firewall?
- How can I protect my workload from malicious requests?
- How can I enable & configure ModSecurity inside of the Ingress NGINX Controller?
last_review_date: 2023-11-07
last_review_date: 2023-11-23
aliases:
- /guides/advanced-ingress-configuration/
- /advanced/ingress/configuration/
Expand Down Expand Up @@ -267,11 +267,11 @@ This functionality is based on the [auth_request](https://nginx.org/en/docs/http

### CORS

To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule add the annotation `ingress.kubernetes.io/enable-cors: "true"`.
To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule add the annotation `nginx.ingress.kubernetes.io/enable-cors: "true"`.

### Rewrite

In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. To circumvent this you can set the annotation `ingress.kubernetes.io/rewrite-target` to the path expected by the service.
In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. To circumvent this you can set the annotation `nginx.ingress.kubernetes.io/rewrite-target` to the path expected by the service.

This can for example be used together with path based routing, when the application expects to be on `/`:

Expand All @@ -297,11 +297,9 @@ spec:
number: SERVICE_PORT
```

If the application contains relative links it is possible to add an additional annotation `ingress.kubernetes.io/add-base-url` that will prepend a [`base` tag](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/base) in the header of the returned HTML from the backend.

### Rate limiting

The annotations `ingress.kubernetes.io/limit-connections` and `ingress.kubernetes.io/limit-rps` define a limit on the connections that can be opened by a single client IP address. This can be used to mitigate [DDoS Attacks](https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus).
The annotations `nginx.ingress.kubernetes.io/limit-connections` and `nginx.ingress.kubernetes.io/limit-rps` define a limit on the connections that can be opened by a single client IP address. This can be used to mitigate [DDoS Attacks](https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus).

`nginx.ingress.kubernetes.io/limit-connections`: Number of concurrent connections allowed from a single IP address.

Expand Down Expand Up @@ -351,10 +349,6 @@ The annotation `nginx.ingress.kubernetes.io/affinity` enables and sets the affin

If you use the `cookie` type you can also specify the name of the cookie that will be used to route the requests with the annotation `nginx.ingress.kubernetes.io/session-cookie-name`. The default is to create a cookie named `route`.

The annotation `nginx.ingress.kubernetes.io/session-cookie-hash` defines which algorithm will be used to hash the used upstream. Default value is `md5` and possible values are `md5`, `sha1` and `index`.

The `index` option is not hashed, an in-memory index is used instead, it's quicker and the overhead is shorter. Warning: The matching against the upstream servers list is inconsistent. So, at reload, if upstreams servers have changed, index values are not guaranted to correspond to the same server as before! Use with caution and only if you need to!

This feature is implemented by the third party module [nginx-sticky-module-ng](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng). The workflow used to define which upstream server will be used is explained in the [module documentation (PDF)](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/raw/08a395c66e425540982c00482f55034e1fee67b6/docs/sticky.pdf).

### Configuration snippets
Expand Down
83 changes: 35 additions & 48 deletions src/content/advanced/connectivity/ingress/multi-nginx-ic/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ user_questions:
- How do I configure Ingress NGINX Controller for internal traffic?
- How do I override the NodePorts on KVM Ingresses?
- How do I configure Ingress NGINX Controller to allow weak ciphers?
last_review_date: 2023-11-07
last_review_date: 2023-11-23
aliases:
- /guides/multi-nginx/
- /advanced/multi-nginx/
Expand All @@ -23,7 +23,7 @@ owner:
Ingress NGINX Controller handles [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) resources, routing traffic from outside the Kubernetes cluster to services within the cluster.

Starting with [Ingress NGINX Controller v1.8.0](/changes/managed-apps/nginx-ingress-controller-app/v1.8.0/), one can install multiple Ingress NGINX Controllers in a Kubernetes cluster. The optional Ingress NGINX Controller can be [installed as an App on your cluster]({{< relref "/content/getting-started/ingress-controller/index.md" >}}).
[Ingress NGINX Controller v2.2.0](/changes/managed-apps/nginx-ingress-controller-app/v2.2.0/) will start installing a IngressClass with default name `nginx` and controller value `k8s.io/ingress-nginx`.
[Ingress NGINX Controller v2.2.0](/changes/managed-apps/nginx-ingress-controller-app/v2.2.0/) will start installing an IngressClass with default name `nginx` and controller value `k8s.io/ingress-nginx`.

Some use cases for this might be:

Expand All @@ -33,18 +33,18 @@ Some use cases for this might be:

Most Ingress NGINX Controller configuration options have controller-wide defaults. They can also be overriden on a per-Ingress resource level.

In each case below, one installs a second Ingress NGINX Controller with a different global-only configuration and separate IngressClass. Ingress resources managed by this Ingress NGINX Controller installation cannot be customized on a per-Ingress resource level.
In each case below, one installs a second Ingress NGINX Controller with a different global-only configuration and a separate IngressClass. Ingress resources managed by this Ingress NGINX Controller installation can still be customized on a per-Ingress resource level.

Further information on configuring Ingress NGINX Controller can be found on the [Advanced ingress configuration]({{< relref "/advanced/connectivity/ingress/configuration/index.md" >}}) page.

## Quick installation instructions for a second Ingress NGINX Controller

1. Install a second Ingress NGINX Controller app (and subsequent apps) with a different global-only configuration. Ingress resources managed by this Ingress NGINX Controller installation cannot be customized on a per-Ingress resource level.
2. Change the ingressClassName to the appropriate IngressClass name. Make sure the IngressClass name and controller value of each Ingress Controller do not collide with each other.
1. Install a second Ingress NGINX Controller App (and subsequent apps) with a different global-only configuration.
2. Change the `ingressClassName` to the appropriate IngressClass name. Make sure the IngressClass name and controller value of each Ingress Controller do not collide with each other.

## Set the ingressClassName of each Ingress

__Note__ that if you are running multiple Ingress Controllers you need to use the appropriate ingressClassName in your Ingress resources, e.g.
__Note__ that if you are running multiple Ingress Controllers you need to use the appropriate `ingressClassName` in your Ingress resources, e.g.

```yaml
...
Expand All @@ -62,9 +62,9 @@ spec:
...
```

Not specifying the ingressClassName will lead to no Ingress Controller claiming your Ingress. Specifying a value which does not match the class of any existing Ingress Controllers will result in all Ingress Controllers ignoring the ingress.
Not specifying the `ingressClassName` will lead to no Ingress Controller claiming your Ingress. Specifying a value which does not match the class of any existing Ingress Controller will result in all Ingress Controllers ignoring the Ingress.

Additionally, please ensure the Ingress Class of each of your Ingress Controllers do not collide with each other and with the [preinstalled Ingress Controllers in legacy clusters]({{< relref "/platform-overview/cluster-management/releases/index.md#apps" >}}). For the community supported Ingress NGINX Controller this is described in the [official documentation](https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/).
Additionally, please ensure the IngressClass of each of your Ingress Controllers does not collide with each other and with the [preinstalled Ingress Controllers in legacy clusters]({{< relref "/platform-overview/cluster-management/releases/index.md#apps" >}}). For the community supported Ingress NGINX Controller this is described in the [official documentation](https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/).

## Separating public from internal ingress traffic

Expand All @@ -75,45 +75,32 @@ This is how one can achieve it by using multiple Ingress NGINX Controllers:
- Deploy one Ingress NGINX Controller App using default settings, for making selected cluster services accessible via public internet
- Deploy second internal Ingress NGINX Controller App. Use these user configuration overrides:

- on AWS and Azure

```yaml
controller:
ingressClass: nginx-internal
ingressClassResource:
name: nginx-internal
controllerValue: k8s.io/ingress-nginx-internal
service:
public: false
subdomain: "*.ingress-internal"
mcharriere marked this conversation as resolved.
Show resolved Hide resolved
```

- on KVM

```yaml
controller:
ingressClassResource:
name: nginx-internal
controllerValue: k8s.io/ingress-nginx-internal
service:
public: false
subdomain: ingress-internal
nodePorts:
http: 31010
https: 31011
```
```yaml
controller:
ingressClassResource:
name: nginx-internal
controllerValue: k8s.io/ingress-nginx-internal
service:
public: false
subdomain: ingress-internal
# Required for KVM only.
# Do not set on AWS, Azure & others.
nodePorts:
http: 30012
https: 30013
```

Each Ingress NGINX Controller App installation has to have an unique IngressClass. Ingress resources can then declare which Ingress Controller should be handling their route definition by referencing the respective IngressClass in the `ingressClassName` spec field. Default Ingress NGINX Controller IngressClass is `nginx`. In the above example we configure `nginx-internal` as the second Ingress NGINX Controller installation's IngressClass.
Each Ingress NGINX Controller App installation has to have an unique IngressClass. Ingress resources can then declare which Ingress Controller should be handling their route definition by referencing the respective IngressClass in the `ingressClassName` field. The default Ingress NGINX Controller IngressClass is `nginx`. In the above example we configure `nginx-internal` as the second Ingress NGINX Controller installation's IngressClass.

On AWS and Azure, Ingress NGINX Controller `LoadBalancer` service is fronted by the cloud provider's managed load balancer service. By default, Ingress NGINX Controller will have a public load balancer. Changing `controller.service.public` flag to `false` declares that internal load balancer should be created instead.
On AWS and Azure, Ingress NGINX Controller's `LoadBalancer` Service is fronted by the cloud provider's managed load balancer service. By default, Ingress NGINX Controller will have a public load balancer. Changing `controller.service.public` flag to `false` declares that an internal load balancer should be created instead.

Similarly, cloud load balancer created for each Ingress NGINX Controller installation on AWS and Azure has to have unique host name associated with it. Host name suffix is common for all, and equals to the workload cluster's base domain name. Prefix is configurable via `controller.service.subdomain` configuration property and defaults to `ingress`. In example configuration for second internal Ingress NGINX Controller, it is overriden to `ingress-internal`.
Similarly, the cloud load balancer created for each Ingress NGINX Controller installation on AWS and Azure has to have an unique hostname associated with it. The hostname suffix is common for all and equals to the workload cluster's base domain name. The prefix is configurable via the `controller.service.subdomain` configuration property and defaults to `ingress`. In the example configuration for the second internal Ingress NGINX Controller, it is overriden to `ingress-internal`.

For Ingress NGINX Controller running on on-prem (KVM) workload clusters there's no out-of-the-box `LoadBalancer` service type support. Therefore, Ingress NGINX Controller Service type defaults to `NodePort`. For every Ingress NGINX Controller installation, one must assign a set of unique http and https node ports. The default Ingress NGINX Controller http and https node ports are `30010` and `30011`. The example sets `31010` and `31011` as overrides for the internal Ingress NGINX Controller.
For Ingress NGINX Controllers running on on-prem (KVM) workload clusters there's no out-of-the-box `LoadBalancer` service type support. Therefore, Ingress NGINX Controller Service type defaults to `NodePort`. For every Ingress NGINX Controller installation, one must assign a set of unique HTTP and HTTPS node ports. The default Ingress NGINX Controller's HTTP and HTTPS node ports are `30010` and `30011`. The example sets `30012` and `30013` as overrides for the internal Ingress NGINX Controller.

More information on this topic can be found in document [Services of type LoadBalancer]({{< relref "/content/advanced/connectivity/ingress/service-type-loadbalancer/index.md" >}}).
More information on this topic can be found in the document [Services of type LoadBalancer]({{< relref "/content/advanced/connectivity/ingress/service-type-loadbalancer/index.md" >}}).

It is also possible to only install a single Ingress NGINX Controller and to delegate both internal and external traffic to it. Here is a minimal working example on how to achieve this goal.
It is also possible to only install a single Ingress NGINX Controller and to delegate both external and internal traffic to it. Here is a minimal working example on how to achieve this goal.

```yaml
controller:
Expand All @@ -125,33 +112,33 @@ controller:
subdomain: ingress-internal # default value
```

In other words, it is sufficient to set `controller.service.internal.enabled` to `true` to create two services: one for public traffic and one for private one. On cloud providers, the Services we create will be of type `LoadBalancer`; on premise, depending on the platform, they might be either of type `LoadBalancer` or `NodePort`.
In other words, it is sufficient to set `controller.service.internal.enabled` to `true` to create two services: one for public traffic and one for internal one. On cloud providers, the Services we create will be of type `LoadBalancer`; on premise, depending on the platform, they might be either of type `LoadBalancer` or `NodePort`.

## Using weak ciphers for legacy clients

In [Ingress NGINX Controller v1.2.0](https://github.com/giantswarm/ingress-nginx-app/blob/main/CHANGELOG.md#120-2020-01-21), there was a notable security improvement: weak SSL ciphers were removed from the default configuration. Some older clients (like web browsers, http libraries in apps) could no longer establish secure connections with cluster services exposed via new Ingress NGINX Controller. This is because these clients only supported SSL ciphers that got removed.
In [Ingress NGINX Controller v1.2.0](https://github.com/giantswarm/ingress-nginx-app/blob/main/CHANGELOG.md#120-2020-01-21), there was a notable security improvement: weak SSL ciphers were removed from the default configuration. Some older clients (like web browsers or HTTP libraries in apps) could no longer establish secure connections with cluster services exposed via the new Ingress NGINX Controller. This is because these clients only supported SSL ciphers that got removed.

With single Ingress NGINX Controller, one could restore weak SSL ciphers configuration in order to support services with older clients until clients get upgraded. Problem with this approach, since SSL ciphers are global settings, was that changing default SSL ciphers back by restoring weak ciphers would apply to all Ingresses and service behind them, not just the one with old clients.
With a single Ingress NGINX Controller, one could restore weak SSL ciphers configuration in order to support services with older clients until clients get upgraded. Problem with this approach, since SSL ciphers are global settings, was that changing default SSL ciphers back by restoring weak ciphers would apply to all Ingresses and services behind them, not just the one with old clients.

With multiple Ingress NGINX Controllers, one can have separate Ingress NGINX Controller installations with different SSL ciphers configuration. This allows one to limit restoring weak ciphers only to services used by legacy clients.

Here is how this can be achieved:

- Deploy one Ingress NGINX Controller using default settings
- Deploy second internal Ingress NGINX Controller. Use these user configuration overrides:
- Deploy second Ingress NGINX Controller. Use these user configuration overrides:

```yaml
controller:
config:
ssl-ciphers: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ingressClassResource:
name: nginx-weak
controllerValue: k8s.io/ingress-nginx-weak
service:
subdomain: ingress-weak
configmap:
ssl-ciphers: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
```

For the second Ingress NGINX Controller installation, ingress class and host name subdomain are customized for uniqueness. Additionally, default SSL ciphers are overriden to include weak ciphers for legacy clients.
For the second Ingress NGINX Controller installation, IngressClass name and hostname subdomain are customized for uniqueness. Additionally, default SSL ciphers are overriden to include weak ciphers for legacy clients.

## Additional resources

Expand Down
Loading
Loading