Add support for IPv6

[salonichf5]

Proposed changes

Problem: User wants to NGINX Gateway Fabric to support IPv6 and IPv4

Solution: Add a new field ipFamily to NginxProxy API to specify the IP family to use with server and update listen directives in the nginx.conf.

Testing:

• Updates unit tests as needed
• Manual Testing with cafe-examples:

1. Only IPv6

values.yaml file

config
   ipFamily: ipv6

NGINX Config

server {
    listen [::]:80 default_server;

    default_type text/html;
    return 404;
}

server {
    listen [::]:80;

    server_name cafe.example.com;
.
.
.
upstream default_tea_80 {
    random two least_conn;
    zone default_tea_80 512k;

    server [fd00:10:244:1::e]:8080;
}

upstream default_coffee_80 {
    random two least_conn;
    zone default_coffee_80 512k;

    server [fd00:10:244:1::d]:8080;
}

Response from Coffee and Tea services

curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/coffee
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /coffee HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 20:49:51 GMT
< Content-Type: text/plain
< Content-Length: 166
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 20:49:50 GMT
< Cache-Control: no-cache
<
Server address: fd00:10:244:1::d:8080
Server name: coffee-6b8b6d6486-gqg68
Date: 07/Jul/2024:20:49:51 +0000
URI: /coffee
Request ID: 9723f8cc35b422b0a5e385aee4d28ded

curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/tea
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /tea HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 20:50:41 GMT
< Content-Type: text/plain
< Content-Length: 159
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 20:50:40 GMT
< Cache-Control: no-cache
<
Server address: fd00:10:244:1::e:8080
Server name: tea-9d8868bb4-s2sh2
Date: 07/Jul/2024:20:50:41 +0000
URI: /tea
Request ID: fddfb7555f4089c9d4cdce386464bc74

2. Only IPv4

values.yaml file

config
   ipFamily: ipv4

NGINX Config

server {
    listen 80 default_server;

    default_type text/html;
    return 404;
}

server {
    listen 80;
.
.
.

upstream default_tea_80 {
    random two least_conn;
    zone default_tea_80 512k;

    server 10.244.0.6:8080;
}

upstream default_coffee_80 {
    random two least_conn;
    zone default_coffee_80 512k;

    server 10.244.0.7:8080;
}

Response from Coffee and Tea services

curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/tea
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /tea HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 21:08:20 GMT
< Content-Type: text/plain
< Content-Length: 154
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 21:08:19 GMT
< Cache-Control: no-cache
<
Server address: 10.244.0.6:8080
Server name: tea-596697966f-bvgln
Date: 07/Jul/2024:21:08:20 +0000
URI: /tea
Request ID: adfa0f1b4ea2d066f2a1b250325eae0c
* Connection #0 to host cafe.example.com left intact

curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/coffee
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /coffee HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 21:08:48 GMT
< Content-Type: text/plain
< Content-Length: 160
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 21:08:47 GMT
< Cache-Control: no-cache
<
Server address: 10.244.0.7:8080
Server name: coffee-56b44d4c55-885kb
Date: 07/Jul/2024:21:08:48 +0000
URI: /coffee
Request ID: be548614b053439199dfaa449f36b607
* Connection #0 to host cafe.example.com left intact

3. IPv6 and IPv4

values.yaml file (by default it is set to ipFamily: dual)

config
   {}

NGF Pod has both IPs configured when installed with dual IP Family.

kubectl describe pod -n nginx-gateway ngf-nginx-gateway-fabric-7446d69b4-6hsjh
Name:             ngf-nginx-gateway-fabric-7446d69b4-6hsjh
Namespace:        nginx-gateway
Status:           Running
IP:               10.244.1.2
IPs:
  IP:           10.244.1.2
  IP:           fd00:10:244:1::2

Coffee Service created with IPv4

kubectl describe svc coffee
Name:              coffee
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.202.148
IPs:               10.96.202.148

Tea Service created with IPv6 stack

kubectl describe svc tea
Name:              tea
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv6
IP:                fd00:10:96::2e2e

NGINX Config

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    default_type text/html;
    return 404;
}

server {
    listen 80;
    listen [::]:80;
.
.
.
upstream default_coffee_80 {
    random two least_conn;
    zone default_coffee_80 512k;

    server 10.244.1.3:8080;
}

upstream default_tea_80 {
    random two least_conn;
    zone default_tea_80 512k;

    server [fd00:10:244:1::4]:8080;
}

Responses from Tea and Coffee Applications

curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/coffee
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /coffee HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 21:24:30 GMT
< Content-Type: text/plain
< Content-Length: 160
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 21:24:29 GMT
< Cache-Control: no-cache
<
Server address: 10.244.1.3:8080
Server name: coffee-6b8b6d6486-hvxp5
Date: 07/Jul/2024:21:24:30 +0000
URI: /coffee
Request ID: 1833d0a1fd620e91946ea184c7a358e5
* Connection #0 to host cafe.example.com left intact

curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/tea
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /tea HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 21:24:43 GMT
< Content-Type: text/plain
< Content-Length: 159
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 21:24:42 GMT
< Cache-Control: no-cache
<
Server address: fd00:10:244:1::4:8080
Server name: tea-9d8868bb4-82gxc
Date: 07/Jul/2024:21:24:43 +0000
URI: /tea
Request ID: 3ac40253dd7c64e4e92ad65e7c87a322

4. If the cluster supports only IPv6 and a service is created with ipFamily: IPv4, following error shows:

gateway.gateway.networking.k8s.io/gateway created
The Service "coffee" is invalid: spec.ipFamilies[0]: Invalid value: "IPv4": not configured on this cluster

5. If NGF is configured with IPv4 and a Service is configured with IPv6, the HTTPRoute shows an error

  Parents:
    Conditions:
      Last Transition Time:  2024-07-12T04:09:11Z
      Message:               All references are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2024-07-12T04:09:11Z
      Message:               service configured with IPv6 family but NginxProxy is configured with IPv4
      Observed Generation:   1
      Reason:                InvalidIPFamily
      Status:                False
      Type:                  Accepted
    Controller Name:         gateway.nginx.org/nginx-gateway-controller

6. If NGF is configured with IPv6 and a Service is configured with IPv4, the HTTPRoute shows an error

Status:
  Parents:
    Conditions:
      Last Transition Time:  2024-07-12T04:12:03Z
      Message:               All references are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2024-07-12T04:12:03Z
      Message:               service configured with IPv4 family but NginxProxy is configured with IPv6
      Observed Generation:   1
      Reason:                InvalidIPFamily
      Status:                False
      Type:                  Accepted
    Controller Name:         gateway.nginx.org/nginx-gateway-controller

Please focus on (optional): If you any specific areas where you would like reviewers to focus their attention or provide
specific feedback, add them here.

Closes #732

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation //TODO
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

Adds support for IPv6

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.76%. Comparing base (45190d3) to head (e802e82).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2190      +/-   ##
==========================================
+ Coverage   87.62%   87.76%   +0.14%     
==========================================
  Files          96       96              
  Lines        6715     6793      +78     
  Branches       50       50              
==========================================
+ Hits         5884     5962      +78     
  Misses        774      774              
  Partials       57       57              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pleshakov]

Let's make sure we have the cafe example work for three cases when the cluster is:

(1)
IPv6

Ensure you can establish IPv6 connection to NGINX, and you see responses from coffee and tea services.

I tried this case myself and NGINX wasn't configured with upstream servers (they were empty).

(See this area of the code Also

nginx-gateway-fabric/internal/mode/static/state/resolver/resolver.go

Lines 152 to 154 in 8f5b9e6

	if endpointSlice.AddressType != discoveryV1.AddressTypeIPv4 {
	return true
	}

)

Let's make sure it doesn't bind to IPv4 sockets or use IPv4 addresses as upstream servers

(2)
IPv4
Ensure you can establish IPv4 connection to NGINX, and you see responses from coffee and tea services.

If the host machine doesn't support IPv6, NGINX will fail to reload with

2024/07/05 22:46:53 [emerg] 454#454: socket() [::]:80 failed (97: Address family not supported by protocol)

if NGINX binds to IPv6 socket:

listen [::]:80;

Because of that , let's make sure it doesn't bind to IPv6 sockets or use IPv4 addresses as upstream servers

(3)
Dual

• Ensure you can establish IPv6 connection to NGINX, and you see responses from coffee and tea services.
• Ensure you can establish IPv4 connection to NGINX, and you see responses from coffee and tea services.

In this case, make sure one of the services - tea or coffee - has IPv4 and the other IPv6

You can configure IP familiy for the kind cluster -- https://kind.sigs.k8s.io/docs/user/configuration/#networking

See also https://kubernetes.io/docs/tasks/network/validate-dual-stack/#validate-services for controlling whether service shall be ipv4 or ipv6.

[salonichf5]

Let's make sure we have the cafe example work for three cases when the cluster is:

(1) IPv6

Ensure you can establish IPv6 connection to NGINX, and you see responses from coffee and tea services.

I tried this case myself and NGINX wasn't configured with upstream servers (they were empty).

(See this area of the code Also

nginx-gateway-fabric/internal/mode/static/state/resolver/resolver.go

Lines 152 to 154 in 8f5b9e6

	if endpointSlice.AddressType != discoveryV1.AddressTypeIPv4 {
	return true
	}

)
Let's make sure it doesn't bind to IPv4 sockets or use IPv4 addresses as upstream servers

(2) IPv4 Ensure you can establish IPv4 connection to NGINX, and you see responses from coffee and tea services.

If the host machine doesn't support IPv6, NGINX will fail to reload with

2024/07/05 22:46:53 [emerg] 454#454: socket() [::]:80 failed (97: Address family not supported by protocol)

if NGINX binds to IPv6 socket:

listen [::]:80;

Because of that , let's make sure it doesn't bind to IPv6 sockets or use IPv4 addresses as upstream servers

(3) Dual

• Ensure you can establish IPv6 connection to NGINX, and you see responses from coffee and tea services.
• Ensure you can establish IPv4 connection to NGINX, and you see responses from coffee and tea services.

In this case, make sure one of the services - tea or coffee - has IPv4 and the other IPv6

You can configure IP familiy for the kind cluster -- https://kind.sigs.k8s.io/docs/user/configuration/#networking

See also https://kubernetes.io/docs/tasks/network/validate-dual-stack/#validate-services for controlling whether service shall be ipv4 or ipv6.

Have updated description to demonstrate these use cases. Let me know if something doesn't look right