KEP-1880 Multiple ServiceCIDR

content/en/docs/reference/command-line-tools-reference/feature-gates.md

@@ -659,7 +659,8 @@ Each feature gate is designed for enabling/disabling a specific feature:
   [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/).
 - `MinimizeIPTablesRestore`: Enables new performance improvement logics
   in the kube-proxy iptables mode.
-- `MultiCIDRServiceAllocator`: Track IP address allocations for Service cluster IPs using IPAddress objects.
+- `MultiCIDRServiceAllocator`: Allow to dynamically configure the cluster Service IP ranges using
+  ServiceCIDR objects and track IP address allocations for Service cluster IPs using IPAddress objects.
 - `NewVolumeManagerReconstruction`: Enables improved discovery of mounted volumes during kubelet
   startup. Since this code has been significantly refactored, we allow to opt-out in case kubelet
   gets stuck at the startup or is not unmounting volumes from terminated Pods. Note that this

content/en/docs/reference/networking/virtual-ips.md

@@ -414,6 +414,44 @@ NAME              PARENTREF
 2001:db8:1:2::a   services/kube-system/kube-dns
 ```
 
+This feature also allow users to dynamically define the available IP ranges for Services using
+ServiceCIDR objects. During bootstrap, a default ServiceCIDR object named `kubernetes` is created
+from the value of the `--service-cluster-ip-range` command line argument to kube-apiserver:
+
+```shell
+kubectl get servicecidrs
+```
+```
+NAME         CIDRS         AGE
+kubernetes   10.96.0.0/28  17m
+```
+
+Users can create or delete new ServiceCIDR objects to manage the available IP ranges for Services:
+
+```shell
+cat <<'EOF' | kubectl apply -f -
+apiVersion: networking.k8s.io/v1alpha1
+kind: ServiceCIDR
+metadata:
+  name: newservicecidr
+spec:
+  cidrs:
+  - 10.96.0.0/24
+EOF
+```
+```
+servicecidr.networking.k8s.io/newcidr1 created
+```
+
+```shell
+kubectl get servicecidrs
+```
+```
+NAME             CIDRS         AGE
+kubernetes       10.96.0.0/28  17m
+newservicecidr   10.96.0.0/24  7m
+```
+
 #### IP address ranges for Service virtual IP addresses {#service-ip-static-sub-range}
 
 {{< feature-state for_k8s_version="v1.26" state="stable" >}}

content/en/docs/tasks/network/extend-service-ip-ranges.md

@@ -0,0 +1,184 @@
+---
+reviewers:
+- thockin
+- dwinship
+min-kubernetes-server-version: v1.29
+title: Extend Service IP Ranges
+content_type: task
+---
+
+<!-- overview -->
+{{< feature-state state="alpha" for_k8s_version="v1.29" >}}
+
+This document shares how to extend the existing Service IP range assigned to a cluster.
+
+
+## {{% heading "prerequisites" %}}
+
+{{< include "task-tutorial-prereqs.md" >}}
+
+{{< version-check >}}
+
+<!-- steps -->
+
+## API
+
+Kubernetes clusters with kube-apiservers that have enabled the `MultiCIDRServiceAllocator`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) and the `networking.k8s.io/v1alpha1` API,
+will create a new ServiceCIDR object that takes the well-known name `kubernetes`, and that uses an IP address range
+based on the value of the `--service-cluster-ip-range` command line argument to kube-apiserver.
+
+```sh
+kubectl get servicecidr
+```
+```
+NAME         CIDRS          AGE
+kubernetes   10.96.0.0/28   17d
+```
+
+The well-known `kubernetes` Service, that exposes the kube-apiserver endpoint to the Pods, calculates
+the first IP address from the default ServiceCIDR range and uses that IP address as its
+cluster IP address.
+
+```sh
+kubectl get service kubernetes
+```
+```
+NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
+kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   17d
+```
+
+The default Service, in this case, uses the ClusterIP 10.96.0.1, that has the corresponding IPAddress object.
+
+```sh
+kubectl get ipaddress 10.96.0.1
+```
+```
+NAME        PARENTREF
+10.96.0.1   services/default/kubernetes
+```
+
+The ServiceCIDRs are protected with {{<glossary_tooltip text="finalizers" term_id="finalizer">}}, to avoid leaving Service ClusterIPs orphans;
+the finalizer is only removed if there is another subnet that contains the existing IPAddresses or
+there are no IPAddresses belonging to the subnet.
+
+## Extend the number of available IPs for Services
+
+There are cases that users will need to increase the number addresses available to Services, previously, increasing the Service range was a disruptive operation that could also cause data loss. With this new feature users only need to add a new ServiceCIDR to increase the number of available addresses.
+
+### Adding a new ServiceCIDR
+
+On a cluster with a 10.96.0.0/28 range for Services, there is only 2^(32-28) - 2 = 14 IP addresses available. The `kubernetes.default` Service is always created; for this example, that leaves you with only 13 possible Services.
+
+```sh
+for i in $(seq 1 13); do kubectl create service clusterip "test-$i" --tcp 80 -o json | jq -r .spec.clusterIP; done
+```
+```
+10.96.0.11
+10.96.0.5
+10.96.0.12
+10.96.0.13
+10.96.0.14
+10.96.0.2
+10.96.0.3
+10.96.0.4
+10.96.0.6
+10.96.0.7
+10.96.0.8
+10.96.0.9
+error: failed to create ClusterIP service: Internal error occurred: failed to allocate a serviceIP: range is full
+```
+
+You can increase the number of IP addresses available for Services, by creating a new ServiceCIDR
+that extends or adds new IP address ranges.
+
+```sh
+cat <EOF | kubectl apply -f -
+apiVersion: networking.k8s.io/v1alpha1
+kind: ServiceCIDR
+metadata:
+  name: newcidr1
+spec:
+  cidrs:
+  - 10.96.0.0/24
+EOF
+```
+```
+servicecidr.networking.k8s.io/newcidr1 created
+```
+
+and this will allow you to create new Services with ClusterIPs that will be picked from this new range.
+
+```sh
+for i in $(seq 13 16); do kubectl create service clusterip "test-$i" --tcp 80 -o json | jq -r .spec.clusterIP; done
+```
+```
+10.96.0.48
+10.96.0.200
+10.96.0.121
+10.96.0.144
+```
+
+### Deleting a ServiceCIDR
+
+You cannot delete a ServiceCIDR if there are IPAddresses that depend on the ServiceCIDR.
+
+```sh
+kubectl delete servicecidr newcidr1
+```
+```
+servicecidr.networking.k8s.io "newcidr1" deleted
+```
+
+Kubernetes uses a finalizer on the ServiceCIDR to track this dependent relationship.
+
+```sh
+kubectl get servicecidr newcidr1 -o yaml
+```
+```
+apiVersion: networking.k8s.io/v1alpha1
+kind: ServiceCIDR
+metadata:
+  creationTimestamp: "2023-10-12T15:11:07Z"
+  deletionGracePeriodSeconds: 0
+  deletionTimestamp: "2023-10-12T15:12:45Z"
+  finalizers:
+  - networking.k8s.io/service-cidr-finalizer
+  name: newcidr1
+  resourceVersion: "1133"
+  uid: 5ffd8afe-c78f-4e60-ae76-cec448a8af40
+spec:
+  cidrs:
+  - 10.96.0.0/24
+status:
+  conditions:
+  - lastTransitionTime: "2023-10-12T15:12:45Z"
+    message: There are still IPAddresses referencing the ServiceCIDR, please remove
+      them or create a new ServiceCIDR
+    reason: OrphanIPAddress
+    status: "False"
+    type: Ready
+```
+
+
+By removing the Services containing the IP addresses that are blocking the deletion of the ServiceCIDR
+
+```sh
+for i in $(seq 13 16); do kubectl delete service "test-$i" ; done
+```
+```
+service "test-13" deleted
+service "test-14" deleted
+service "test-15" deleted
+service "test-16" deleted
+```
+
+the control plane notices the removal. The control plane then removes its finalizer,
+so that the ServiceCIDR that was pending deletion will actually be removed.
+
+```sh
+kubectl get servicecidr newcidr1
+```
+```
+Error from server (NotFound): servicecidrs.networking.k8s.io "newcidr1" not found
+```