diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 223aa166b0b93..6f5076180d42e 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -659,7 +659,8 @@ Each feature gate is designed for enabling/disabling a specific feature: [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/). - `MinimizeIPTablesRestore`: Enables new performance improvement logics in the kube-proxy iptables mode. -- `MultiCIDRServiceAllocator`: Track IP address allocations for Service cluster IPs using IPAddress objects. +- `MultiCIDRServiceAllocator`: Allow to dynamically configure the cluster Service IP ranges using + ServiceCIDR objects and track IP address allocations for Service cluster IPs using IPAddress objects. - `NewVolumeManagerReconstruction`: Enables improved discovery of mounted volumes during kubelet startup. Since this code has been significantly refactored, we allow to opt-out in case kubelet gets stuck at the startup or is not unmounting volumes from terminated Pods. Note that this diff --git a/content/en/docs/reference/networking/virtual-ips.md b/content/en/docs/reference/networking/virtual-ips.md index 623cb37525caa..8c4a65364e5d4 100644 --- a/content/en/docs/reference/networking/virtual-ips.md +++ b/content/en/docs/reference/networking/virtual-ips.md @@ -414,6 +414,44 @@ NAME PARENTREF 2001:db8:1:2::a services/kube-system/kube-dns ``` +This feature also allow users to dynamically define the available IP ranges for Services using +ServiceCIDR objects. During bootstrap, a default ServiceCIDR object named `kubernetes` is created +from the value of the `--service-cluster-ip-range` command line argument to kube-apiserver: + +```shell +kubectl get servicecidrs +``` +``` +NAME CIDRS AGE +kubernetes 10.96.0.0/28 17m +``` + +Users can create or delete new ServiceCIDR objects to manage the available IP ranges for Services: + +```shell +cat <<'EOF' | kubectl apply -f - +apiVersion: networking.k8s.io/v1alpha1 +kind: ServiceCIDR +metadata: + name: newservicecidr +spec: + cidrs: + - 10.96.0.0/24 +EOF +``` +``` +servicecidr.networking.k8s.io/newcidr1 created +``` + +```shell +kubectl get servicecidrs +``` +``` +NAME CIDRS AGE +kubernetes 10.96.0.0/28 17m +newservicecidr 10.96.0.0/24 7m +``` + #### IP address ranges for Service virtual IP addresses {#service-ip-static-sub-range} {{< feature-state for_k8s_version="v1.26" state="stable" >}} diff --git a/content/en/docs/tasks/network/extend-service-ip-ranges.md b/content/en/docs/tasks/network/extend-service-ip-ranges.md new file mode 100644 index 0000000000000..fdce843c68c41 --- /dev/null +++ b/content/en/docs/tasks/network/extend-service-ip-ranges.md @@ -0,0 +1,184 @@ +--- +reviewers: +- thockin +- dwinship +min-kubernetes-server-version: v1.29 +title: Extend Service IP Ranges +content_type: task +--- + + +{{< feature-state state="alpha" for_k8s_version="v1.29" >}} + +This document shares how to extend the existing Service IP range assigned to a cluster. + + +## {{% heading "prerequisites" %}} + +{{< include "task-tutorial-prereqs.md" >}} + +{{< version-check >}} + + + +## API + +Kubernetes clusters with kube-apiservers that have enabled the `MultiCIDRServiceAllocator` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) and the `networking.k8s.io/v1alpha1` API, +will create a new ServiceCIDR object that takes the well-known name `kubernetes`, and that uses an IP address range +based on the value of the `--service-cluster-ip-range` command line argument to kube-apiserver. + +```sh +kubectl get servicecidr +``` +``` +NAME CIDRS AGE +kubernetes 10.96.0.0/28 17d +``` + +The well-known `kubernetes` Service, that exposes the kube-apiserver endpoint to the Pods, calculates +the first IP address from the default ServiceCIDR range and uses that IP address as its +cluster IP address. + +```sh +kubectl get service kubernetes +``` +``` +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +kubernetes ClusterIP 10.96.0.1 443/TCP 17d +``` + +The default Service, in this case, uses the ClusterIP 10.96.0.1, that has the corresponding IPAddress object. + +```sh +kubectl get ipaddress 10.96.0.1 +``` +``` +NAME PARENTREF +10.96.0.1 services/default/kubernetes +``` + +The ServiceCIDRs are protected with {{}}, to avoid leaving Service ClusterIPs orphans; +the finalizer is only removed if there is another subnet that contains the existing IPAddresses or +there are no IPAddresses belonging to the subnet. + +## Extend the number of available IPs for Services + +There are cases that users will need to increase the number addresses available to Services, previously, increasing the Service range was a disruptive operation that could also cause data loss. With this new feature users only need to add a new ServiceCIDR to increase the number of available addresses. + +### Adding a new ServiceCIDR + +On a cluster with a 10.96.0.0/28 range for Services, there is only 2^(32-28) - 2 = 14 IP addresses available. The `kubernetes.default` Service is always created; for this example, that leaves you with only 13 possible Services. + +```sh +for i in $(seq 1 13); do kubectl create service clusterip "test-$i" --tcp 80 -o json | jq -r .spec.clusterIP; done +``` +``` +10.96.0.11 +10.96.0.5 +10.96.0.12 +10.96.0.13 +10.96.0.14 +10.96.0.2 +10.96.0.3 +10.96.0.4 +10.96.0.6 +10.96.0.7 +10.96.0.8 +10.96.0.9 +error: failed to create ClusterIP service: Internal error occurred: failed to allocate a serviceIP: range is full +``` + +You can increase the number of IP addresses available for Services, by creating a new ServiceCIDR +that extends or adds new IP address ranges. + +```sh +cat