Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Virtual IP configuration to set different address/port #986

Merged
merged 5 commits into from
Dec 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions api/v1alpha1/common_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,25 @@ type ControlPlaneVirtualIPSpec struct {
// +kubebuilder:validation:Enum=KubeVIP
// +kubebuilder:default=KubeVIP
Provider string `json:"provider,omitempty"`

// Configuration for the chosen control-plane virtual IP provider.
// +kubebuilder:validation:Optional
Configuration *ControlPlaneVirtualIPConfiguration `json:"configuration,omitempty"`
dkoshkin marked this conversation as resolved.
Show resolved Hide resolved
}

type ControlPlaneVirtualIPConfiguration struct {
// The virtual IP on which the API server is serving.
// If left empty, the value from controlPlaneEndpoint.host will be used.
// +kubebuilder:validation:Optional
dkoshkin marked this conversation as resolved.
Show resolved Hide resolved
// +kubebuilder:validation:Format=ipv4
Address string `json:"address,omitempty"`

// The port on which the API server is serving.
// If left empty, the value from controlPlaneEndpoint.port will be used.
// +kubebuilder:validation:Optional
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=65535
Port int32 `json:"port,omitempty"`
}

// LocalObjectReference contains enough information to let you locate the
Expand Down
18 changes: 18 additions & 0 deletions api/v1alpha1/crds/caren.nutanix.com_nutanixclusterconfigs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -598,6 +598,24 @@ spec:
virtualIP:
description: Configuration for the virtual IP provider.
properties:
configuration:
description: Configuration for the chosen control-plane virtual IP provider.
properties:
address:
description: |-
The virtual IP on which the API server is serving.
If left empty, the value from controlPlaneEndpoint.host will be used.
format: ipv4
type: string
port:
description: |-
The port on which the API server is serving.
If left empty, the value from controlPlaneEndpoint.port will be used.
format: int32
maximum: 65535
minimum: 1
type: integer
type: object
provider:
default: KubeVIP
description: Virtual IP provider to deploy.
Expand Down
24 changes: 23 additions & 1 deletion api/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ prismCentralInsecure: {{ .PrismCentralInsecure }}
{{- with .PrismCentralAdditionalTrustBundle }}
prismCentralAdditionalTrustBundle: "{{ . }}"
{{- end }}
{{- with .ControlPlaneEndpointHost }}
ignoredNodeIPs: [ {{ printf "%q" . }} ]
{{- end }}
{{- with .IPsToIgnore }}
ignoredNodeIPs: [ {{ joinQuoted . }} ]
{{- end }}

# The Secret containing the credentials will be created by the handler.
createSecret: false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ data:
- name: vip_arp
value: "true"
- name: port
value: '{{ `{{ .ControlPlaneEndpoint.Port }}` }}'
value: '{{ `{{ .Port }}` }}'
- name: vip_nodename
valueFrom:
fieldRef:
Expand All @@ -46,7 +46,7 @@ data:
- name: vip_retryperiod
value: "2"
- name: address
value: '{{ `{{ .ControlPlaneEndpoint.Host }}` }}'
value: '{{ `{{ .Address }}` }}'
dkoshkin marked this conversation as resolved.
Show resolved Hide resolved
- name: prometheus_server
image: ghcr.io/kube-vip/kube-vip:v0.8.3
imagePullPolicy: IfNotPresent
Expand Down
89 changes: 88 additions & 1 deletion docs/content/customization/nutanix/control-plane-endpoint.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title = "Control Plane Endpoint"
+++

Configure Control Plane Endpoint. Defines the host IP and port of the CAPX Kubernetes cluster.
Configure Control Plane Endpoint. Defines the host IP and port of the Nutanix Kubernetes cluster.
jimmidyson marked this conversation as resolved.
Show resolved Hide resolved

## Examples

Expand Down Expand Up @@ -51,6 +51,93 @@ spec:
name: kube-vip
namespace: kube-system
spec:
containers:
- name: kube-vip
args:
- manager
env:
- name: port
value: '6443'
- name: address
value: 'x.x.x.x'
...
owner: root:root
path: /etc/kubernetes/manifests/kube-vip.yaml
permissions: "0600"
postKubeadmCommands:
# Only added for clusters version >=v1.29.0
- |-
if [ -f /run/kubeadm/kubeadm.yaml ]; then
sed -i 's#path: /etc/kubernetes/super-admin.conf#path: ...
fi
preKubeadmCommands:
# Only added for clusters version >=v1.29.0
- |-
if [ -f /run/kubeadm/kubeadm.yaml ]; then
sed -i 's#path: /etc/kubernetes/admin.conf#path: ...
fi
```

### Set Control Plane Endpoint and a Different Virtual IP

It is also possible to set a separate virtual IP to be used by kube-vip from the control plane endpoint.
This is useful in VPC setups or other instances
when you have an external floating IP already associated with the virtual IP.

```yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
nutanix:
controlPlaneEndpoint:
host: x.x.x.x
port: 6443
virtualIP:
configuration:
address: y.y.y.y
```

Applying this configuration will result in the following value being set:

- `NutanixCluster`:

```yaml
spec:
template:
spec:
controlPlaneEndpoint:
host: x.x.x.x
port: 6443
```

- `KubeadmControlPlaneTemplate`

```yaml
spec:
kubeadmConfigSpec:
files:
- content: |
apiVersion: v1
kind: Pod
metadata:
name: kube-vip
namespace: kube-system
spec:
containers:
- name: kube-vip
args:
- manager
env:
- name: port
value: '6443'
- name: address
value: 'y.y.y.y'
...
owner: root:root
path: /etc/kubernetes/manifests/kube-vip.yaml
Expand Down
4 changes: 2 additions & 2 deletions hack/addons/update-kube-vip-manifests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ docker container run --rm ghcr.io/kube-vip/kube-vip:"${KUBE_VIP_VERSION}" \
gojq --yaml-input --yaml-output \
'del(.metadata.creationTimestamp, .status) |
.spec.containers[].imagePullPolicy |= "IfNotPresent" |
(.spec.containers[0].env[] | select(.name == "port").value) |= "{{ `{{ .ControlPlaneEndpoint.Port }}` }}" |
(.spec.containers[0].env[] | select(.name == "address").value) |= "{{ `{{ .ControlPlaneEndpoint.Host }}` }}"
(.spec.containers[0].env[] | select(.name == "port").value) |= "{{ `{{ .Port }}` }}" |
(.spec.containers[0].env[] | select(.name == "address").value) |= "{{ `{{ .Address }}` }}"
' >"${ASSETS_DIR}/${FILE_NAME}"

kubectl create configmap "{{ .Values.hooks.virtualIP.kubeVip.defaultTemplateConfigMap.name }}" --dry-run=client --output yaml \
Expand Down
29 changes: 26 additions & 3 deletions pkg/handlers/generic/lifecycle/ccm/nutanix/handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"context"
"errors"
"fmt"
"strings"
"text/template"

"github.com/go-logr/logr"
Expand Down Expand Up @@ -140,7 +141,15 @@ func templateValuesFunc(
nutanixConfig *v1alpha1.NutanixSpec,
) func(*clusterv1.Cluster, string) (string, error) {
return func(_ *clusterv1.Cluster, valuesTemplate string) (string, error) {
helmValuesTemplate, err := template.New("").Parse(valuesTemplate)
joinQuoted := template.FuncMap{
"joinQuoted": func(items []string) string {
for i, item := range items {
items[i] = fmt.Sprintf("%q", item)
}
return strings.Join(items, ", ")
},
}
helmValuesTemplate, err := template.New("").Funcs(joinQuoted).Parse(valuesTemplate)
if err != nil {
return "", fmt.Errorf("failed to parse Helm values template: %w", err)
}
Expand All @@ -150,7 +159,7 @@ func templateValuesFunc(
PrismCentralPort uint16
PrismCentralInsecure bool
PrismCentralAdditionalTrustBundle string
ControlPlaneEndpointHost string
IPsToIgnore []string
}

address, port, err := nutanixConfig.PrismCentralEndpoint.ParseURL()
Expand All @@ -162,7 +171,7 @@ func templateValuesFunc(
PrismCentralPort: port,
PrismCentralInsecure: nutanixConfig.PrismCentralEndpoint.Insecure,
PrismCentralAdditionalTrustBundle: nutanixConfig.PrismCentralEndpoint.AdditionalTrustBundle,
ControlPlaneEndpointHost: nutanixConfig.ControlPlaneEndpoint.Host,
IPsToIgnore: ipsToIgnore(nutanixConfig),
}

var b bytes.Buffer
Expand All @@ -174,3 +183,17 @@ func templateValuesFunc(
return b.String(), nil
}
}

func ipsToIgnore(nutanixConfig *v1alpha1.NutanixSpec) []string {
toIgnore := []string{nutanixConfig.ControlPlaneEndpoint.Host}
// Also ignore the virtual IP if it is set.
dkoshkin marked this conversation as resolved.
Show resolved Hide resolved
if nutanixConfig.ControlPlaneEndpoint.VirtualIPSpec != nil &&
nutanixConfig.ControlPlaneEndpoint.VirtualIPSpec.Configuration != nil &&
nutanixConfig.ControlPlaneEndpoint.VirtualIPSpec.Configuration.Address != "" {
toIgnore = append(
toIgnore,
nutanixConfig.ControlPlaneEndpoint.VirtualIPSpec.Configuration.Address,
)
}
return toIgnore
}
45 changes: 45 additions & 0 deletions pkg/handlers/generic/lifecycle/ccm/nutanix/handler_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,16 @@ prismCentralPort: 9440
prismCentralInsecure: true
ignoredNodeIPs: [ "1.2.3.4" ]

# The Secret containing the credentials will be created by the handler.
createSecret: false
secretName: nutanix-ccm-credentials
`

expectedWithVirtualIPSet = `prismCentralEndPoint: prism-central.nutanix.com
prismCentralPort: 9440
prismCentralInsecure: true
ignoredNodeIPs: [ "1.2.3.4", "5.6.7.8" ]

# The Secret containing the credentials will be created by the handler.
createSecret: false
secretName: nutanix-ccm-credentials
Expand Down Expand Up @@ -127,6 +137,41 @@ func Test_templateValues(t *testing.T) {
in: valuesTemplate,
expected: expectedWithoutAdditionalTrustBundle,
},
{
name: "With VirtualIP Set",
clusterConfig: &apivariables.ClusterConfigSpec{
Addons: &apivariables.Addons{
GenericAddons: v1alpha1.GenericAddons{
CCM: &v1alpha1.CCM{
Credentials: &v1alpha1.CCMCredentials{
SecretRef: v1alpha1.LocalObjectReference{
Name: "creds",
},
},
},
},
},
Nutanix: &v1alpha1.NutanixSpec{
PrismCentralEndpoint: v1alpha1.NutanixPrismCentralEndpointSpec{
URL: fmt.Sprintf(
"https://prism-central.nutanix.com:%d",
v1alpha1.DefaultPrismCentralPort,
),
Insecure: true,
},
ControlPlaneEndpoint: v1alpha1.ControlPlaneEndpointSpec{
Host: "1.2.3.4",
VirtualIPSpec: &v1alpha1.ControlPlaneVirtualIPSpec{
Configuration: &v1alpha1.ControlPlaneVirtualIPConfiguration{
Address: "5.6.7.8",
},
},
},
},
},
in: valuesTemplate,
expected: expectedWithVirtualIPSet,
},
}
for idx := range tests {
tt := tests[idx]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,7 @@ spec:
- name: vip_arp
value: "true"
- name: address
value: "{{ .ControlPlaneEndpoint.Host }}"
value: "{{ .Address }}"
- name: port
value: "{{ .ControlPlaneEndpoint.Port }}"
value: "{{ .Port }}"
`
Loading
Loading