[meta-issue] Windows port of K3s agent role

[seanyen]

This is a meta ticket to include the work to bring up K3s agent role for Windows.

Mainly this effort is to contribute the Windows porting effort done in AKS Edge Essentials.

These are the associated pull request or work items:

• [K3s][Windows Port] Build script, multi-call binary, and Flannel #7259
• TBD

[mdrahman-suse]

Validated on master branch with commit b8dc955

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-1031-aws x86_64)
Windows 2022 

Cluster Configuration:

1 Linux server, 1 Linux agent and 1 Windows agent

Config.yaml:

write-kubeconfig-mode: 644
cluster-init: true
node-external-ip: <public-ip>
disable-network-policy: true
token: summerheat

Additional files

pod_client.yaml (clusterIP service)

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: client
  name: client-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
      - image: ranchertest/mytestcontainer
        imagePullPolicy: Always
        name: client-curl
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - client
            topologyKey: kubernetes.io/hostname
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
---
apiVersion: v1
kind: Service
metadata:
  name: client-curl
  labels:
    app: client
    service: client-curl
spec:
  type: ClusterIP
  selector:
    app: client
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 80

win-app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: windows-app-deployment
spec:
  selector:
    matchLabels:
      app: windows-app
  replicas: 2
  template:
    metadata:
      labels:
        app: windows-app
    spec:
      containers:
        - name: windows-app
          image: mbuilsuse/pstools:v0.2.0
          ports:
            - containerPort: 3000
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - windows
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: windows-app-svc
  name: windows-app-svc
  namespace: default
spec:
  type: NodePort
  ports:
    - port: 3000
      nodePort: 30096
      name: http
  selector:
    app: windows-app

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s on server and agent nodes
3. Build k3s.exe as suggested here: [K3s][Windows Port] Build script, multi-call binary, and Flannel #7259 (comment)
4. Copy k3s.exe on Windows node
5. SSH in to Windows node, activate powershell and run the commands in Windows node

New-NetFirewallRule -Name k3s -DisplayName 'k3s' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250
New-NetFirewallRule -Name k3sOverlayTraffic4789UDP -DisplayName "K3s Overlay Traffic 4789 UDP" -Enabled True -Direction Inbound -Protocol UDP -Action Allow -LocalPort 4789

6. Setup config

Set-Content -Path c:/Users/Administrator/k3s-config.yaml -Value @"
token: '<K10-token-from-server>'
server: 'https://<server-public-ip>:6443'
node-external-ip: '<public-ip>'
kubelet-arg=cgroups-per-qos: 'false'
kubelet-arg=enforce-node-allocatable: ''
kubelet-arg=resolv-conf: ''
pause-image: 'mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2022-amd64'
"@

7. Run k3s.exe using command .\k3s.exe agent --config k3s-config.yaml
8. Ensure Windows node joins the cluster
9. Apply the deployments and ensure they are up and running

Validations

$ k3s -v
k3s version v1.28.2+k3s-b8dc9553 (b8dc9553)
go version go1.20.8

$ kubectl get nodes,pods -A -o wide
NAME                    STATUS   ROLES                       AGE   VERSION                INTERNAL-IP     EXTERNAL-IP      OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
node/agent1             Ready    <none>                      21m   v1.28.2+k3s-b8dc9553   xxx.xx.xx.237   x.xxx.xx.56      Ubuntu 22.04.2 LTS               5.15.0-1031-aws   containerd://1.7.7-k3s1
node/server1            Ready    control-plane,etcd,master   31m   v1.28.2+k3s-b8dc9553   xxx.xx.xx.39    xx.xxx.xx.228    Ubuntu 22.04.2 LTS               5.15.0-1031-aws   containerd://1.7.7-k3s1
node/ip-ac1f2610        Ready    <none>                      10m   v1.28.2+k3s-b8dc9553   xxx.xx.x.85     xx.xxx.xxx.138   Windows Server 2022 Datacenter   10.0.20348.350    containerd://1.7.7-k3s1

NAMESPACE     NAME                                          READY   STATUS            RESTARTS   AGE     IP          NODE               NOMINATED NODE   READINESS GATES
default       pod/client-deployment-54fd46977d-9nf2v        1/1     Running           0          3m10s   xx.xx.0.9   server1            <none>           <none>
default       pod/client-deployment-54fd46977d-c4mzv        1/1     Running           0          3m10s   xx.xx.1.3   agent1             <none>           <none>
default       pod/windows-app-deployment-6964ff4fb8-hkrf2   1/1     Running           0          3m10s   xx.xx.2.3   ip-ac1f2610        <none>           <none>
default       pod/windows-app-deployment-6964ff4fb8-v8ll6   1/1     Running           0          3m10s   xx.xx.2.4   ip-ac1f2610        <none>           <none>
kube-system   pod/coredns-6799fbcd5-mdb45                   1/1     Running           0          31m     xx.xx.0.5   server1            <none>           <none>
kube-system   pod/helm-install-traefik-8hd6z                0/1     Completed         1          31m     xx.xx.0.6   server1            <none>           <none>
kube-system   pod/helm-install-traefik-crd-9cz9r            0/1     Completed         0          31m     xx.xx.0.3   server1            <none>           <none>
kube-system   pod/local-path-provisioner-84db5d44d9-vzfzp   1/1     Running           0          31m     xx.xx.0.4   server1            <none>           <none>
kube-system   pod/metrics-server-67c658944b-c8tnv           1/1     Running           0          31m     xx.xx.0.2   server1            <none>           <none>
kube-system   pod/svclb-traefik-72e62b7a-dlbz7              0/2     SysctlForbidden   0          60s     <none>      ip-ac1f2610        <none>           <none>
kube-system   pod/svclb-traefik-72e62b7a-jdgwg              2/2     Running           0          30m     xx.xx.0.7   server1            <none>           <none>
kube-system   pod/svclb-traefik-72e62b7a-qbtds              2/2     Running           0          21m     xx.xx.1.2   agent1             <none>           <none>
kube-system   pod/traefik-7bf7d7576d-6vttx                  1/1     Running           0          30m     xx.xx.0.8   server1            <none>           <none>

Additional Observations

• svclb pod in SysctlForbidden status: [K3s][Windows Port] Build script, multi-call binary, and Flannel #7259 (comment)
• With Windows 2019, observed the error below when tried to deploy windows workload. (Due to wrong pause image)
  The error was not observer on Windows 2022

$ kubectl describe -n default pod/windows-app-deployment-6964ff4fb8-574ck
...
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               4m41s                  default-scheduler  Successfully assigned default/windows-app-deployment-6964ff4fb8-574ck to ip-ac1f0e4f
  Warning  FailedCreatePodSandBox  4m26s (x2 over 4m40s)  kubelet            Failed to create pod sandbox: rpc error: code = NotFound desc = failed to get sandbox image "mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64": failed to pull image "mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64": failed to pull and unpack image "mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64": failed to resolve reference "mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64": mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2019-amd64: not found