Antrea works on Windows Node

[wenyingd]

Describe what you are trying to solve
Antrea could be run on Windows as worker Nodes in the cluster.

Describe the solution you have in mind
Windows HNS Network is used to support the container networking, and "Transparent" is the chosen driver type.
For each Pod, a HNS Endpoint is created and attached on both the infra container and the workload container. The IP address of the HNS Endpoint is allocated by IPAM. The IPAM is leveraging host-local by default, which is the same as Antrea on Linux.
OVS Extension is enabled on the target HNS Network, and OpenFlow is used to implement container networking connectivity and Network Policy rules. Each HNS Endpoint has a single port added on the OVS bridge, which has the same name as the HNS Endpoint.
Routes for the cross-nodes connectivity are also needed to be setup on Windows host, and the routing configurations are similar to Antrea on Linux.
The physical networking interface which Windows Node uses to join the K8S cluster is adding on the OVS bridge as the uplink interface. And OpenFlow entries are added to support Winows host networking.
Named pipe is used for the communications between local processes, including, 1) antrea-agent v.s. OVSDB, 2) antrea-agent v.s. OF Switch 3) CNI plugin v.s. CNI server.

Describe how your solution impacts user flows
The work flows to enable Antrea on Windows should be:

1. Setup a K8S cluster, and the master Node should be a Linux node. Antrea controller should be schecduled on the master Node
2. Prepare Whindows Server hosts. Install docker engine on the host, and nable Hyper-V feature. Note: in the first phase to support Windows, Hyper-V is required on the Windows host.
3. Install K8S binaries on Windows host
4. Run kubelet on Windows host to join the cluster
5. Run Antrea agent on Windows host

Describe the main design/architecture of your solution
Please refer to the design doc:
https://docs.google.com/document/d/1lSis0XnKz8UcJSkxTgRtDhP2DAwQtcZDjT6ZRySUb48/edit?usp=sharing

Subtasks

• Create HNS Network in antrea-agent initialization phase, [Windows] Implement Windows host configuration #373
• Move host networking interface IP/MAC to OVS bridge, and add the interface as OVS uplink, [Windows] Prepare ovs bridge #564
• Leverage HNS Endpoints to configure Container networking, [Windows] Implement Pod configuration on Windows with HNS Endpoint #372
• Change CNI server default address on Windows, [Windows] Set CNI server default address for Windows platform #502
• Add SNAT support [Windows] Enable Windows Pod to access external addresses #393
• Support named pipe in ovsdb connection, [Windows]Connect to ovsdb-server using named pipe #389
• Support named pipe in OF binding connection, [Windows] Connect to ovs bridge using named pipe #461
• Prepare OVS bridge uplink, [Windows] Prepare ovs bridge #564
• Add peer node route [Windows] Implement Windows host configuration #373
• Add default internal port/interface when creating OVS bridge, [Windows] Prepare ovs bridge #564
• Set fixed datapath-id, [Windows] Prepare ovs bridge #564
• Support DNS config, [Windows] Support set DNS from runtime config in CNI request #642
• Add documentation for Antrea on Windows [Windows] Add documentation for Antrea on Windows #706, [Windows] Add Antrea Windows design doc #751
  - [ ] Register Windows Service for Antrea Agent [Not4Review][Windows] Create Windows Service for Antrea Agent #470
• Fix wrong tunnel source ip caused by kube-proxy
• Support cross compile window binaries in CI, Add windows binaries compiling job to Github action #606
• Provide script to install k8s essentials for worker node. [Windows]Add antrea-agent dockerfile and deployment yaml for windows #695
• Provide script to install OVS. [Windows]Add antrea-agent dockerfile and deployment yaml for windows #695
• Support deploy antrea-agent and kube-proxy by applying yaml file. [Windows]Add antrea-agent dockerfile and deployment yaml for windows #695
• Support windows worker node deployment in CI.
• Add Kubernetes tests for Windows nodes in CI

TODO

• Support windows components livenessProbe and readinessProbe

[antoninbas]

@wenyingd I haven't taken a look at #372 and #373 yet, but what's the testing strategy for this? Do we have a K8s testbed with Windows Nodes and can we enable at least a subset of the tests we run on Linux as part of CI? @edwardbadboy @lzhecheng

[wenyingd]

We need a K8S testbed with Windows Nodes as the worker nodes, and the master Node should be Linux node. Since K8S components(kubelet, kube-proxy), Antrea agent and OVS are working as processes on Windows node, we will provide a script to help deploy and run the processes in later PRs, and we also need a Windows node to run Windows specific and common test cases.

For unit test and integration test, if the new test case is Windows specific, it should be placed in a file with "_windows_test" suffix. For existing cases, if the case is closely releveant with Linux, it should be moved to a "_linux_test" suffix file, otherwise the tests should be passed on Windows node.

For CI test cases, I think we could refer to these e2e tests: https://github.com/kubernetes-sigs/windows-testing

[lzhecheng]

Currently, we don't have any Windows testbed :(

[michmike]

this is awesome. i love this work and can't wait to talk about it in Kubernetes SIG-Windows.
a couple of questions.

1. What versions of windows will you support? Windows Server 2019 (1809) and beyond for example
2. Does NSX-T NCP also use the transparent driver?
3. Will heterogensouc clusters be supported? meaning a mix of windows and linux worker nodes in the same cluster. i assumed it is, just wanted to make sure it is called out
4. why is Hyper-V required on the Windows host? i know we are working with Microsoft to avoid this. is that still a requirement?
5. What are the scalability and performance goals?

[jianjuns]

@michmike thanks for the support.
@wenyingd and @ruicao93 can provide more information, but let me try answering some your questions.

1. @wenyingd and @ruicao93 can provide the exact patch version.
2. Yes. We can use transparent driver only.
3. Yes.
4. We do not have all changes in upstream OVS to remove Hyper-V vSwitch dependency. And I feel it will be long term work to maintain OVS support for Windows without Hyper-V, because even with our private OVS build we meet compatibility issues time to time with Windows patches.
5. We have not done complete scale and performance tests yet.
   For cluster scale, Antrea itself should not be different from that for Linux Nodes (so far we tested 1K Nodes, 100KPods, and 50K NetworkPolicies, but it does not mean Antrea can not scale higher).
   For a single Node scale/performance, as the current implementation uses upstream kube-proxy userspace mode, we do see some K8s Service traffic performance issues. We assume it can be fixed once we implement kube-proxy functionalities with OVS (which will be supported in 1-2 releases).

[ruicao93]

Thanks @jianjuns for the answers.

@michmike, for the qustions:

1. What versions of windows will you support? Windows Server 2019 (1809) and beyond for example

We plan to support Windows Server 2019 (1809) in first release because it's the latest LTS version. The newer versions of windows(1903+) currently are not in test scope. But I think we can verify Antrea on these versions after we complete the CI integration for windows.

Thanks,
Rui

[jayunit100]

@jianjuns "long term work" ... as in months or years ? :) i think getting off Hyper-v is emerging as a high priority for us .

Is HyperV supported on AWS VMs ? If not, we won't be able to run antrea on windows in AWS.

[ruicao93]

@jianjuns "long term work" ... as in months or years ? :) i think getting off Hyper-v is emerging as a high priority for us .

Hi @jayunit100 , I think the "long term work" means we need to wait OVS remove the dependency to hyper-v. Actually we already have private patches which support it but only be used in VMware product. I'm not sure when the patches could be ported to upstream OVS. Maybe Jianjun know more info about it.

Is HyperV supported on AWS VMs ? If not, we won't be able to run antrea on windows in AWS.

I search the problem, but can find the answer from AWS official doc. It seems only baremetal instances on AWS has full hardware virtualization capabilities.
https://forums.aws.amazon.com/thread.jspa?messageID=926556
https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/system-requirements-for-hyper-v-on-windows
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/windows-ami-version-history.html#Virtualization-types
https://aws.amazon.com/marketplace/pp/Amazon-Web-Services-Microsoft-Windows-Server-2019-/B07RJTTGML

[michmike]

i think we can use our good friends at cloudbase and Alin to help us push the fixes upstream in OVS. do we have anyone from antrea team involved in OVS community that can help here as well?

[jianjuns]

We did talk to Alin, Ben, and Anand on these changes. @wenyingd can update the status. Probably let us discuss offline.

[jayunit100]

lets sync about this at the antrea community meeting this wk, and then ill update this issue with details afterwards. Im also going to talk to @yasensim on wednseday

[vicky-liu]

Alin didnt want to merge the private patches into OVS upstream to disable hyper-v because he is working on a new implementation on v2 HCN API. Let me check the progress and update here.

[jayunit100]

thanks , this is actually a blocker for us now so, anything we can do to help please let us know !

[jayunit100]

FWIW, we do have hyper-vless branches we can use now, but we just dont have them in a production ready context. @ruicao93 has details.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment, or this will be closed in 180 days