Skip to content

Commit

Permalink
Update docs for windows container resources
Browse files Browse the repository at this point in the history
  • Loading branch information
@feiskyer
feiskyer committed Mar 8, 2018
1 parent bb8c59a commit 3bfa28c
Showing 1 changed file with 71 additions and 23 deletions.
94 changes: 71 additions & 23 deletions docs/getting-started-guides/windows/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,21 @@ The Kubernetes control plane (API Server, Scheduler, Controller Manager, etc) co
{: .note}

## Build
We recommend using the release binaries that can be found at [https://github.com/kubernetes/kubernetes/releases](https://github.com/kubernetes/kubernetes/releases). Look for the Node Binaries section by visiting the binary downloads link.
We recommend using the release binaries that can be found at [https://github.com/kubernetes/kubernetes/releases](https://github.com/kubernetes/kubernetes/releases). Look for the Node Binaries section by visiting the binary downloads link.

If you wish to build the code yourself, please follow the next instructions:

1. Install the pre-requisites on a Linux host:

```
sudo apt-get install curl git build-essential docker.io conntrack
```
```
2. Run the following commands to build kubelet and kube-proxy:
```bash
K8SREPO="github.com/kubernetes/kubernetes"
go get -d $K8SREPO
# Note: the above command may spit out a message about
# Note: the above command may spit out a message about
# "no Go files in...", but it can be safely ignored!
cd $GOPATH/src/k8s.io/kubernetes
Expand All @@ -42,7 +42,7 @@ If you wish to build the code yourself, please follow the next instructions:
KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kube-proxy
# You will find the output binaries under the folder _output/local/bin/windows/
```
```
More detailed build instructions will be maintained and kept up to date [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/compiling-kubernetes-binaries.md).
Expand All @@ -56,7 +56,7 @@ In Kubernetes version 1.9 or later, Windows Server Containers for Kubernetes are
## Networking
There are several supported network configurations with Kubernetes v1.9 on Windows, including both Layer-3 routed and overlay topologies using third-party network plugins.
1. [Upstream L3 Routing](#upstream-l3-routing-topology) - IP routes configured in upstream ToR
2. [Host-Gateway](#host-gateway-topology) - IP routes configured on each host
3. [Open vSwitch (OVS) & Open Virtual Network (OVN) with Overlay](#using-ovn-with-ovs) - overlay networks (supports STT and Geneve tunneling types)
Expand All @@ -72,7 +72,7 @@ An additional two CNI plugins [win-l2bridge (host-gateway) and win-overlay (vxla
The above networking approaches are already supported on Linux using a bridge interface, which essentially creates a private network local to the node. Similar to the Windows side, routes to all other pod CIDRs must be created in order to send packets via the "public" NIC.
### Windows
Windows supports the CNI network model and uses plugins to interface with the Windows Host Networking Service (HNS) to configure host networking and policy. At the time of this writing, the only publicly available CNI plugin from Microsoft is built from a private repo and available here [wincni.exe](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/cni/wincni.exe). It uses an l2bridge network created through the Windows Host Networking Service (HNS) by an administrator using HNS PowerShell commands on each node as documented in the [Windows Host Setup](#windows-host-setup) section below. Source code for the future CNI plugins will be made available publicly.
Windows supports the CNI network model and uses plugins to interface with the Windows Host Networking Service (HNS) to configure host networking and policy. At the time of this writing, the only publicly available CNI plugin from Microsoft is built from a private repo and available here [wincni.exe](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/cni/wincni.exe). It uses an l2bridge network created through the Windows Host Networking Service (HNS) by an administrator using HNS PowerShell commands on each node as documented in the [Windows Host Setup](#windows-host-setup) section below. Source code for the future CNI plugins will be made available publicly.
#### Upstream L3 Routing Topology
In this topology, networking is achieved using L3 routing with static IP routes configured in an upstream Top of Rack (ToR) switch/router. Each cluster node is connected to the management network with a host IP. Additionally, each node uses a local 'l2bridge' network with a pod CIDR assigned. All pods on a given worker node will be connected to the pod CIDR subnet ('l2bridge' network). In order to enable network communication between pods running on different nodes, the upstream router has static routes configured with pod CIDR prefix => Host IP.
Expand All @@ -92,7 +92,7 @@ The following diagram gives a general overview of the architecture and interacti
(The above image is from [https://github.com/openvswitch/ovn-kubernetes#overlay-mode-architecture-diagram](https://github.com/openvswitch/ovn-kubernetes#overlay-mode-architecture-diagram))
Due to its architecture, OVN has a central component which stores your networking intent in a database. Other components i.e. kube-apiserver, kube-controller-manager, kube-scheduler etc. can be deployed on that central node as well.
Due to its architecture, OVN has a central component which stores your networking intent in a database. Other components i.e. kube-apiserver, kube-controller-manager, kube-scheduler etc. can be deployed on that central node as well.
## Setting up Windows Server Containers on Kubernetes
To run Windows Server Containers on Kubernetes, you'll need to set up both your host machines and the Kubernetes node components for Windows. Depending on your network topology, routes may need to be set up for pod communication on different nodes.
Expand All @@ -103,7 +103,7 @@ To run Windows Server Containers on Kubernetes, you'll need to set up both your
##### Linux Host Setup
1. Linux hosts should be setup according to their respective distro documentation and the requirements of the Kubernetes version you will be using.
1. Linux hosts should be setup according to their respective distro documentation and the requirements of the Kubernetes version you will be using.
2. Configure Linux Master node using steps [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/creating-a-linux-master.md)
3. [Optional] CNI network plugin installed.
Expand All @@ -119,7 +119,7 @@ To run Windows Server Containers on Kubernetes, you'll need to set up both your
More detailed instructions can be found [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows.md).
**Windows CNI Config Example**
**Windows CNI Config Example**
Today, Windows CNI plugin is based on wincni.exe code with the following example, configuration file.
Note: this file assumes that a user previous created 'l2bridge' host networks on each Windows node using `<Verb>-HNSNetwork` cmdlets as shown in the `start-kubelet.ps1` and `start-kubeproxy.ps1` scripts linked above
Expand Down Expand Up @@ -287,7 +287,7 @@ Because your cluster has both Linux and Windows nodes, you must explicitly set t
```
## Support for kubeadm join

If your cluster has been created by [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/),
If your cluster has been created by [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/),
and your networking is setup correctly using one of the methods listed above (networking is setup outside of kubeadm), you can use kubeadm to add a Windows node to your cluster. At a high level, you first have to initialize the master with kubeadm (Linux), then set up the CNI based networking (outside of kubeadm), and finally start joining Windows or Linux worker nodes to the cluster. For additional documentation and reference material, visit the kubeadm link above.

The kubeadm binary can be found at [Kubernetes Releases](https://github.com/kubernetes/kubernetes/releases), inside the node binaries archive. Adding a Windows node is not any different than adding a Linux node:
Expand All @@ -313,9 +313,9 @@ Secrets and ConfigMaps can be utilized in Windows Server Containers, but must be
data:
username: YWRtaW4=
password: MWYyZDFlMmU2N2Rm

---

apiVersion: v1
kind: Pod
metadata:
Expand All @@ -338,7 +338,7 @@ Secrets and ConfigMaps can be utilized in Windows Server Containers, but must be
nodeSelector:
beta.kubernetes.io/os: windows
```
Windows pod with configMap values mapped to environment variables
```yaml
Expand Down Expand Up @@ -374,14 +374,14 @@ spec:
nodeSelector:
beta.kubernetes.io/os: windows
```
### Volumes
Some supported Volume Mounts are local, emptyDir, hostPath. One thing to remember is that paths must either be escaped, or use forward slashes, for example `mountPath: "C:\\etc\\foo"` or `mountPath: "C:/etc/foo"`.

Persistent Volume Claims are supported for supported volume types.

**Examples:**

Windows pod with a hostPath volume
```yaml
apiVersion: v1
Expand All @@ -403,9 +403,9 @@ Persistent Volume Claims are supported for supported volume types.
hostPath:
path: "C:\\etc\\foo"
```

Windows pod with multiple emptyDir volumes

```yaml
apiVersion: v1
kind: Pod
Expand All @@ -428,19 +428,67 @@ Persistent Volume Claims are supported for supported volume types.
nodeSelector:
beta.kubernetes.io/os: windows
```

### Metrics

Windows Stats use a hybrid model: pod and container level stats come from CRI (via dockershim), while node level stats come from the "winstats" package that exports cadvisor like datastructures using windows specific perf counters from the node.

### Container Resources

Container resources (CPU and memory) could be set now for windows containers in v1.10.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: iis
spec:
replicas: 3
template:
metadata:
labels:
app: iis
spec:
containers:
- name: iis
image: microsoft/iis
resources:
limits:
memory: "128Mi"
cpu: 2
ports:
- containerPort: 80
```

### Hyper-V Containers

Hyper-V containers are supported as experimental in v1.10. To create a Hyper-V container, kubelet should be started with feature gates `HyperVContainer=true` and Pod should include annotation `experimental.windows.kubernetes.io/isolation-type=hyperv`.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: iis
spec:
replicas: 3
template:
metadata:
labels:
app: iis
annotations:
experimental.windows.kubernetes.io/isolation-type: hyperv
spec:
containers:
- name: iis
image: microsoft/iis
ports:
- containerPort: 80
```

## Known Limitations for Windows Server Containers with v1.9
Some of these limitations will be addressed by the community in future releases of Kubernetes
- Shared network namespace (compartment) with multiple Windows Server containers (shared kernel) per pod is only supported on Windows Server 1709 or later
- Using Secrets and ConfigMaps as volume mounts is not supported
- Using Secrets and ConfigMaps as volume mounts is not supported
- The StatefulSet functionality for stateful applications is not supported
- Horizontal Pod Autoscaling for Windows Server Container pods has not been verified to work end-to-end
- Hyper-V Containers are not supported
- Some .Net Core applications expect environment variables with a colon (`:`) in the name. Kubernetes currently does not allow this. Replace colon (`:`) with double underscore (`__`) as documented [here](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?tabs=basicconfiguration#configuration-by-environment).


> As of this writing, the Kube-proxy binary requires a pending Kubernetes [pull request](https://github.com/kubernetes/kubernetes/pull/56529) to work properly. You may need to [build](#build) the binaries manually to work around this.

0 comments on commit 3bfa28c

Please sign in to comment.