From 57a99a86d24d7f2009280d563a1cd77994925fb8 Mon Sep 17 00:00:00 2001 From: Jitendra Bhurat Date: Tue, 22 Nov 2016 15:51:47 -0500 Subject: [PATCH 1/5] Adding Getting Started Guide for Windows Support --- docs/getting-started-guides/windows/README.md | 131 ++++++++++++++++++ .../windows/windows-setup.png | Bin 0 -> 51061 bytes 2 files changed, 131 insertions(+) create mode 100644 docs/getting-started-guides/windows/README.md create mode 100644 docs/getting-started-guides/windows/windows-setup.png diff --git a/docs/getting-started-guides/windows/README.md b/docs/getting-started-guides/windows/README.md new file mode 100644 index 0000000000000..22943f1a32a27 --- /dev/null +++ b/docs/getting-started-guides/windows/README.md @@ -0,0 +1,131 @@ +# Running Windows Server Containers using Kubernetes +Microsoft, in collaboration with Docker, is introducing a new feature called Windows Server Containers with Windows Server 2016, which enables Docker containers to run on Windows. We have enhanced Kubernetes to support Windows Server Containers (and Windows Server 2016 as a container host node). + +Windows Server and .NET still account for a significant portion of the workloads in the enterprise, and with this work, customers will be able to run Windows-based and .NET-based applications inside containers on Kubernetes, making Kubernetes the first fully functional cross-platform cluster manager. Services that span Windows-based and Linux-based containers are now a reality. + +Windows Server Containers will be supported on Kubernetes as an alpha feature. Kubernetes control plane (API Server, Scheduler, Controller Manager etc) continue to run on Linux, while the kubelet and kube-proxy can be run on Windows Server. + +## Prerequisites +With the alpha release, we support Windows Server Containers for Kubernetes using the following: + +1. Kubernetes control plane running on existing Linux infrastructure (version 1.5 or later) +2. Windows Server 2016 (RTM version 10.0.14393 or later) +3. Docker Version 1.12.2-cs2-ws-beta or later + +## Networking +Network is achieved using L3 routing. Because third-party networking plugins (e.g. flannel, calico, etc) don’t natively work on Windows Server, we relied on existing technology that is built into the Windows and Linux operating systems. In this L3 networking approach, we chose a /16 subnet for the cluster nodes, and we assign a /24 subnet to each worker node. All pods on a given worker node will be connected to the /24 subnet. This allows pods on the same node to communicate with each other. In order to enable networking between pods running on different nodes, routing features that are built into Windows Server 2016 and Linux are used. + +### Linux +The above networking approach is already supported on Linux using a bridge interface, which essentially creates a private network local to the node. Similar to the Windows side, routes to all other pod CIDRs must be created in order to send packets via the “public” NIC. + +### Windows +To support networking requirements, we will need the following configuration on each Kubernetes Windows Server node: + +1. Two NICs (virtual networking adapters) are required on each Windows Server node - The two Windows container networking modes we use (transparent and L2 bridge) use an external Hyper-V virtual switch. This means that one of the NICs is entirely allocated to the bridge, creating the need for the second NIC. +2. Transparent container network created - This is a manual configuration step and is shown in **_Route Setup_** section below +3. RRAS (Routing) Windows feature enabled - Allows routing between NICs on the box, and also “captures” packets that have the destination IP of a POD running on the node. To enable, open “Server Manager”. Click on “Roles”, “Add Roles”. Click “Next”. Select “Network Policy and Access Services”. Click on “Routing and Remote Access Service” and the underlying checkboxes +4. Routes defined pointing to the other pod CIDRs via the “public” NIC - These routes are added to the built-in routing table as shown in **_Route Setup_** section below + +The following diagram illustrates the Windows Server networking setup for Kubernetes Setup +![Windows Setup](windows-setup.png) + +## Setup +### Host Setup +#### Windows + +1. Windows Server container host running Windows Server 2016 and Docker v1.12. Follow the setup instructions outlined by this blog post: https://msdn.microsoft.com/en-us/virtualization/windowscontainers/quick_start/quick_start_windows_server +2. DNS support for Windows recently got merged to docker master and is currently not supported in a stable docker release. To use DNS build docker from master or download the binary from [Docker master](https://master.dockerproject.org/) +3. Pull the `apprenda/pause` image from `https://hub.docker.com/r/apprenda/pause` +4. RRAS (Routing) Windows feature enabled + +#### Linux + +Linux hosts should be setup according to their respective distro documentation and the requirements of the Kubernetes version you will be using. Linux hosts also require to have CNI Setup. + +### Component Setup +In order to build the work node components, the *kubelet* and *kube-proxy* for Windows, the following needs to be installed on the host +* Git, Go 1.7.1+ +* make (if using Linux or MacOS) +* Important notes and other dependencies are listed [here](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md#building-kubernetes-on-a-local-osshell-environment) + +#### kubelet + +In order to build the *kubelet*, run: + +1. `cd $GOPATH/src/k8s.io/kubernetes` +2. Build *kubelet* + 1. Linux/MacOS: `KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kubelet` + 2. Windows: `go build cmd/kubelet/kubelet.go` + +#### kube-proxy + +In order to build *kube-proxy*, run: + +1. `cd $GOPATH/src/k8s.io/kubernetes` +2. Build *kube-proxy* + 1. Linux/MacOS: `KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kube-proxy` + 2. Windows: `go build cmd/kube-proxy/proxy.go` + +### Route Setup + +The below example setup assumes one Linux and two Windows Server 2016 nodes and a cluster CIDR 192.168.0.0/16 + +| Hostname | Routable IP address | Pod CIDR | +| --- | --- | --- | +| Lin01 | `` | 192.168.0.0/24 | +| Win01 | `` | 192.168.1.0/24 | +| Win02 | `` | 192.168.2.0/24 | + +**Lin01** +``` +ip route add 192.168.1.0/24 via +ip route add 192.168.2.0/24 via +``` + +**Win01** +``` +docker network create -d transparent --gateway 192.168.1.1 --subnet 192.168.1.0/24 +# A bridge is created with Adapter name "vEthernet (HNSTransparent)". Set its IP address to transparent network gateway +netsh interface ipv4 set address "vEthernet (HNSTransparent)" addr=192.168.1.1 +route add 192.168.0.0 mask 255.255.255.0 192.168.0.1 if -p +route add 192.168.2.0 mask 255.255.255.0 192.168.2.1 if -p +``` + +**Win02** +``` +docker network create -d transparent --gateway 192.168.2.1 --subnet 192.168.2.0/24 +# A bridge is created with Adapter name "vEthernet (HNSTransparent)". Set its IP address to transparent network gateway +netsh interface ipv4 set address "vEthernet (HNSTransparent)" addr=192.168.2.1 +route add 192.168.0.0 mask 255.255.255.0 192.168.0.1 if -p +route add 192.168.1.0 mask 255.255.255.0 192.168.1.1 if -p +``` + +## Starting the Cluster +For now, the Kubernetes control plane continues to run on Linux and as a result we cannot have a Windows only Kubernetes Cluster. +## Linux +Use your preferred method to start Kubernetes cluster on Linux. Please note that Cluster CIDR might need to be updated. +## Windows +### kubelet +Run the following in a PowerShell window. Be aware that if the node reboots or the process exits, you will have to rerun the commands below to restart the kubelet + +1. Set environment variable *CONTAINER_NETWORK* value to the docker container network to use +`$env:CONTAINER_NETWORK = ""` + +2. Run *kubelet* executable using the below command +`kubelet.exe --hostname-override= --pod-infra-container-image="apprenda/pause" --resolv-conf="" --api_servers=` + +### kube-proxy + +Run the following in a PowerShell window with administrative privileges. Be aware that if the node reboots or the process exits, you will have to rerun the commands below to restart the kube-proxy. + +1. Set environment variable *INTERFACE_TO_ADD_SERVICE_IP* value to a node only network interface. The interface created when docker is installed should work +`$env:INTERFACE_TO_ADD_SERVICE_IP = "vEthernet (HNS Internal NIC)"` + +2. Run *kube-proxy* executable using the below command +`.\proxy.exe --v=3 --proxy-mode=userspace --hostname-override= --master= --bind-address=` + +## Known Limitations: +1. There is no network namespace in Windows and as a result currently only one container per pod is supported +2. Secrets currently do not work because of a bug in Windows Server Containers described [here](https://github.com/docker/docker/issues/28401) +3. ConfigMaps have not been implemented yet. +4. `kube-proxy` implementation uses `netsh portproxy` and as it only supports TCP, DNS currently works only if the client retries DNS query using TCP \ No newline at end of file diff --git a/docs/getting-started-guides/windows/windows-setup.png b/docs/getting-started-guides/windows/windows-setup.png new file mode 100644 index 0000000000000000000000000000000000000000..e11c58d596e35013d644e6cd263b5f5de8116c7d GIT binary patch literal 51061 zcmc$_bySpZ_dZJZ&=Nxq-6bI%L$|01ND4@aASfx_C7qH|iYO_lbb~0}O1FS?55u`f z-_QGg-|zYTan2uSomtD}V&<87?t9;RU;EnE-cOXit{Q{@N`Quj2GLMgHb6teG)F^2 zPr}0h@4O(1fP??g-3`=~&^`>%Z-PIt>=bnr(a@^m2@zJ<;BS0abz^rlG~y1_KlC1# zH#TT!Jp>xcig&%tcdnejF^p%P5~{Xi5K1hx6)>3pz+}v}4$fQXju0pD+HR}muYB>? zMs$tF%Ijw@eYQ2ZyfaQEHsdIxIgS$DF@I(hu5lwhJ-WyeJF|k(EtbU6WeR;xTsHl7 z$*2_d^VWzL8zkpLhs#LnN%yKUkJ*NkCa2w*j}=-0vnC%M$4%Z%e3=Nink*~x3s{zK zcdT@I!3q`UP zxYwW8)B=P(&7r#4q`c2jSc;DjwUw}AoB!9#y00D9e@mSiMLv~Z&GH)hYOh;AFWyO+t&UvqYzx*oxZ?!yPkz^jRCZUnWu)b}9 zT(!9S`?UE~_Nxl(BTf#;RT+YWQAYi<(?oee?tQc8H`1AHQdKQB3z$83=dNaV{ZA{9 zzbgEZhk09fjZXjD_|IJ;NSJbEeZ>pAwL0dJC-sr#4E5=7b5J=SanWl!n3udn8> zWu8BSpKIC z^SASQ7=5+(Q5F?wLAccx5_f#5|1W1K*Mf(_ifZ@QiedZeqJn45=ignD_@rG=;;+x* z=dXTy)`YzMvtQCM(-93Qiz;nP0N6&&^7R=BR-UsFxt?=ke4hUW_s6TrfUEmm)?X4@ z2Lq7qx-ilk@wc5j?L9~*KK{2un#~LsswEm@CEUqisR9QYa5muCoXnsNwVQnZAf4gX zgU@d4`wgDE9IoI{kw>NS-*s2Em*eFpYRqa)I33QKcKrk;RX)GFUtxGwoNoS$4eru> z{%8lf1{wD~Uro2ewK99~Ux8#e&XKwpkQ`1Y{&+8f=7KLO@jI}0cPuTP#1mzP*R6O= zQqQ=g*S7@&&I!wOF9X;|NS|h=ELs2lwEVLDK4W5RtdB3zj!p$z60?Ul>G z%bWs2$$Ln#XGWa=9ph!tgMKFwEaQVs9J@rnX+=9YhX`7-VDPZ}O}l{?SI3h9(hjIo z9+cjd!LWF-uadUJTljM&asSKnt-Hdq@w&>TbE$0bC|wtOL&pQX>m$ACr$#FO>mD;j zA_$ieQZo<%e!S8dwYx9YrROi_POCvIg44UnGt~dFe$Izi5OVtLEN{2dj=Q-!g315v z)jZ;nhmZr8$`K3t-GwYMCrWNO*=xeX!NkNUVBLi=5Q6C{VfQ4YxKO|V8Tto=?(rvv z?>1iwEN`F=a?vH=@&WOoyuuZT)fBI54kkDLYlZU|!RCyg0^iDQCiQ7&N_zs%I+(N< z-S}rH*n~x_RxWJAXeYb^M;_T%0a*Pf>z$TLluDoEh25G5UnxP!x z?rYk9+HnA5uIRA+KqRsAy4Zg&REuv%v&m5Lv;87e?)$%N3WW+JK1HqBu)_SieY@cY zQ-^9I_=o9w2DTrSR+wMnY}Ru;a|0T+vR1-WXdPZ4v;>=!!%v8^h$hR8Lk?KsHm^Md z65|io2k&ESw+3OFCe@?j4+O#har2YuR4X&OoxCuR190sBNg4jrf530kmGiLWj;x>7 z*Q=!%!Gzdp#VG~8|4*~R|DP!;j?PQ|YEw^Q-e;>_OF(itc6v`~YGEgKy5@S#WzG5L zr~TCX=6yc4l~i&=T!qvEXRo}kPZ!VOM)$0-W-nIpE_7q_t!Vj4Vu(G*8j%-U5?ArB zg4JJHJ{)eb_~ zFuaiuy(gwRRMJOwbbS`kjR)nZzJ1u4UbyChE9Z!u1P6L>jf~z!ly;HGNRd4m!WzPM zL2OUzG5Tz_AoM)1j(@tme^CFMB;X7_@gT0YM7<;6`ZVCWq4?T+^?2)Lz(r}14ko(Gmil(aUDhE|#FB&sV)F7NTjEXBM+v8BQAFPEOx3}c=Ng1KS%JIr zhppF@Ps}Ftp3j_mA|{(guNlT7!Q{%77J9zh&|>;o2q zIS)>^$XYIEcaeIFn5-C-2)_moTUqJ}B5SN1NlsOLIC_l#VZ}PqMXY866&EgojVp^$ zaE;9%oQG}?uMDl{r5}fiBH9LgT^iigMqGo|EDsqxW%pWh%jC(u}JS6v4VxWV}Cu0;%gQNcr^aI|*g@_j%d<%pCGahq0KIyJx{I=dVyZTssR4 zY8VKZE+zR+SrU)8?i;NxgRXsU#XOm$EirI!?FPmZOItC>^*r^wY4Z#332!S%C9Mur z@oXosXCP*m{?rcLX6{YT*4=}jhg-J8HwQ9gZ5tpo2}X<0G@OPMrH)451=Z-aAjHjH ztZYQB++`FhExEmWBe+pu*FgvEM#F;3nr+UckP|qAZi(hZ(do*{(+J!*-+*~Fmj)$w zgFi|aM^U* z7WyjgJpHH?Z7DF(zGxOp$mWeJI9i&WG)k#wktE5RL>kI$HSdY9jtDlMeL=3=dE~!| zn_Rb$q~g7~D}1``zNn3%*R+@Ajs0-{>*^#At{bmsX#rZg!go7h5JHDDF?DFWW|S#! zVtD=R=MXLZG6>fG5#tzjcjs6^5w{GW(36Qf~}UYdGg* zCuX>duQh?sK1%0KmxUbZ(#ojK$jmSBWqgs1H1qiG$li+z{12aad?k7tm&UD`9L#Ol zA)fl^nhpz`5o0_khs^kFOQB#}`MZbsMgNo{)Ny;^1S-vI`2KdV42`rlf`F9)aS&C| z6SgwX<$(9+&KY7Z?#vFov75HWQKB~Dosb021lW_nn+t4wHy`Cma<*J@wI;8W`1LR} zoO-LSD#`Z1Z!K9SlHe9Nkr#B(ru`NizD-Cuhx>9z9K0W!?4cJZTr{yiLBEwpSc*Y88T zsOt$h07qYXDc0f~7AD;rzgtNdp8LAGzi3rR-PkA+)GU$s;)V_GoNwx}?1fN8N2}FGPo}Z-X|u(RtFo1d8|C822Jo2|M8*mNlqh105g-U zEtRFoSOJ4L);|VF5bYFvYTL4M+`Lr`8;I`ba6}X8amqS9JCN7N_C(j!Cc{Eo=~_+u zaegT9!fPoZJT9G8M`4!nCq0vpKD*%uJUOY}FP}z_!UwsUbZI}iE={vWMN}eEF^4Y8 z!|^lj1Mm5KKi)4zJEF4!L_oX)25?*u*Fx_79D!d!m<3dK{}%Pw^JwRwB#MPTGcT@! zW89+swZ`*M2};bM4$uEkukl_PN?zwIPx4{4j4f#uee)@%PN9Oiq6~2>yMI+%(yJPe zDbleWa@hy!9Tkcr31}-z>5X%qO`?s@_LIV5sCZ@A_a0}bc2T60qU~{Dlj&nPQ-+B? z#(P7K4D3ABn$sQqAxk2s`{`BuCR%9My>k1{q35xc-eJu(hh9qyR!L%e65Ma@{aN)F z*3Eoa@K+&^Br%WDs*bEo*WxPc5Egll#|^W|9&m*m6c&_wzrxgjOLbLLNo&Amp}M*4 zsc3E`^O<4$=({)0{W7gvF0KH$i0{dN82{I*+lpXVE9s3nMKE-P`J#Xo)2OhCZjRX2 z`Q-emouYb2cpy(yBOodl0x&OZRU!$c?;!69ZyVCZs~EZy2u6 zQ8l}k#>A#`7wq+u^o?mHs6T`Q5@d1UkrZBTxGx#jfEB~pB>Y`p@;oKezvlsnA5wkx z#B^{$)?Dr3Z4IbflwZL^^F{R00z%`>8CCVd)~iQ3)Dvho>mOshiS2%AVeJW(_9?9# znOY>LlRl*8<3p?o7FkhPj*#}EZyf24?9Ec-k+nFlUq$cYypa`<6hC)*`D1tX)%9Au zjG$Q~mf6&5-BF<8YR!OCMwlo9+9--ux3a?qnVhecJb$R*g*-leI%>mE?g{B5m0)kR zXvgA{fE8K66;fjxANT!m@VWEJHTuiW0K>_n;QZ6#1V07h){h4S{Z^CkOKvh;rxF3I zG0Fb`(&OzZXv64~6~6LRfUxC48DsZL1#R_?f(|YC1M4liL~#3Owu^9b@^w!N4Go`a z7F_AX}$scz!h0DBdMar zp~Yg@5cE~^iR_kd^s2-iyOGpyMP89thKWHxGaM^EIfl`r=_TAJt*GfVt+R0uA(J5Z z2RYP>U_(jiy-E&vdoo@G*!gfaj5&>qAJNEh*^wbe1wq6w&u#KhRG1~@rPZ#mkCbUL zk=5nS91r;uZ+bow?2@^@xD?9DE#dBV-M-(#SvBXUi05;Xl4Ubclkq56^Uo#Y!wAh{}tYjO_b5%iyvBErU{>4kfI`0b}>WIUQ0balaAwg*%@CwXrnb8wt z(KljUOVOPUDU~hcV;1hW%79CJGm$-`TSjIP^xCTeV{FGVx0{vAs{%` z^T`iy`D&NTCSOAAJOyYKEI6r6pNq-(x4vn0q1>~voi+%4?U51^azkoHCRFb0Ddksx zkc2ZnBt)HQl#215PaQr?8}?WHcF*QjzH66^sc70*_b5~>cy9_ZmFabYw@*O*l0l%V z>aeq|gWbC4WQ;rUg3&=H}b8{G+G7_0VtZ{te4@EqlWRWR|P{jXsVOf#4| ze?op$k$5f@>zi{tDhQiBUySZM6I7k>xvJS{#5|}3wpA*{&n-OUYBMVid9b(RVs}?$ zPF325Uqn_*ifc&J=IU-KJ39{aW%-X1E3vAL+9KGv=I?YOGMqq`z* zy>Dz9-wO^ffgAnLcfE=#s8CD%^}BE{D6!+HA(mc*ht_2)=CB7W^?$z0xcM$jl;~O5 zmrJe}mMcP60<00Sn%zGxmJq_hS@Hk#U9}6xWiCkk)vwbq`FLdap#S^9h08F3n{39V z@rUT82l5vLb5cl+;sAAdov~Rp%tLf2|NLdGP?P@DfYf-gbwarid(In+dfmO4`4_`} zYj+#^894{pN{F>v`g>u8N&9upx2{`N6IwrU+3=gz4vqV&mR2PP+OR*Fci2vFUo;yF z{c>qSO#R0+zzE_VBc$uM z0_MW{cIjlF?p6AiK_7hdfOAfIZ@0RyYa=UEF@-gtD73W8lT@Px+`_DssD8fTw03k* zzZyH8@9VX~WMU!bk>J>iBL5$E?wd6q{c6*vvt<{)Qkgv`ti5ryYRxPb8UO z?==IXs>6B*)pSp^SV45?cdlFxBZ z6E22z8{<%B%F8_MC8vDJ@Zn*fj?BGPB~<~D#6)|1N2OW6yPn9vxd@u%lQDWnWwnk0 z*-O)7xn$}nD;25WN9#NWR0FRts^gEMoC=hma<0i;Yyg1a3=+4!RmA*d@M{+>SC<9v zLS}+|v)a|ZMAGxDRdkAXkD?R8WoM3dH8<|dtp>Y;1cJ6T_)*tP` zScRj$g^UYY_pCr`#JY@;IQZClfp-UKl1v<|lL#)WI)3H6%Qt0k%t;$;+y$rqYaov! z*nER}qqm;(F%G(k9a(-*7M!V&~?=I=Szb@y3T7D%Pox_PZt!yvWwy2xK{VN>F@feIM7&&Q{tkP zC3YTBKVKgVZObk?shv~PJ%{IGsmo?%J*w}+1eK=fIBx749W?JD#>cKWeCd6F+P2bS0&S4lt6NZj#>vOiIgCuiPY zw3*SV%}-gnlYDJ3jd>l!*hHPG4o5_=>wu7t*z~0R8}O_S_W8FeaQ5!X=U|erSQsT% zOf?8{>d1}jjG!eF&%L$9MdHe);wfWNsy_TdG|3(nqUp)<07blhwHom(ewPbgO3FaK z{*CbSHQA-mpu4zkd|PV9hsLMM%zjAgBO66s9`JeMlH8Gmyc2!HSXWU>{7RvsT*H+d zi`OikVy28a$ToNADTWr+i0ewZ;FVa0u7rfvnhSCA@Y5>@9mQwng+n|s`dk9*BJLL$ z>KHg2N^J5Tnt?N{RdE*YoLL}I79Tw@6Diy9RJ?_7b{_lXKhe+*xb2Ay4~@wP&!F8Q z9jXg!sxUJ*tkh^!TQV`xQiylwoUe?yd-X(=fitU^S3{xRq|_7R!fjy1R}NgG-jNV) z!MIjzMUF{o)5(haly5A9W1%se*=}k7woEE|Yp3Az3^uJm@>L#k&i$wm@&(fRW4)%i(7vRI{jJaIoTtevLIna#k z^>-)}kly*UoI^_Sdu#qRsVEO3oF(%nqoz6}>DS$d1)?^RPw*LYZz;zH2$@lEu}5tR z{u;38)#?BRc4sXt3a5*CNn=4w$|q4w7(X^vv-Km8MKo)oG+#BF=q7Nk{t6PqPsrWN z3h>WSQ|ief{f|eith=l|lIDvg+w-{QAhM3jTBnUwpAw^$4igk4;8I`nReid6Cq)^` z!n|O!d@u1N#bQ90=})ZC9hinj-cLtYk7uv8bGz1^Q=)3h<+hyb!(1 ziwM?0#MQ!q0*gWy5SzHi!k81o?F}a$dEil*R&zzY#e9A`-1;IvAx8hv50ANPz;kI* zKb-F1Z{DugolR$c%I`<)5`kt{2*=k~u12)t5>T_DN%r3Bop>^*{pVB%`#%`(aBJU9 z4WJFyN~)*W>qmTmJq+$|G93NNaV-b=l0a`+i*Tw5ZwyJABM~zEp%I*=!2LJKl+2-W z-W%){Gk$ja)Qsrtg8iHeCiBu~<2UZVuY~(#N#9w z3C8qaePxf|l~IC*i+;=K%C%>n)W;-PR=pLG9c5!`Pr{Jvz1ozlm+9ax+4q<~7}_fR z(Le0{p4)?-gzm<;FiOc37qwM}l-ofuuZC{h_TwatZ0-34}St8esy3rH{iE zri2Su#AKP7q}b=TZGIQSuoq1x`~V5G$_-C$eNxxg#{FR&OC!oOskAcmC3@etEg~31 zZQ$P)S^nTZUbguYD^HS#c<@)RO{S_TOxAImM)vj1TVpIbwdFy5_AKV}BI$oT;L~I` zadCo1Ur4Wi!EZTe1__&KO5AxBk`^dV^{c&U=0wpUH;+<3Hq+l@Ipqk^VanyWT41va?lG8I{o z+=phKSWqXbMLuoe`btBap;rt3CHz65#wPSjERq~FeJoDY2F9q9EW8a^N3`I9clz$_ z2@lMeV92#=Bpe?VHcQl}>xjKyGv*eAXry%pwyAeCf5JNc_NO+w5)0HlZQVy8Ahicp zir8@Sy?KO>n&0Dt2Ur<~tS@1fAs;l^OXuq^ zxi2Gm7&ad?HFMW=`Wh!`aabX*zP$$J(_+dB8H};JI=qZ)vHj%k4bcp3547oS*!)Na-U5#<$@MX70w)Dj$GNK z6@_XxoHtK2Z07etL2~GF=dOxQk+|Cg^*WPK%#znF8}g`K;ZZ$}c&}Tm#DfuRwAfU7 zS)I?Fx6kV3XrOGPTqVSz6f3;pb?U0ow=6O_KfcGuw7mUHH8O|satv(VISD@U;%4#Uk^$*|Tup?F9{Ej0RpCG)shvrW}W@37H_-_vc7Q^d|&a;>gAnh%!{# z=9m~27`o@ItcA(>W+)x-K+@q9C#2?|J03zB9T8D~69(m#O_uTSITI1eVvjHaP%7^g zmG$EF@#desumKZ%Z>uz-WR@NPU7^&;_i*NJX&gwKeU*JkQ9-NGmlvDkr2|60S- zajjA9(-9$3dkD^+)+5%qEOqA<&@(6HYU`{~OJZsxrcr|8V6cWKgs8sSh;J~ef?F{uNy ze&jhJ6{G)Q$p7K9gdSX`>wVC-L4s~G+deuV4*$Jh8ULf@Jk-9(Y1ILkgC9mU3BzC2 zv!M*|=l2tOUd~>EK3$qJ&cAk!(l8SKvwB8gOO%YFVZ#hW{r|QZqx+!TDQJzcCkA2t z?<2$qDpB9|&2_fKAdqGM+bhUrp6=~cI~j8RpFaCpef+^$dRLDH)BEpH0F4FFT1Wvb z459{gB5a?Q%5H$r14sO?OPpMO)&ip@^#;QJI@ce`6E%0hCX(>i=>IwDz#h<-l?Q5( zhD|={|6VpDNY1D#Be_=r_P>`cBmT3|7A3y?`?DMGCcsu)$!UfD<+zN50>(#(tzED) zrN66B7i_vvlSgX&=bG^4Jmez}QAU{FBNv=#)}GbQzo*8Kh7QG=v5TrD7`5p~j@`Ex^38=pbHJbw#FW2JW9 z>A27GMcY9*e@bxcvjY|E%IEE@aZ2Ic*D?2k03nhLXj5X5Y;YfBMGaT<4pZnnun+&Y zw^{}cS{(yX1s|AI4CBVx7-0phOMLFjF=*gsA*G*B@KH{IsDCo;Lv45aF;}GpVl8In zpbhiJ8R$EZnscd(ZUFCOrl2;h#shxKZIIOKcS$05_|;cU9wGDD!}33o*IEgtDfB(b zX*W;^(YzR!cXa`52UI<3{Ex1~O910SK4iO3m@1DSOFS069cHdAK##93h9akidFrZ9 z2#RBDjxaoh>Q#q9|J~z1=Yz9x0rmx)1F%yZpPpe7c{QHd{g6wL4t&6Pg?Hr;B4FJB4xMFs1U?vK4y*t{o(hGPZ+Y5^a?_OTVh|}Rk1WaMyscU@^?Nm}J7%vt4*-iv-`c%(8B zd(-kTf|I!FHwptr6DI-G-==sQmyMk$PG3%^gU3>;D`(4L7ET}f*eVLx`6<@q$J+R1s1DdZNO5OSnn87{PTZ}#ss{tI@s&!> z?e!h}nBRJnvbYs1IICaV6047(Lh)~^6Nl{Po)LX4t=BPfsZmz0-dKGcs)ZTA`7T~z zK_oT9D6~^o`ie+Lc=nFPV1Bt*_dQf(Su_u{ix8{y2B&sy)#aS_q??9d#e%vy7Y$2) zT_YF^Xw)^w^QYRle-aQp!hAsWsvJ1W_OJPF+JgD13jA8MV47|?Qz2zy6gZ~KZ@jz5 zew@LZV4|Q`0p!`2f<<_sH|Y($ol3wYmZf~Ra7IDG@|fuM01zi0k*&o8+KQu&%E;|{ z;n-BZZ?gN-&A;>Ly6pi>(U$RLo9pHdGVh<*4d!onBnH7lKe2^0x?C$3c>^B?c>J`J zPX1EtZt?wB#>36PYuOYnKPeOzj)Az3>=K~NZ0upcEJ9`GMa~c`zUb;_w!6|H+ma$&N*pW1K&4>4?*G zd%H?Z=?g8+1m>beBeWZDoRthMN%ZIm)}Z{A;oF13w zYIOy|(QV|%?I)rv$qZ$(C5>gEE$1fWW#j@QuzppEJ;brZebeqk)6+YRG&A5#0a5T% zdyQvJL`>+z-#{0KXE$Q3VPmNYnvwi}lK9hc0;9(u_dNVQ#zB8MzPk&O!^QK0vO&Xc z^gmy}YJbopZ`GMe8YSl89(ZuGs@eLq86h|(bXAqD%Yu1_rO1kKAK+_}Zf335+C zX3d^A;j%Ipiv^L9&0zN@f$1`Vv^xCY37yL4gC!=)oe7EP(#KFuzjN{;c`EP*Hkk z6g`n#H)}oMLcV>X;00hFWN{zB@~kiKnE$c1S$j;oky$;-#S=lShhWR}_vo+o+&74@ z_kfAnOEH2_0P_gd))7Y7{Uw5TSdr0XN0$xrW-)yL{J9wX>5&C5FX@qYlxV%Yf&Bew`Pn8+ulj zSO3rpnOy(#9l%eO3ij27_WMM>OhiGhjLmK}2XDoLN%ydR#rKY1 zKl=`GsIPps%abO)?k(UjppztIIkq%=2ACQx28c;bUF>cu(&Y~kCpW8}%v3uQS@a7i z>VO7rS(=S`N1u_#Y6Li9U1S z+P+#uFX^5L_^ciB?`o(nP;&o;p_=d&?t$fWC9oS9gcI`b8_}nT2^Zbw1V#I;BU)KO zNEL4z8$d4s{M)gL|RALx$6h% zLeoi~XGJ6v`1|VS8|p{z+}41PWQ$sdU+=LDzsP2>OPW1gSU&ld#Z%>KTIC2CS55N4 zN3Im4C&4cTwLuU-PaBXW-;vksZ@51x{*IX;*Lz=%k&-Lg8(1Kw+R)-y+IAE4g-568 z-8WXZCoW%DUU+#e4S}&fVZ#dB^n4B~h{Ke`!rQ%>p*&0~%?>cIl`Z8&g}y@Bq>51% z-L2%7$^5FtauE=zTCIf7>Jmp34r^x5g!}aGzu_`dm+|}pxY&vZ;19%j2DKA*rkY4(ZC*JJ9f6vJLZKK|v+$&=@T6C<0zFaQN>*PruD}{_ER{ae zd-qL0vIrWWCXw!SujG%1K7L!-m3*`MMnM|e9i3V$4a1ay0uKUqgU@?Fc_(uaPaJ^p*Jmpoj|O^Tb07w5hT6!^9f>u{UVmmx}`eD-p$n z#>@qsr|Z}O>GT=&(*Jx0llgTSlYu_{$KcT9GXbXq5NxB38a5UR?@LB0M{)hId`Yf^ zXNMHkP(52hhbdgt3?!Bse4@d-8qW6Kqyasr(|8khy)e!devxb-?My&dI`uU+&@I7e zekQB2OjqShSh~ufBM3@7rIEb;1ve`p~ zMu8Y#9_^O?yB_32sgZ=mJVa4QX&d>o%yL})cUf*92-3dS#+MWcE2A*U)dV##eap7c z-=-j8o5)*s<#Wi5DUF+t@}q|weMSfX2Bh}_S?>y*$x{L!mWW2t7YoDq?f&odDmlk+ z5_1^JG?h(s&feW~!w@d4*aO7(`^W61T`*ez8fp4Q9sey#{t@G;%IDX>2pe%7pMktl zEwU;UCq{2lDD_I6$N1}XeXt&g(6=Frc5WN3Pr9OmlUrf zATS2a9Y(j2xk>0I!Tk-P>p|JZv);oR;5g3{UcT(}kYE9<4u^3^wxKhPx6W?VH z9u25wl|=*xWK8G#F|t1GJBH^D$#V;=Ph?A|WP6d{gflDT#KT4#H)L~F?>{YdWwokK3Vp~2uPc94!2E{%g!gU)CFg0v z+0FA+hlB_DNj{koh((2cq2p3Kfg%hZGwt5$OresK(z4Kd8YA%B&?x0SGC}CwJNg@7 ztW%(>zknyOb22!4kF-r^7;l1O8B>a7IkDM9y<@HoZ=bLtoLSF<5%>)^P9(9+mSS+Z zH0?QQ52)G)@;qT{iftrwXhz-%0>;3P!x1N(ByX9K18MGdEztpb+46&!;vtGI1hD`k}k_gm}yo%A^J$d^NO@ljP?PWw4;Pc*;qMll3rZNuw2%1T<&hk z?1!*uJXUsIZQa#km~g0-`UqZvxUhdQk88A7*p;>IJN#I+qE*bhZ%n3=(2Y!-o_`IV zwp7_L*(Z4Ba}_RaaOdp&CH$^f0NpWHNK}Z4#=%Scjo?@?wFQItDyzeDAP%?G0W~j1 zWt3VPqN;>&7{m#XKz}ef$QZ;{t(G0s+KTp`ZZcl#Prbbl8v%kuT&A>OU^sSP8`}u8 z?26M?DitCCa5DUIFaX{3GuG>&MxRp3J<+U9)G!Ocqo4k}=Iz=n_=qgS{9C(>MOJOv zBJc16j5#m0v6_gMdB45R>gH^hszx9BU*=pk-e%ul34`D&jwDp<)Pd-8gd7F7y5(yV$;Is@2vc( z(?;G8G`DxY6R}quxf$~`((5Cq5S?2KhHw9h#26MqaeTf%8smd%L+yIiGTrLFuicVC}D-P0kz-l4q!2( zDYT!biCGdV4@dMb-QQ3-9~*y7O=jZrJoF{@DUW^2VG}b}V*)#PGv3%ISdM;U;*N_I zsFQkck#CURCOk@^<+8h+P$scE_A!VOReqYJF-o%u$bjk7@#w~}C;wg+Z^*~l$EWT? zL9KxW-d(#Cv0Q%+^(nIfmS&;3CPfK1+Xz5G9Nh8%$GUqH+ zgJ7Hb4%w@Mi|rCCy_~y@8$vAaIeb_XmU^SfVYt3#FlPp3XcwQm`f8bW)-*Q#aJsYtF%^fEUd4du^R@n%d*1cvtrpAvcsnS&nPXzP-4^ zb|dhmrimMu0#kg4uLT&h#YnS7zWS(Oxfd7-$YHYbOL|Z3C3Z6FS_g5o`HGNds&AF*cvE#i|4q$B_2e zJ(TyDr>~Up0l9fBly8lBZG-^#D>OCmBk-23+mO+i*)zIqgV5)hIy$xx+C6 zrT`q;`bmPwx`2eUrIl3&3UtQETkMI2VPLUT6r1mo<&8#bt0-ugKvjB3C1c4A zbVJnzAdHCb*gkkh^ua}{n^`48x$ee#wx=Rf@gm)C>fTT=2U#7EGD2F(NOL?`fU=Nw zQMviw-T?4>pxc@>x-g7Xl4Z15T8;WY^6UBScX1nu)nS)}q|tY&=Ev`!=to#- zKZ2Tv?*S3v7n45uqpo-v7Dg|{SemBwww}*eBd794`QK7|1={6H1EY$$VB=Rx_CG4p`lJ&h1%|X}P@(q9-OeGdIZPe{GJii&K!ZFg2tB>ZNNb251j^#-N6G1 z%Rmp3Eau$ok-r*kj1f>}NhUvD3vf;qy&Ha~({<-<&7)~vAE035D|gb~PYi!2YQ?>XK5Qw~l=-5co*=)EC(<5CqdVQqS~_G!$Jw%$C_3iS&C-8M z1skEa?Dn-RWhGweqjQZeNy_v76R5}`x``_o>&D+w)!unZa52>KVe(!bFE_Kfi|6%y z%Z>c@@7bOWxc}i<(nH_kPefhkWcO)f75k}88Uh?~`*H(XMHpI(PlVzOoh`RME)+KD zwwnD5T*B|romg{c{O+T4`8*-&zFi+thq>{>T~S;b3le_zm>{ zO~R2d4LCrPWU*OxhXjDMr`My$w06=-Q= z!%+$Y?Nj;@iz$$Pq5I^`;Pm@wUoTTC@BdzDVZ_m=Iw$SpUfLmNku3Hs>AMDr6tVQ~ z{!wngm7f6*ukLL-+2=uSp9x;7%o;%KqqQt=QvE1e+1>fETc@3p^y2Oj&=|QX%D%#; zlN7iiUsger1^3-M`|AZfj(U@}>ohRd^_7M~p{o0YN7Q=#+nVhBO z3_NtzboR3Hm*u@+y+z9Fo5C}66oX0C@gtz3*CalPtljF0==Xh zMW!P|d_GSj4aLT_*y<T-L#^vz=r7IF#&{ z$-Chl#E6Bbj~|lfZ!>i>=_BRDQatu+3-hf~8%ov0Aw1JC*n2)W@o>664tn*tCUvW) z&45&rS2yPv<}jCpEX)2q7U9e{hNc){o053U#YL|aQnzsmECpaqrFRorYy&KQzLjNx zW#0F%ad4?{DoYWsnuZc@;cLCSEd7Fj#_6e_d98M=TtWx`hZe_KK=KN;({n4^(eS* z!5o%RKRKxo4i4Lv0n{f^O;e-8HrXZmfg;+z1ZX`IR$!#uyTuD3jC6}SJ_+jnSG6Fb zuY%Xf<`#*1fR0RJrk+kCWs1MgJFdZ}|{q)5Z-G zODx^pu!PbL(%lVG(v75ubhEUubPEU)iXZ|4f^-?Qh=g=1DX9qWtX^LC^*rxi@O<%$ zu$((H=ge{ZYNB0>*yVx))biLE{LLUxfPr50DDhj zuBn%+REN%tZZQdDj`QQV6mIP3Mb87dq$z#7_=2G8o=8RH`=K*{#gt};+isYPulhz(wL>aF(% z@HOqp4PCFQgejGgS&eeFs-DVMy;0@l?%Qk(YZY2$0aRCXLHWv5+FZXaInAjDfHq;n z&)X6qM+?K&+(d*8u65cSWeLM7e^_LVA+CtLCEAF5)N>6(GvP!!4PRBFw}`blT;rWi z1bQ*FcNyzm{pYdtKGk~>P!-P9Ywls2XaVs^;PB7yAJ zIH!Wm(Ad;<`WhQCwJ}Mp^Uj_Kod3=+3nLdGoRRC8GaMI3*BAudLB$mPYs6krs2kA|Yx+c#(RKMyI}7 zT~_-oZ!=->SW*oJGK{rwH^m#R@6a-7tRFOMrT`xTNe`3JPIir{tV64T&? z3_yZjbm)7%K21ipw9jglX1h{P{l9(T7|%vY$@&(|T##lG))9K#JBrjXq#zlDRuKZ8 zZhNfa?}{YG4#63CrqjNla&BMV-~qHpgITmrI6!fHnG_pN2vLV{E|jGxcMy#Z_>M4R zG;9qv@dW+2Hw5L!`(1jl*YXS4`Kj-DP=^+MeXX@Z-q=y-()h9(#DU7sU}mTnZKa7m z)eI<7>dkBtoLHHOsHlKYn1KJ<4HExP+y8iIeL6zU~6vAtE#)<6#$fzs>yN&i;X zPRsod9)g};r(p*N;?@uF$eyq19?|2qgaD2J}3J^E-VtPUhk1Y6B{S#>e_d!s0 z0BMd7pvoxHlthU@ZB2Tj)jWx41`MudDrUNv4j`F4q(zWi(S<5Wk@sBDt@KA#5tPTu zGXoZ{eu&*hq28gVxnl2>H^mNUzbwiE)0k?&=?;2$C?j*}w+{?&9TUqe_u3{!2S7_D zS||^yJDw~p7F9~niSe)-446MI=0U?ot>r(q{ur4=b0TWh*=4yJuIj&K41E` zt~sPnKJ?|ZR#!COrBro0mcu?INq*IKHd=@pw&rb0;ml-7XoWEMR~tHixWTv?uQn0< zD~_b&?(oEC=MniB0KrGTJasw-b}n5w(Ds?MCYzGg+{wjy>c-1wW^qr$OXV@aaGsp6 z$s9UTPvJ0?SfId7GNRc9str}MgZm_lbS3Go!ufz5xbr{Kpb^ii*=5;HI#DhebEV1QThZLaN2=$inXMLIp&Z}-TQPemgfvo8QRJ4J ze+P&CR3ETVRwe;)PnlGHt98nV;*9jc7DU>yW@USLar$Uf@Oafi4F(?r#NjC+Vmz=Y zi26Rps{2cxiw^W_|8XS5^UQAsG-SW_U70^gb{t1XIj9L$rv-wnE2X_~+O*D3{#DYA zx9%u%UG;yzjK$spEOP0_hfH`-KYySwfWUG8J>T(?>&{*H_yZcZLB~;*psYx9iV7kn z&=1GE9A7Z_vvReQOq7Oy#N1WZTK=*L%znGabsYMI^Map!YpW^zsNw)=9=MTp=0cHK zVf})cnmF@Dd}mBY{4vwL4peo|qVi?X;y-o?#p@x^uI49~vz+Bh1&5ALX@_m~BlD&D zyot0#=I-6nkOH|)r{4xqHl9gA{u#5m1owzVY+?JJq|rRUwUo1{7iZ z&5&eyOr|9DwrxNg(G;LMU^Uhm@-sX$&~XX!f!WR?k;ip02|eCp7X$dSAtL`3vfJAW zxe8muOv)_I!D+?@12;WQr7#&2NKN(uyj z>yEw=Qr$=~j!O`RxUdgM^}-EbXfQh!t48V^ZIRQlKC%!W6iOBzk-I0h%7CdBD!p1zb~i+w>siJDO-!p5hHZ#?&J_bkk{C_7Wde^%t!U8G+Ra?=Sd+4L4{?Tdh}o8`cu=S}bM^nHi0QrX2>EwKa0P?*At+KYMk zjqJy>gAaFCe&s|RU@EUI%0{loi|M~3lQWjq&w%b^2kg?Awuim!xyO|^mO3;zpwVj* z23Gm8&Y#Na7bM*dsO|LQVw0L}fi0eN#+fsw6$4X>?6Z0rjOaei=h9*ELBT&>mf}fT2`nNanH8z^Yu>qHR9C;a@EO|4sHms!DP}7X zQA)Df)W@=u+TB1{u9V_=d?fg)f3NnzGI!^C!YRzLn0NWXj+e{O^HR~+8h(H{#-zpR-Wg+WFzAeAje9Hw$#F%5l{q@MYbS(0KLAxskb4 ztwR1<{`I_VFJQ#g5)Z?GZ31w>>^ za!Kmb6L3(=!W8GS=3fLe=BTxcM)HtWQO^DVC*6|5pSE@yLkZ6}MqvL@00D-Y9Q;Li z1z}=jKK1uwCN1aS7{IydzLoN$h#l-QoRMp0Q_FtINxN6{-d4DifW737x{j;--AYi6 z<&dw*vxPXvwNWN5`@Tjp4>dH;a);;Hi-CA&BV8&a8hf7pqZ$hqrW5^4I9S^2FEtC~ z>Ky-`DxUmzX`d31x?(dCQDLGEmSy&HxFtY}52;!wVxjmO`keT#Sehk+Z}_Qr&`V!v ze8Qm^fKz|VX~lT~&W)V}h}H@Q9|VJsqV?8fI2w6N*pVy^)n~!S-JAZYmUEo0@rL8L zM%9rz#y7ol|Mai!kdeefyO_;aX9Xy5y|B$^shqJxdzGzdqJU4Z@z%I_7UgR;p*z_5toX!xMlEX(80YZY zMEH~5QN%ndz5>;$k4 zy%d02cLTi_nP*74plsFm32(4T4)?fodqu zp)H0#XHHjGEl5V&@s{37$NMgrGgPYCs}eX;FBkr2a}n2L!s+J=%c7vV*|hpp~b~>z|Ulb#BP_<{sUhA+D<*l+(t-sE)aDN8=sm^q)b?p)01)?zsk*W~Mt- z?QUKG8$Mdgtr~<90p9!a)l)ERfRrVqdIfN&1`u&(e;)K%9Gw^LkcU=h8Rl`XxCEd0 zx3Cc9aC|NG6zv0Qx*wkG>{p;|_dstWx=)4XLuXTV0D{?m4yxP#HlG+Mf`Xc}`%vJ@ zr4H$%BxNR=OCgH=qw+q92Oknk^*!}e5-6(+mND$ObLLPfT9|SE*jn;c!7uaosRn}| zp9%XGdf62iF60HclP^H%{^W8uLeBmWNb|7nX^0?zL zrXvyk@qqIX@~`^>-biSA@bq{sI`z8cYw0hd6+o_hc@b<)dPS{YLJArfXnq)YG$1Y9 z?j?=8d!P2NHrTMI(>J3KrxEWk%pZs_?@BZ}<|;*j$?)1uDFDLhWPoHJ0_(`(hW5RP z{KeKy;=%SqYYxo~d)_P8irk&Ikp zy4vuI|5)C1@ynD9MS4T>Dr_3rQi%ROMb;9AU1cdH6%QB_X7*~HXSbCm1xFNW8e?H1P5%=i{bT(9tOnNwPT zoQ^XK#DFb=xD5)67w|p4hjh{}z}R4S13@jy(T)KzP0J0d5?*;QrbMoUL_hmggIRH& z+-R8!Zye+n?YnLlYXf6z?$_x!12y0W2RAYv1-wC_04pKQD2Mt%-L;N?0I5$q@P#2Y z;7HTTa2wzW{|M7hQ5u)N)Lq&7gfxmNY#om?AV5F=jQhNy%M?06#l0 z&||{&n?UN0qyiSa{8BgwIxkT1D+Z4PL`d<1LP1E$WJsydU;oxqGzI&r4VVq+RZRIO zv%(v(ppRttd)NejYk)~77X`Ud+hSrxEP>Ia2aJH>^qXH;*3zf?>38uCL844IcLT`o z814pE6sYqHr%=EmhlY4@{rmCA05SV6b_8aO{gHw{czABV>v4u$;3E)_Hda>ar6}9C zfWu0j0e!AVOW-@O(wr8M!bSZ=uVPl$wZ{SGMt68Jpzb>P%-!K&H;!iP;G z?R}SlPyW9b4ofXgK=O-1C18+f=uk;^{t%6-xl+Tp8_D}x1h}~BW2^4JtA`w{f|`9S z7K|P4Kp{|20#)%+za8U)5llSKU67Qf3PEm6nSR?Vuq_+9tWNAEP)I1_%AEALN9*_g zHKCc41EbZeiLX-=@rUD9wPO~u^}H{kA#)up!u&V0?qVWsMD-q2YuUD9NJ<);%$)U66}i4h2UnZv2BXPuhdhy%G+w_u-`2*0e?-NN?pH?npR>eTR;fcYboJ%FCb;z4C>98$DGb!9*J*9au9YXvD*Ohcnvy4=b^!s ziBW~391T#DGMf0?Lw8b{tJowmGerXaHNnB|&IN{d$~Y>2uHQ;YJ-vY%RJZzAI?^g) z7}ynn@2?q0dk#EG4No^#ipGTc=!=ld;%F`#3vV#5u-XO4h{57)pLo=rV9W8Pyy$TO+6a%jkdw8R>oWWxg zFk;A)2x|AvJ4DWU9|++88lU{3jZ;7>r^Z72@Z6;it59}df$SczYe0DaNUfITh%Mmu z82aJh7k*>v^5u7s&6jMrpN;3~73c#RaIcm?&40gEG$@i~fO{AD^GMb6zrMqVqF@m4 zUOYIu_JsZ8huqE#Sat4naxGHAG{^g;r+;aY$wfTj9Rt4kYQkSvYY_uowTeIhsEN;Mf2qbT#4p)CKrJ=+wbyi-Y6k&u70Q?z9(dj<@E{ zCjnA>5-^w}{|+WeUmatr(8`06N-!Pcir^1}-}z&^ZG;@kP7ct$FFjl&)&yx9$3$RG zAA%E;&zw{s&Kg<_%k`xV%bWsqiTApia2q-BNA2;$YxNN={|{)LVyhZ07%<` z$Mdk86pPRrsFQxb4dQlSD&+p2CH_6Y&;{UAPZDaXn_HDhNaG_YY#zsePXvaIKilbU z%+tK--3Q300JfeGVERe3HJwZjD}6wf2j=J!93O~dn@*zQv5?2sdTq#349J|t zdylVgVFdmK-VDhFLh`-PnI^oW5&Xnl3&`$c^7=d<57{5!Yt!s-K#?NL=Te8?5P;6_ zTm9q|6^2Cr{PzMKK*M!>dr|DPmsADg7+2{Ou=d?9Nbd-6QekU3r$0sVJW*4rz&%^xOi#7HzUHQ7qsFl7)`jLuWv;0b`wM(+2Ro_2oM$A*-5h zr53@j#cM{e7>-7CaOU3vA-lnbalh+M`}j9Rt}XJn>OqlF@8VTF1|AR$4$^oUr?Z0o z7^C&b?@I=@t_!|i92mF>#oKVCkLDrI6ZP_C%v#jqF3HlkJ8Pn9y9YA2P%X&zEG4ovEukNT^uR49E_9!c4z;c0TYT zF?+}yg!ZnXLsklD??6sj?*|K{J6(N@wk?p#Z;){B>4|kW(yRt7E&4d&mEQ?mMoV

{>V{k_u(+QAy%&Q-5R1AY`MI7LNb{+z@!dhi zyJ3=sOw1;cML`PEu~`zh2kWIGb!5Sd37Vn}Ak)%|12zS;@QB$O8O)QCb?V`TdUD5) zs&0D2;7s6YaUeeHx8CzP(9O3ofII*X|GV>Mi_f(61;t^nPykKW?o*kI8xR)OgyHB& z!;eityG(*fV1%Ql>Rm%~GF@CW* z!hjgRr64@^mD?`7trc$K7&1YrXtv%S^zAWlwB{;h+jACo)yH)%8D0zBeDBd{A@*wq ziYzhK#!@y|9YY^zQ=XsYKwm-{db=ZN9UNeY_21-bzHWJDJekdKHw~?q^Qqll`K(Jd z_-o(50VpOj>Db_E2Sj(RtbM8^vQ$Q}3EK~qzf#^_>(^cKa74M~yGsqhD|7h>1F%?4bAEcbnuw!S zray%9ntcnxA+5$gf8py@{5xu}f>@u+r!yjs6+4|?hDA4Dz+SRWed#^=ScoCor;i=p zslDlNnn*=w5Q;a}@^zE{3DE830RGHD@reIE>T8HG@em!L@4z?>I< zJM8TTSy#tX#{?!mh}H#nlvxWCnba?N(iT20iTj{tJG{{ke73w49(b5@#17>BO>F4p zudF%JaY<}u(r9tHP41MJip%C0l3MEJ2pxD6^c(8>F7X{Ee8QQhUpG)kQ$)Z^+(rrd zNloLlTdC4UUV}6G(_%fMJK+mS-FSW9V zPLZ$o$2jj?CN?^?>xocFWwtv5W*mvh|xB3E=MfxV7wh4|tmV>gGFh3_fqS}Mz|MfCJ;dTXr9 z(fFhkl6auzrPEGI<-`;D=)s9tBj?8`VyH!Z9Vw*yRtJsnW{ry(%4AbW6Avme;Wg40 zBB9oDvI?Z?p?wc3&{Ij1N<0g<&!pHvZE0wDO{fM9R3qU20Wqw0Ed9QwnNz;b~oaG%^l00B_*p z#(|M|{gmXcNr~%UGv9Pi^JSjR#PwM4v{7<*(R_Gr7_C%MT$|ZqZF7|0V7VdQ_EB78 zMJow4pORrZr~tDvX`2A{GJ(V!nrBKdyNKWboZGvsWSRcJMwvMN?V$N+{t|JVk2;mP zmKdL}G?{b5s~O07Ip*^9s_&^6{&6^{^=_BBjhbjLQi387<^T81KzYhvHT5!!U#73l6nQSjK>giSL`! zht>5=c5H9Qn>oiSQ7AbTXcL?(eO!w%z(@ddG5GpgVJy0Q9FLiQeT*+4>c_|#n&sm! zXA?wniYhQa0Yl=jGDS`HZjNojkVV(^l7L9#NVSx=_og4+fV8Exb?08GU}Ac*C2(PF zBPp||Q=Z1Ofj`m)SE|(84kN|p0o`o$Y0-uPi3uq0^2)NUR zhNQ# zaNRyeX5q;jkgoQ4jWt5Yj3X_2K2_L?##dt)#j}Kt>3z&FS9}%}RiJfx8CHU}2A^zj zQq#f5G!jhULQOyxdQH|e*`2oRufF4Alc5L%eTYSgajb0rxiZO)k5Nb^Y0vfSm$LFK z%vWO<1g%y+&n}7os+rA0YdNpgd){Rw{@o47qW)S9i2E-~G^<2A-W98MC`yJOp@pmJrOxc9!a+lPfFtgH3 znN9q>FTScE!8r#s1cOaz!|}GUPnZX|Ue}NgY4mz<)(3yXL@u1!6VA_Gk3Xv~Rg)00 zHkSvI6NT*mY$ezE0uZy>r>zvcO;RbCA1i$cyTLwNc{EVnvjdaM61-mN=$(j_YuY7Y z8v*u>X?T;HGhNkA_Nu=nJXi&Mwj-|QoY%-FNb%u?qo#J7a> zzQ&u(2NLfhJGH9W)VjDH88f#Kix;ZTGpyia{AZn6((EWdzeG1<_gcy8BtLk~eZ(*) z!tEzxWA*Us%V*1E+|_1gfj9E*q(9Jj7by&~cwPh$$_8l2H9P1Kt6$pKh|R%jp6ufu zmn#xF;}6N4(Ax&#yE+mIb^-ORLn4db?~zL={4lu!_yAKdXjddAYzLu(WoR>8mhQkA zn4g|PipxxX^NOD3GW0(Be>crYMnU6E#vwsdDnV{1Qw;r^*M5XX3nEj3R{-1{PEnL? zP;<(e86%~le9Jt#2H)F8JWF22BK-48GDhww9CPgAX)i;ExLRX6#4wKCB57o|!IfD7 z@l8?^=P9-)&jLQWjJ=Mtkyy(M0XYg;w#CEJ@q@0mOSaPY1&nqqNs#hc_g0|cT7C?b*ip0cM*MF%x ziLQh!+_R>yFW4n9o;<@1>WW#iua>I=_--S?DAs{7*S z4rci%GKbFKwXw(An}`>^eqTS$h)dv_vz(MrpMAfoe!*nI_T^EJ&@P2kr+m=qI*v_1 zdU9Wk`riSRm-@EOrA2Tdr0}{@wLuWhDD{ouwR}ZiXv`b2{7xHYdZmI@lAy@9GE0-+ zsjKh^13`L)*`2v0N%h5xFU}Kvs6+VE6EI*JD5Nu``Wo5LHtM1mdgF{J{`MkuhT^wl z8P@{?V_nbfym`edyzloCy7^A9;Qu)+A|z<5h_sZblbo8E3XD{hO&Pc@X%U@8oHZ=A`SVF#Hd z!Fx~9$%^_iAG`gKiL%u-G(@G=-nHiJhLx}tmFaryT1-5MxLW64P&F0ej8l_b`Sl}9 z?Sdp6`qez`nBKV-zLucMRXl%rxl_v1+kW4!>xW$F|5;>(qH+&fCJ70J_LPasLaG9d zpx#CV?J zL}FrMq{4~_iQg|~7}oHUCKF!7hg_qPt2|~V-Q}KrSRqK?4VJG}0;>8Var2vA55I)z znS4b7c9Y|>z^L((A4bVHh`d6vGA!_YZdfWj&vo`i|K2pgv2jz1U(b`o5be(z?i7Z! z^nidM(@!h$mmR?OyzvaBGVd&gRzJ0y7GZLsD(~e zMOTlGQsdqXO#CL|jb%+FkrCBNmv{QSlI5&MEM(yl2MUq{5*AP&gL-$I6k=W=lkPtn zJ;1&8kQY_RSHcR>cRPyV0~_LlD&ap(fIsc@0kN(^5D~p_f98HGF%q^{O0*w$5QQws z?S96#bTkyV$8U>Pw!MQ%aPxa$hU`6bYQgy2apmH%k~=G|q=12)5RwSYx7r3H_LeH_ zfC?(cAHO9zPS!DMHo7zhmBl1%_Q zf`MDR4!~^$7msv<0Z7yg(hjsP_gNy58DNpB9Bre8g}+Aq*W}q9l;NP=qj{?KWa5P^C7)>-8cMPonbyeevj$ zeFPbTq`l#Q3^-3m=nm^fBU)WrrOaxpc7C^C1IjcY9wVjrR6zVKI3X-sTn<5l{`1-q zE%)919e@>`Gj8obd`GT-Fzq2*v0+AJXw%gQG1map z00L*@SKvpFEscZyv@HAZTiooid7UDDHJh58j09Rhg9+6}ebR<&yfSvfm)EdL(RN zcR7j9tuPC7DQU>=kcIE|NO{x7BZkLS6>8VyI1D(IM?ntKS0-Zay;7X91ph3osSjp9 za_Cwv0a@f-%ZATnF(SyCmb}7AMqF4Z0b#req!m+cGa^8vl3?d$(iqdD-bv9*dZl2BXC&Bl=T1VGJHF| z0p@rhfSHu)5Fw`*uRu(n(T0hwRLOYYZ0ViImFEE4IjCKI#J`tr5e&lnnS|Le&?$U| z%`6HMY^dE;>SJl80)wVxnY87$*W_gIsBdUBRcjrg*rI^FL#|}fBy+D-P1BQAa!p*j zwnOoWM9a|h+;O#T9tu6X*nuI#no8;tW|^I2{_{^E+JlIhD-ZI|zPXh50$49y$=MmF zZoKS-@h{z&FKV+Ij?m?&@}^YRf~q0=r`q)iBC(zdZ^QP<(a)SHQe961TN^f9hem?Y zStIi|ACaKZVbc$oNqtNug+ma{P$l{yx&dbHMUljqPvVW&#or>u-1QKFAY3bS^^>}# zf*HEikuNSdh>%5BZlT_VIa{utl8&@Y<_sw>` zWiPT?iR9{$(S~}9N~iG#Q(FEnAEdT?EXS`WWd-*eBc2l$d6ts?w^GIGh5YF0&9m;9 zOr+iIX4%89W+8=YW>AlB8*#)rWD1B5%eIObp|QRrD3woP-&wt&W7rrwj#SM#9-29v zjD*Bd1*iSf6{Vd9Mz{pepUNW`qWAABtMcplg`xeC*SxINEk^<$oY`Z8w@MKB^SUczY=^8$cIN zgoRO9Pb#DhAX>A5i@qL20up$2A@-NulZ_8npdc>9=@$smYY zf%W~jXTxMmgnk{>;UOrA(-gWA-bUH@>JDhxL|Y3-gNq`iH5^EJAE;?#FxFHOgxB~3 z8zMBxdEJ`G9eyq|;E~M4$qDL!q(Pd|=-IUuz4BT^eJHoC#5?R|Ul*gVOHwVCH`u5- z*2nHqd~~y^ntiEw2w|?f%tZ(KmCpnc%26N$P8@}9Ex47)^6Nsf1YzZhf@JXpaYl6oIWk+R+a}g0*n-q>tnh4G`$kBWI_}v4$;yYb7$p2-e*?eb z;<()l1b7zqccceW62c4Gh#|w`h<1MYi2AOq3ejUpjMy^l>RxdE_vXg#)6k^*OhBLc z35yal98Z*%>DScK3Y2rqWQy{$;(h9uTpLo)oAT<2 z5(=`SPw*T?!mq`FVzyah*Q5#YGhT^iW%QiQ6LYuAb&RVQ^5_+$%g=T>nlcptIuvmR zC=m@qaV*%Z*j1QuRS*^7DUH(M5}h%abULkagm0iesY*XS2%08}ZKQ3A;@KuPsl195 ze2Vr1w?d?X7c!hV$o%<7ii+JDG=P53xhVGKG8N21LLBNTx6o!3T;_NUVl)0(Kq=M2 z5Iu6^KqG55a7zS;2 zzB4?$-(`&+C%*In4hdDbYh`Khjb-zFC-JNYcoB>>1!{TUc3_`25G{FFg+s#Ip~j>< zXYK;%D6s$Yl{zB^iga;?wP@#tkw6gK{KV~%T)QLyV=I4W_n%3fi^p9F96lc=yhLD2 zu79LmEnEBef_EFCpBmEvLSIX_6qdZgHHKpT4@-4xNwa_ZIU+N$1i{ zg4}{^IK8hKUK?sMND1^NzsS7DEu!V`_N2X>!06=nl32Zo|BIc-Pw@osPgRN?)-xRw z8UxU$*%#K=mYP%Q5pHZ=!YmZ;22BNZj%>K3*2`Lv6+3ef2=t#Yn;jr6A?#}trZBuC zb_<`uVQ(S%{kBg=mX?HxWYV2Zwz29X)vknsz)P2-UCz$!p_KaOLme3uy*RcfbpPp$ zWzZH+-x9JBV7~ioCI0d*tODFW;xQB8I<&$aM|W6i8VvR0q-5|- zTHfSKlk&7N4B=S14Z#0#fLKYI#T5)LD$?n(3+vVxU7?(Ky&JMQbe2u6;z zgUj2C-2$Ra^wN58G5wT{x034>>I|Taz=#+m3>%#L(`g+|5Y>r;kfbydzr-h49VvZy z@=`bNkOaps>7^WNUM+A5o_S)vOk++$HE5uUWn>oI)}df%p)=}d9`@5*4%Bae@`Cw*LZ>D@+OhlsTi{IB}2Ntss#`c_<$pOF}tIhOhS1f$$Zw;$9sQdwS2 z3#0t{L3cDYNtMV{LHtYy_SC0TGiZ&XsTD0gxqZA46xi`*>rSwx85cU+EeFZmR8 z!J4U_@OR7v&b8XlVbgvv?n z`pxRwIs~Uo+{2$_`g7zz#}}=kt6JSEsyNtMf6Rzn{Ok%4N*?eV;UeTQE1^0%WPA%9 z7H1~48;X-^jq!;JcAc)Y6m&N04d}YIktBtvVF`}Dq>|uRh4g&nQ}%(1Y~FZUnRc5T zSqFe7_Glu_c2p3d_Wqrpqyi#go7}*MsFmwQq}sky1dOjkHgG#V z$cfSmYA#fsRMfn+PdvWCjOgJ#UIBuV5qMBdB0WesPWKlOyhkMNFvLX)xii6<6t1(q zU5ZH{wUnXVIzICNInYv+-+48>9^J?; z3Xif+|0qrPTVoOWI!vl3ol+%Q_;B__%%j-{&2(#NJ@Z|n(mVQocj)@74;a3p8ssgM z{Y(b6aG)T1iguqs_v0(c80&UIDo%v&_t&Y?(a+pE5WfFgVax=KoWck{7A;eP=h26E zQ`1hGo!PnrlD(}067FQE<(hGSu%?<@;fl}Kn%Etup6}Y$CiMT7%hl-iapYN+YQTo%?i}FtJ>oL}x-VIQ&82`6>vpa7jU&ciJ zXF1`zQswggzxN|v#Rnibcfo)2Ay8#=zjitSx{Zm{Cpf=5@%O*L_0kyt4zNDblnY)M zEz}_e&|3PXNe{5Dv8F392KcdD>{1zkOCVAWFLyU?H&m5(-dYCgMb@}WZTf=CPTRNb z2WLJcfaI^Z?2_#nwg6uVIwvmt1LV|>PY|iv&c42TwCD?1{Q+bO3h7L0*Z5WyL=*4W zFc!!R=TrrIk?%t4Hg$5<8b`HHCQ#_Kc#qq;tz2tUcWra!7y@}hlqA^|O$OGhY^?!H z1-zULXz$GR4|<`qSz0l!d-+2vw@_u-!S!r}4bnij3N#|1)HR+Ig1ZJj??FAzu>X-_ z_?p!&{klZsK^l_}VS819N-l{WdS0+WH1BYsq#o(7+zO~F)g}nXpD>iX5AMHQf2$?i zRFp>ZDIJc6aR zMA%^90qOH15Y!Pc9QKM4Gg|1^=(*ypdK^U9o)IA7wMK2oRUJc;!^#gptRN>i>x_mC zkhg$op%x;QOc--gEVn-bLGaKV~0)D3*tY(~Z+y-~iq6 zM?WSK76c4*BYLttpU_iZ!#1B^&I)|VE$$>Je2}~ZZ{}OysLu_0l+S4iNLA|0k8*yC@@pe8rL!H`fXn7pjf@Glg?;z|z1@;czt5kldo%9eYOjfMYkOoJ8`MWml=Q-yynf6k` z6&K~!C}-qmW13Jg9A&y!I5zruEuk~8Y?XWDJy#*!*^}lz^mHH2QNBLJ@bzrrdF>FY z*>Fd1)qGC1B0+U>7Vb~-FHAuyQnI4t5lIVTE)EbAm-A6wb$nkz>wJ9)@$15V@ zz3=y;Q)JTCYGVS|PzPqX0xFt7UV~e-9 z4c*5Gna@y)Tp==D+VlY_RFbu9>Pw~k>#7GwwMya3YEVfJ9ha4H^D$rc7ku8Jr!1v74aZzp`LfaJaAmo7|pjhfDg0GgZ`Tf!1W zxm0|qmpJ|g%b~y4XL@s9y5&X!7vN(e?z0-OTYJ!aUa4+BgGj}gz|vby1@d)qEx1bc z3>w6fNNN@F5F=Glq5I5lO ze0nZTsPDSI-_~!zWY^u$G{ShkzIm6xZ1pqq5_b|Q8c`01X;(+C)?taBr@UeGnT0gl z$-J5=Tb=FGeo>}$#iQ}k$YU+Vw9bB)BI}m6$1Z9rPd|s47)Cu!&m}R^cvesmntl$+ zHzS*Nf-oqsWyG_HJlfc#5wQA#Bay~sz@Tt1khfW@RJvC`fIN23$&S4!2FN^0DGB6o9EUBbCT9;F^-q`tCqPEZABPBw(JM7-*YzvQjRkrPjq+=r~; zPc%K{B`gT4w9YerjJ?znTfgLE5*TLdjNFmOO;fq&4)Zqtm!7K<&h-0^p2e?ySFh5I znDyE|^u4gUT+0;yiPskj5Bh5(`5buT|@*3T;sH!7AyA`ijxyOaS%^GO(Cn+owpH< zkC(HKaKDf!U^%d$Ef5#=72OX$knK&ZzVoAj^uQyWXKj=@%Q+Gs-5tXbjlf(EXH1Fe zzK!S$+*|T&`-==*5ZCnOMf`uZzzKokhGYIIrk?W2?a)B9DGeIfbU_IucmIjyXMe)C zCzj<2z5t2RgPCXVJ$VUq$moUwl2e+O3TqY2JT|mqLEG_P*xt*Pu)lAf7yrV!Yx-s& z?C#OYFRqGMC!8bQmCAowhiMnrstaAcPx;lYrJ_AUXy<4O<17;e|hT3a;4i2|U`5dCMAv<%mN?mO`olh`f@aM0M2Wz~(kbP#93 zhwMD3N3j~yQFO;O5{;}^p~4S$1>fFm;Tj03Y)v_C)IwYBQ>fxQcNg&EJt^m2*(RtR z4~^PeK=H0^r50j|Gyje~t6DxgPcNm@>Yu2lla=AL;#4HZYC7jnsG<8`e6TP5K@?_7 zULe}n9Z&42m(QlAH~6D$iXZB*tPkj(JA8I5Jzf_Wz1T-=_Vh3u9k%#^-8g!;f_LA)0Yp} zz1hWKW)<&9TdduBcy2N{M?ckU_C40C3VBE*Y#1Y4h?YgN$;dhPoO!j-aC-g#`!Nm2 zKvwO8bps_q3yX5BURtlA-lTCUPBH3(DI&BAadxlPOb$DZQWLE&3)pBHXR9%a63 zh8~vyWQ&sU7-4EOVqeT8!`5}|des}1NYL+^z&+0q&C_~=aORf!b|iQyXqR3k6R<2B zz{%04i!md5y}}}B+DpB5dSa zLqujSgNLjCh}OGczG*5vD&|A3s*Bt?#psKHWvdp;uNI4T@G$jW2shO?9w2fx2m=m* zg%;GVqV@vU{Q&~hkA`S$5Zmy?fWq!U>=x9K1X>sS`x;dq9xPiV#tE4$Nc>yGh(>{; z#cBCwRZF#(r;7NhSzWp*%_a24vXLzo`W8Ioh;s^o`4*OvdbmxE$yP6*l6AmrQLmHW zP^kC4{ODOhQ?L@(aD2XR^0;`y8D-S#r&HP`Sn^Bi&KmqQW?RuxtV};IT`DnL1h#t5 zh)M0s^5Eid0QP=%H6>1_%&(uc8z|nsiW_(yJi7cL-HLlp;+* z1f)w>5fu;+P*j=Hg)f@a$yFo_Fz?Pqs=8VJJ;hwxbcR6b_JH$9&OFgk|aJZ+>(9lqGkbS zTVmdxT7}H&uLLAsn$cP(`!roB-JnBYCT5*HeP)^|EcwDOet?o6)sNU+Y~QFp+!A4B zYVRQ_P{|zLidOm^@x1Ef=TmPVyPz2HB!UlD4uYM34SVOcD4&Y{RtT!^R`7hSskSPjx zvf8{Dl`+0AR?g?833|j|Ig+8MYewLGdBqlv#eK#9^_$>E-60Eb86lv(W?62ISyIw;(nMjZ8GwQv13rijEe9SvBG;BQI z%G=`)2WDm~T}y?I6kai>&J17lnBg_Aaxh0#{e?3}XU3UdX8+TdC-hs_Ep*&wT9&h6Wl}AhpX<6cBaS{XV$Sf5 z`qj@Y%`wJ}?IVK>8nZU4o~flj>!;IlhHk>)N`@r9HX;>i{VsbXMNtPeW;~w?d21u; zv|rYVn=+&S@Ez%!^xe&s3%*t(k=NAge~DN%H;UVii)rh=dTi|Eh@uXOX*LOA*m_IT zr@!Zt+|(MJ<>$tFbWv05m0(z6o{}&x@pn67i|k65OvF%B!qMsLWt;yqqh;eANR`MQ z5e&CB0X!%@x@WK@EoZ3#iy_B)91Xigxq}vnqddkW$LNW?U%US96{fE8Iv>+?X*g`> zxc-cL5BWzYzF*B;QrVq~mHi_vXw*cR^I+(w;e z_Mw{JG_+Qg&SKdqPY15}8`+V=O0e8Yp>%_IjX|?mC&_@mojLuAX1dLHnfS<`Cpi<# z#@TqSD;x&Tvq@7l=%XT@(x0`gn#5)T?#9!#Tg6`gs1Iv(2u~{-ar8Kay%}-a&Rd)M z3=zc067by}%f14T<_C7aOUidjn{R@&BJfj6XnEr}FpH80ngkv(TGbA}!l@PngT<9c zP2a5b>|6VMo`7h?QBBZW1W`N`P$a>PqGk1@4dI)S#j7(kEz#*|F;JclJJDhtl~*74 zsiaoe1^sFOBUb&$0EkejQQVjGGV#$GZz8uC+;j;J9e&i}`l|Mb^Vd2+cKiA41LJ|r z_$vyagvZ(aOXG*yKN&9mV>R$+RNi%igU=u-Ezh~>*)gDBDIWbwJ zlu!ud<>Q< zMDQhn56xqrZkD_tE|9SlhEBPkH|02D6{Zqf>E%2gCbGS!P9kg(K`9zDi%D{6*h#FH^j%m0 z2ZPu{YL-p_^!XG*Oi(g=BOrl1kH9+;67r+sCur8Q@!H@7htfg8Uac*Dis0#O4fk8X z*7&Ddr8XxA#&#L3bogjzgY$Iul zXX~ZM?~FgC3N7QrzsPdy^sT`|1Znz2BS1SFwt_R@&ihbzX(h!-_=~X` zlRkFx^m@2zAsE1l=VVlmuIb#d82#Gh3D^veif|!TcOZ*P97%B#U*pT)>Oc9Y`m1d1 zdL|8MYG!)DyOQHas)N*D3)2^{Q;5NC~7IE5sA^DMCTlrY&>E&MbP_ zFB1n>7_q%erW!LnS(^_?_;<)2G<%w4Tuh4LKut6`6ZtZM+9Hj-+_jGD8JMD9 z`8&i=7bKGOQ*S*J@$YQCgKgiWpIoferZ+iW+T_NuKo+^4s&;C7&6PGG@AZqw>AZgs zTFRXvwFUFBlNLKlE9^ZxT5($JXR2t$kBZvo$URqc=yUl!5A(L1mhpH4N+7noWw?K7 zletOb+tCt3$7ywgiYxB$>`fe_e8ej4n&t;n^-mLIvDrWR)`xS#eCT_M3(zn&qqzx9 zeg6N$=n3u*fQ~qc*CZQ3YiCV!tL0*EIXZx-PJw=>Z2mPsRo81BcZkE?qRKd9>*4qX z=)8>2XHq-$JJmHqJkhb~*o?uCLrba5SJgBY(nM_ZcN3({3U2gJu02|a*v4*|y?ahQ z5ZV?w<#6SZdY`%L89ujBHHISasAd_kvDg2$;fNs{rc;Q$lE$&j+B%_`N9TCV&+8EQ z$o-PQ7TLDmAYL<1ufszjs*6#aJ;sL&XQGfT;hag)S#CX_S8FB2r=ix@&H@vba}v-Q zyf;L~j9Xt1G>iMPEab;img109slyBzPPr{D$KdK9lYr$5e&4J+S3>1qxB;I_8&7Xh zV0Ja}YcG@(kZbE*uP#>|V@S+tW1)^JG-ZzQH*quxUiRPLjq4u`9p+^G5+HHK>8mTp z-02*WJry>5@gL{}79q~{7X*==kQJ;Z6nT_N4R9m)7v>Nt+TEO)J~ZKVmPe)=-^Jav z*))EI2tArusP3}OMgK}RhHyTI&IJ?bHRmt=vhVi3buwZ-%SMJ-e`|RaoH9C5761y0 zb5fPOcBWcDeK7pR?~SVREw=Z=F@ajcjG-I&ysmW8<3uVR$plpa*Dtww%R_bkouX^+ z+-~d~`KG$c#PXLYSH#y4RNG$p8)l^_R~uq=2A%^rypH0G*KojkToX#peNfPEHpSFD zsXYh$>w!e4_U|;(%NXwH{%03b0ks1mCxRSKVcjqEkBI8IZ==f^#VQr3u`UR`*%imi{-(4b%h=aG_PNs>&cHqwAO7vs1O5ii@>^`ytt2;9<6=2Wv7Nl>urS|q8-ZCF z$?~atEfFXGaK>oJpwCo^z+#I;d_Mp6diyTriCeOfLaRm&^n6MqTH%cmd3eV?w>*{K zpD)oh9cDxw{Q_IB3{w9c0I`*v7CL7hS%P@?-HmF+Kl5&@DUp_Y*xLOU9m|^LG*X!< ztyf}fq62!~=rPw(L;YC^*FY-~9EZ$^qBExxU_&F~H)YLD938rVc5ze!?zv@s3!z>u!<(9SLz%ei*!E-VjnalV})sQ27=i7Ybld%m!sk8N+1S z>$FSv3zUhQBgzxG|J+CYhAc7Dnu`!eNyVeA{E>livzs%??28Z84D#iff8^pHr9T8~{ZK3JnfD^{ zo?ZuY9HV;zO@U3Q!5YG?_AOr)1hyFqm=Si4Q6^O9?~{5#I@Ny#?ZjG1K*-YK7L!|R zw(TvaDWnM__jr{CJnY=>#;oT&Ucw1~Jb%y|@|mY7Wn-_?xA9Lkr*SM+tIAmY4(>MZ zU9GWN?g0F+n~Qs71nI!R*abzupyCQT=7~M^8U@uI?@nW3e{n2b0P3x~Fa!(Dk-M>3 zP8^RX^7Q*de!G^pS^tldRZOW6Lu^4~;PB_Ww`bHfBUo3a0#cnSl>MBkK8n9a>h4kC z)gL;C?0;MBi}vtrs`46Nn(=f!vvE1LPQ)*!dA<7JxCHC)^kdvMiXw*6K^k&llUJ(R zs#{gL);#Z*j$g}&Rhc|XBR7Uu^E$EzlVT)5aVrQQ5G=wDcg17~>LW}@=l)nK53MBi zz()#`pZFmxQGBK?BqIoB`X9VVjY%h6PeyIBO&sciM%%o_njS}@+0HlK&U6~aI(ftzmCtpVy|x0cC}`utPRBp3m0IzK zf3SfuN1oJ?wjjdISt|>J<;K>9O;=C8J7sXr08ljhx0}B$OUqCS=@<}{W2<{A?QIHs z&DZ(b31d{`e7C-AdYHX4CwyqaJOfeO&s;xN_Y9;s7?^}M0Wjs$_Yqc@2F=d3A9pls zsKa!(qbp7`XLt5Nt-*SV>2~KpLkp8D#50~l#0(z)O!xYC znLpKgnwr7QtGk^Liy=Sx)5TjOD+z1&AD{nFzM9piH1jNu3HF&cu>ji}bY(J{yqrUt zF1*pD^L4`(|AgO^u)L&17RAlc-`nqgCue_4AA%#|G4%p+d084tkF~u{+O)10;3hC~ zqZ1WRR=ddxE=ZGR_sAGV|8&q>AIYDr7;9*;H?h$xPVf$}5uQaXMR|vJmCV$Uz7z$} zv~vbB*gS#k(re58=a^TvV4DIBU)KS!Q=Weh9^`K<}G&nyPj3{?L^>RDN`F zyf9TKS3`EkTvQJg8h0$<_Nt`Nu@A7kJaXes86;wE&(M*jOyV4cZBF6sbyYYH8$A_T zsascgnWM5yNLec{`|h6ImuL}$cw#&l6F*7yZu@FN1Cke>vZ|ksuX4Zv#Y3Hiae3%Hh98;jWw2m!*G!387 z-Ft_z_;t@0uNef`R@S1Qgv!X)v_0OC)`fZ5Y7`1f+*W&QM$3F|n|~yQ1cm{e3O@2R z=3C_h0!l}k&^^$NPC73es1XvTr;N|*o~=B;K(mH;^xSY07pq)Lx|$k#uw(f0(>zKb zxjQpth&`=$G_qp9bi>%;*cf*34KKHve!que(3;w`O3hi130ZtWmz1wY2m3cnK-4Tj z!5cj%V3w0+$Bj!2xN_19=Je!J?7jqgb>J^F)?x+j~4$>@CVRGLK*ZtNB^g2mtP zC4-B*0K!(nir zb~)#-G%*&Ve@75=87feqz@+ZMBa_8B6L|@d5!zZ%D2=NFFIoO5b$-4-JC)osd+u2G z?Dvyv2I+JFq40LaSuRTZc9bZ=kyXqm=~M@$+;OH?S+lZkZ^{0OXJ*DZJ(Th$W#~Iq z(={1o>{0mLV%#MG~*jD-g)24h^^t5GOFzj!D<*!B)tmy&K1y%t*05Q%|{_LPy zFlFS$IaQ9lA?fuW7+PLIr7)-}i%}e__AR{ve570G3+(T@FctOHBi)`CDOs-EPOMQd zV`s;^)?9&V%K3KGa-}q88KJw!o?wAceC7JRMFV)i4VXF;wkR`CZBwod+b8`^xh%miX!NK>8Kkm3cP>&>Qmxyz17Qyy+@iaa3 z6k1E4i;(rL!q>LI(=O}|^Dzm;Vs(-7*F#y>wSEHV5~{?bZ6zkJl5sfa(ubjRSH4^( zAe9lR8XS%k2V0v00~ytEWOp{F%-X_rMG?p`&GXEf^t>g?%pIWM(h{^OGExu838iQy zcl+!Y>6aair4D`n?b~*ekTIEv9?7GRIUx7ekQ`<3l>_?lVNMNniSnlYFCV0XC#gR^ zdbT}?`cUutz7pEpb6-9cXkYO|Wm>mm3}An|6E^C2h=Zr~cks+2+!Bx8Tx56Rm_7Jf zUQ7?&BxyQ)!+Dg*Z+takYgV|%Pe5G}Cnqi@X)YCOO3P=wU1qwe!526eecv#OAQ`iB z&79#OX}0DeI`OA2b)MHg(b)_37z28xVv~k55hx&xgyr^&AF9$I`4U#HRgrgSQB{?| zZ`7mnnbZqTPMcIYbiekkmusbHqkm}Zvwra|&lH&al}Iscxe!;V!7^XT>`<+0-&=cy+H?W^v^NJ#<+S zi*{y)OS#U_tbEnPvH^hKVg4c;$)J&5z~V)R=YaCD(RunUF5jjc#U}IduQ4X3b^u@I z!oa3=a~aJJgPs-#w9xdz;DJ~$dh4oG06#igmEl!!hDgn@RDKBTW9f|hi{z0fMscX+ z;ZeY$k*+Lep;|AC?PN_cn2vE9C+MuDVl>AEO}gW1MRAE=siWKP_w?}^hKkasscbV! zBK_}tJO?TSOYjlA*OduG4VuE8N0AIU)>O)FTI&Eo)1Z zGe$Xw-^ZDQcXx8LnwHpqI2lAoTO_Ei(z-iNKh;)m2B3{i_=sXP!!?;uDi|=XXeDbi z%&~Ogs1keiY^Zxr8%d-rGX4Y*kVmGyGp>W|fIBWF1X~=WEOJuU}2F zL+V|+Ay-AkQyc7M8O)1dzHm{~G`n>-q78 z6UC_ZOiWJa!Mzvuv@d#3RWhdc&UTzZ?ZiK1FrQIh^M3LCg07bC<>~%Cian%cy4W03 zN>eU_Kym%{uC%9z1pUCwND?8uYKi!N0T{xL4MALxLg=)7+QNC^Zd0_$kid8JuQxn_cG(u zipcahr@K@*#)>2XU8yBN3FFgOdDKI$FZ8UqqH2l(aA%tz3ttPM>r8{?p(u4d#d(G0 z27FD5veLxpe6fW-J0~OI)dW1fNr9~}&FrGBE_dxY4qdhQh3b+el?$1@3jKugEZTXV z>tz81`qvz-&*Fnn!h1*;=4>=#M;qFe5j44YrHtii9a^p)rvyznF2jr@4P+<=mfDx@O@o%y=-PLm+*B*^J({Kj!ZqP_t#cFfMT`FY5Q4rNU+eOFyAg& zK?8ctw%i;F!N${A2caS5ddDY=TosCL5D$so2Lsl26n)lq=7tZ5%B>NRk(P(H0(6GbOkVVpB&b9etGK!NlUzl`8+|ERf5KL`DWYAqghDZ zdGGZ^Htb2P#Rchev^NWy9T6~P9{b&KN2AnhhF(-})`J=B{Ms2nZ5&BUJl0^-CZ20R zfy@U02{|aqLe%lkzym1s_rwMqki28~PcdDU0Jc+dG18z0(x~P9_is7V{RnFvV!1M7gFu9SE->j;D( zbXU);UI+!{eZ$pKaG-R~#=gMF{Nm4_soF|ql4c{%1L`jrsgTWO$BL+X_}81ZB>%(z z3?(Pc+^ur3VtJf0QL<$jaiilq5C%`$iPo=}A!T6bh#MJro(fg99qbPQ0`F2gCriWv#R+=$S!MRQ9JyE-mK=7Ba z-K%fB4z#^Yd#I-;XQ(f<{fMUs>xm4Sl7bv0;{X+tGG4LEJuwj+E`VKr+(PnQDR7}7;~KmFl;nRf!(C)qYM%}lB9iSy8x&+i ztL=#AF*r)LfL(y&7Cv`s3wrJcuMz~E`1`C~l@ggB@TZfKTn`3V%M3ipB|qvBE?={zOj#UgrFd8;6Ud9XQ3jR*63mxfm^=RrrcJ zwF1yy6M!}FhJni5XN<=HsP_MIj@1NiGk{Pn@}^k$WabAplG;i)bNOWsqu7cr$%DEa z7a!p}fLbKcyIE1%@crA8g3Xmb*`T0VJ10-;V@`jy#pU=G*fTkj#?U1tku+3rOG>m* z-T<9}F@QZ-`E8f31_%$#Melq>pPMgq3Dk#-NU`^A+Z>zBr17co{vxxn2XNfjvgv=R zeJ+;Kyx>r1)%PQZ$9E`#Zwyad!#Jc!C|d};R^sOsNo8D48)S!Iu}1<|jC^hS08}(8 z+ax0vc;|Lu!Xmbe;-QN$Vy*WfrMqt0!!%?)AsF zcb$DKpvCPG&im z-kxR7_f{b6EvZ*Fzy=v=3|a#_8=1JkQFtbQe8oixN@?8ch}nB$7II2iXJl&r$>Zef zyoQOE$(r3P$jchPtv^0>d{U-VofT)K*-vR^4{C~(K1i23&`y76^v)BA4AEtc>*q$l z3HoHQj>s{lzRHj`!#NyzPqiSB}mqvD?`wwPoA=F(g~5@H;yFU z9?t*4m5Z;rv4$#?&)0Q8U0>3QRHfe)yyBi}j8W^OTm!(+D&&AJ8~GHKxPw%0tt^xv zIW*q@^IX0Nv|~_FfRw^@S>m#0bL)3J(0tF;=NW5nJavm;=U51u3#1-P|(0p<--B#rD zl)r;34y=OTDj2#4(C&ymE_@EcYeW71STEBE6KJT zBcaJ2Zxi4vpiyPorQ@C&IUeC$$ESZ)w}Xt2c5tSeTDAY0I1A8n#V{_ z3?M?5QRdj!_3{r*N)*Ol>UVottwi&zc24~?HL}pb-a$@cl;h!*KFn@-J+^bvD-OC6 z{IHI*JI)NxKHCrA6{qI(SbZ)d<+c-(l=F>WGAG$!`Ox3JKVz4xv*)ucFO|0FRoo78rlEutrzZU!HXyk{n^MHB~Q!b}EPnn(UWP_MxY0gS>Fh{%*|Hzy5@I zL+y_{&uI2{LVrl>4(!emNfQ#0vt`B}d5I@Q8Cx!qd%0zCk}u6nCL88|lr8b7Z-?m{ zq1|zwqne`aO0fPAYpdSBHENJou*Yg+;(cw0qF1pJG_VAFnHh=8XQ~)vAD{8|!4v41 z1SE8EKgnM0Go>}>=b0X{owo_eMyVm>`WeyXZ&}XDViqsEno0 zbq43Ibh=kaTh}!aZ{vzHvJfvRw|f!T%71h#ufoM`*xvcj3X2D@Ii9)f8ZTWims~?$ z$;oM!Ohz}TuOrCw4nD?qy}i)620TsAgQJ)Z$nsA#DA**ppycRyqr!D#AX=@IG^QNW=zHov)&Ff%49}+8cc`M+TwmkFH|%;p&Q$Fn zkfD|=NnK2IX`3e>`nrVxZ3=zcFq)#jLf`4$t2mN)lN72#>AC)Q78!j3$m1CS<;WLL z%eDniz9hb=g(W3@18}eFe^N(Z^

(=WtOZ#KGW;fsu0HSHgqK%4LP{>!!1;YY}6a zCU93_%TX4E`MZhs1QFSV4`)T1VQEsaHmebl?W3R%iFKx#74PRJlc?^oxw?AN_7vMQ z3774f7VN_>8+|YpbRM|03KndZSaz0;{avn)FoR}=%?k;WAwEbUtr}dh1#XYr@b3iB z;3h+AS$BryA=`z)E+&(Rx7QWZsDDdQN+b}F81+fmGWS*X%l+#&v`RWR8Pho=vxYw7 zor^{@!dJZ_Ig$2+q00$b^R{!Na0;6Lx-rWp`kh}kXm*It@H(>fCR;;S&!+_F z<%;*g-uUb%y9hexutjGc-SfmF9i&QA84-};`7gPIa?t}F>IDA|MuXQgSwVE3-A$aA zDe+k647~XZIzh^}+I&6*B}NeW(L9T?`f;O0W&~nSx;9w*!<~IO=NFge#OCZoK6EP3 zrzVI{K7{2M+=>(!9Jg72L={Bv*MX3p=49 zHAAH`7RiHj0;NKcggolJ6D2w zaHtCt$X%$8)1DkKZS6$anj=3aze;^CfkP%Cs)LIvja z@h>PaZ3#+FzZ~%4P97M|U%7t7_FHnij>TngCj_D?hGin9Z}y63mCx#$DGX-|5Nd=? zAFVP!i z>q<>qU9#RlTB)ve%Y-bRO*LJ1j1cQ!rJ6jeA6K-hc2$9XzPz~~tkN52crA})Ax0#)&sTk< zMyq&ts+H_{Ej}tZvs9M#ukIOL-FQ>ZG#8`OwD?sSS-5)PM#rpA zQs_TQyX4^26#Fak^jRG|2}wS=ClPt$uX&CdMSH;5kakworl@D1$8GZ}bO)=C;g}Jh zqHNZTnKUSt8S6dI1MDus|1_Kaj(V~jhtG8B20oc2(3L9t0Tl4N0DKZ0)&1(b)xuXs z-F>8q7Wez7SM`7zvU>{e;{Yz@S9ULcTM+j~ zF24_#8Z`RBRnPYMubakVhxkAKUiq=oG}QmP#sGHG>fhHjB<(ZeXNZ?1AWpFU>ymZ* cvwQweI52R(7j*6^CjpnXh5@cg%|7aX0e^o#-T(jq literal 0 HcmV?d00001 From 2818da21b7a445c1e050a7080d0fedc8007b295a Mon Sep 17 00:00:00 2001 From: Jitendra Bhurat Date: Wed, 23 Nov 2016 15:08:25 -0500 Subject: [PATCH 2/5] Added a sample POD spec and added kubenet requirements --- docs/getting-started-guides/windows/README.md | 38 +++++++++++++++++-- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/windows/README.md b/docs/getting-started-guides/windows/README.md index 22943f1a32a27..7b49bb19aec01 100644 --- a/docs/getting-started-guides/windows/README.md +++ b/docs/getting-started-guides/windows/README.md @@ -9,8 +9,9 @@ Windows Server Containers will be supported on Kubernetes as an alpha feature. K With the alpha release, we support Windows Server Containers for Kubernetes using the following: 1. Kubernetes control plane running on existing Linux infrastructure (version 1.5 or later) -2. Windows Server 2016 (RTM version 10.0.14393 or later) -3. Docker Version 1.12.2-cs2-ws-beta or later +2. Kubenet network plugin setup on the Linux nodes +3. Windows Server 2016 (RTM version 10.0.14393 or later) +4. Docker Version 1.12.2-cs2-ws-beta or later ## Networking Network is achieved using L3 routing. Because third-party networking plugins (e.g. flannel, calico, etc) don’t natively work on Windows Server, we relied on existing technology that is built into the Windows and Linux operating systems. In this L3 networking approach, we chose a /16 subnet for the cluster nodes, and we assign a /24 subnet to each worker node. All pods on a given worker node will be connected to the /24 subnet. This allows pods on the same node to communicate with each other. In order to enable networking between pods running on different nodes, routing features that are built into Windows Server 2016 and Linux are used. @@ -124,8 +125,39 @@ Run the following in a PowerShell window with administrative privileges. Be awar 2. Run *kube-proxy* executable using the below command `.\proxy.exe --v=3 --proxy-mode=userspace --hostname-override= --master= --bind-address=` +## Scheduling PODs on Windows +As Kubernetes control plane currently runs on Linux, the resulting cluster will have both Linux and Windows nodes. TO schedule PODs on Windows, `nodeSelector` constraint has to be set with label `beta.kubernetes.io/os` having value `windows` as shown in the example below, +``` +{ + "apiVersion": "v1", + "kind": "Pod", + "metadata": { + "name": "iis", + "labels": { + "name": "iis" + } + }, + "spec": { + "containers": [ + { + "name": "iis", + "image": "microsoft/iis", + "ports": [ + { + "containerPort": 80 + } + ] + } + ], + "nodeSelector": { + "beta.kubernetes.io/os": "windows" + } + } +} +``` + ## Known Limitations: 1. There is no network namespace in Windows and as a result currently only one container per pod is supported 2. Secrets currently do not work because of a bug in Windows Server Containers described [here](https://github.com/docker/docker/issues/28401) 3. ConfigMaps have not been implemented yet. -4. `kube-proxy` implementation uses `netsh portproxy` and as it only supports TCP, DNS currently works only if the client retries DNS query using TCP \ No newline at end of file +4. `kube-proxy` implementation uses `netsh portproxy` and as it only supports TCP, DNS currently works only if the client retries DNS query using TCP From c6b260489b5578a66bb51e113802a853fecb1889 Mon Sep 17 00:00:00 2001 From: Jitendra Bhurat Date: Fri, 25 Nov 2016 19:09:36 -0500 Subject: [PATCH 3/5] Updated based on review comments --- docs/getting-started-guides/windows/README.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/docs/getting-started-guides/windows/README.md b/docs/getting-started-guides/windows/README.md index 7b49bb19aec01..29c01348c8bf0 100644 --- a/docs/getting-started-guides/windows/README.md +++ b/docs/getting-started-guides/windows/README.md @@ -1,12 +1,8 @@ # Running Windows Server Containers using Kubernetes -Microsoft, in collaboration with Docker, is introducing a new feature called Windows Server Containers with Windows Server 2016, which enables Docker containers to run on Windows. We have enhanced Kubernetes to support Windows Server Containers (and Windows Server 2016 as a container host node). - -Windows Server and .NET still account for a significant portion of the workloads in the enterprise, and with this work, customers will be able to run Windows-based and .NET-based applications inside containers on Kubernetes, making Kubernetes the first fully functional cross-platform cluster manager. Services that span Windows-based and Linux-based containers are now a reality. - -Windows Server Containers will be supported on Kubernetes as an alpha feature. Kubernetes control plane (API Server, Scheduler, Controller Manager etc) continue to run on Linux, while the kubelet and kube-proxy can be run on Windows Server. +Windows Server Containers are supported on Kubernetes as an alpha feature. Kubernetes control plane (API Server, Scheduler, Controller Manager etc) will continue to run on Linux, while the kubelet and kube-proxy can be run on Windows Server. ## Prerequisites -With the alpha release, we support Windows Server Containers for Kubernetes using the following: +With the alpha release, Windows Server Containers for Kubernetes is supported using the following: 1. Kubernetes control plane running on existing Linux infrastructure (version 1.5 or later) 2. Kubenet network plugin setup on the Linux nodes @@ -14,15 +10,15 @@ With the alpha release, we support Windows Server Containers for Kubernetes usin 4. Docker Version 1.12.2-cs2-ws-beta or later ## Networking -Network is achieved using L3 routing. Because third-party networking plugins (e.g. flannel, calico, etc) don’t natively work on Windows Server, we relied on existing technology that is built into the Windows and Linux operating systems. In this L3 networking approach, we chose a /16 subnet for the cluster nodes, and we assign a /24 subnet to each worker node. All pods on a given worker node will be connected to the /24 subnet. This allows pods on the same node to communicate with each other. In order to enable networking between pods running on different nodes, routing features that are built into Windows Server 2016 and Linux are used. +Network is achieved using L3 routing. Because third-party networking plugins (e.g. flannel, calico, etc) don’t natively work on Windows Server, existing technology that is built into the Windows and Linux operating systems is relied on. In this L3 networking approach, a /16 subnet is chosen for the cluster nodes, and a /24 subnet is assigned to each worker node. All pods on a given worker node will be connected to the /24 subnet. This allows pods on the same node to communicate with each other. In order to enable networking between pods running on different nodes, routing features that are built into Windows Server 2016 and Linux are used. ### Linux The above networking approach is already supported on Linux using a bridge interface, which essentially creates a private network local to the node. Similar to the Windows side, routes to all other pod CIDRs must be created in order to send packets via the “public” NIC. ### Windows -To support networking requirements, we will need the following configuration on each Kubernetes Windows Server node: +Each Window Server node should have the following configuration: -1. Two NICs (virtual networking adapters) are required on each Windows Server node - The two Windows container networking modes we use (transparent and L2 bridge) use an external Hyper-V virtual switch. This means that one of the NICs is entirely allocated to the bridge, creating the need for the second NIC. +1. Two NICs (virtual networking adapters) are required on each Windows Server node - The two Windows container networking modes of interest (transparent and L2 bridge) use an external Hyper-V virtual switch. This means that one of the NICs is entirely allocated to the bridge, creating the need for the second NIC. 2. Transparent container network created - This is a manual configuration step and is shown in **_Route Setup_** section below 3. RRAS (Routing) Windows feature enabled - Allows routing between NICs on the box, and also “captures” packets that have the destination IP of a POD running on the node. To enable, open “Server Manager”. Click on “Roles”, “Add Roles”. Click “Next”. Select “Network Policy and Access Services”. Click on “Routing and Remote Access Service” and the underlying checkboxes 4. Routes defined pointing to the other pod CIDRs via the “public” NIC - These routes are added to the built-in routing table as shown in **_Route Setup_** section below @@ -102,7 +98,7 @@ route add 192.168.1.0 mask 255.255.255.0 192.168.1.1 if Date: Fri, 2 Dec 2016 14:12:13 -0500 Subject: [PATCH 4/5] Cosmetic changes as per review --- docs/getting-started-guides/windows/README.md | 45 +++++++++---------- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/docs/getting-started-guides/windows/README.md b/docs/getting-started-guides/windows/README.md index 29c01348c8bf0..ec3dfffd4d4ef 100644 --- a/docs/getting-started-guides/windows/README.md +++ b/docs/getting-started-guides/windows/README.md @@ -1,8 +1,10 @@ # Running Windows Server Containers using Kubernetes -Windows Server Containers are supported on Kubernetes as an alpha feature. Kubernetes control plane (API Server, Scheduler, Controller Manager etc) will continue to run on Linux, while the kubelet and kube-proxy can be run on Windows Server. +Kubernetes version 1.5 introduces support for Windows Server Containers. In version 1.5, the Kubernetes control plane (API Server, Scheduler, Controller Manager, etc) continue to run on Linux, while the kubelet and kube-proxy can be run on Windows Server. + +**Note:** Windows Server Containers on Kubernetes is an Alpha feature in Kubernetes 1.5. ## Prerequisites -With the alpha release, Windows Server Containers for Kubernetes is supported using the following: +In Kubernetes version 1.5, Windows Server Containers for Kubernetes is supported using the following: 1. Kubernetes control plane running on existing Linux infrastructure (version 1.5 or later) 2. Kubenet network plugin setup on the Linux nodes @@ -26,37 +28,35 @@ Each Window Server node should have the following configuration: The following diagram illustrates the Windows Server networking setup for Kubernetes Setup ![Windows Setup](windows-setup.png) -## Setup +## Setting up Windows Server Containers on Kubernetes +To run Windows Server Containers on Kubernetes, you'll need to set up both your host machines and the Kubernetes node components for Windows and setup Routes for Pod communication on different nodes ### Host Setup -#### Windows - +**Windows Host Setup** 1. Windows Server container host running Windows Server 2016 and Docker v1.12. Follow the setup instructions outlined by this blog post: https://msdn.microsoft.com/en-us/virtualization/windowscontainers/quick_start/quick_start_windows_server 2. DNS support for Windows recently got merged to docker master and is currently not supported in a stable docker release. To use DNS build docker from master or download the binary from [Docker master](https://master.dockerproject.org/) 3. Pull the `apprenda/pause` image from `https://hub.docker.com/r/apprenda/pause` 4. RRAS (Routing) Windows feature enabled -#### Linux - -Linux hosts should be setup according to their respective distro documentation and the requirements of the Kubernetes version you will be using. Linux hosts also require to have CNI Setup. +**Linux Host Setup** +1. Linux hosts should be setup according to their respective distro documentation and the requirements of the Kubernetes version you will be using. +2. CNI network plugin installed. ### Component Setup -In order to build the work node components, the *kubelet* and *kube-proxy* for Windows, the following needs to be installed on the host +Requirements * Git, Go 1.7.1+ * make (if using Linux or MacOS) * Important notes and other dependencies are listed [here](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md#building-kubernetes-on-a-local-osshell-environment) -#### kubelet - -In order to build the *kubelet*, run: +**kubelet** +To build the *kubelet*, run: 1. `cd $GOPATH/src/k8s.io/kubernetes` 2. Build *kubelet* 1. Linux/MacOS: `KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kubelet` 2. Windows: `go build cmd/kubelet/kubelet.go` -#### kube-proxy - -In order to build *kube-proxy*, run: +**kube-proxy** +To build *kube-proxy*, run: 1. `cd $GOPATH/src/k8s.io/kubernetes` 2. Build *kube-proxy* @@ -64,7 +64,6 @@ In order to build *kube-proxy*, run: 2. Windows: `go build cmd/kube-proxy/proxy.go` ### Route Setup - The below example setup assumes one Linux and two Windows Server 2016 nodes and a cluster CIDR 192.168.0.0/16 | Hostname | Routable IP address | Pod CIDR | @@ -98,11 +97,11 @@ route add 192.168.1.0 mask 255.255.255.0 192.168.1.1 if --pod-infra-container-image="apprenda/pause" --resolv-conf="" --api_servers=` -### kube-proxy +To start kube-proxy on your Windows node: Run the following in a PowerShell window with administrative privileges. Be aware that if the node reboots or the process exits, you will have to rerun the commands below to restart the kube-proxy. @@ -121,8 +120,8 @@ Run the following in a PowerShell window with administrative privileges. Be awar 2. Run *kube-proxy* executable using the below command `.\proxy.exe --v=3 --proxy-mode=userspace --hostname-override= --master= --bind-address=` -## Scheduling PODs on Windows -As Kubernetes control plane currently runs on Linux, the resulting cluster will have both Linux and Windows nodes. TO schedule PODs on Windows, `nodeSelector` constraint has to be set with label `beta.kubernetes.io/os` having value `windows` as shown in the example below, +## Scheduling Pods on Windows +Because your cluster has both Linux and Windows nodes, you must explictly set the nodeSelector constraint to be able to schedule Pods to Windows nodes. You must set nodeSelector with the label beta.kubernetes.io/os to the value windows; see the following example: ``` { "apiVersion": "v1", From 77fd224ce4bca7455128df1ed190d8c3cf5e6219 Mon Sep 17 00:00:00 2001 From: Jitendra Bhurat Date: Fri, 2 Dec 2016 14:13:29 -0500 Subject: [PATCH 5/5] Changes as per review --- docs/getting-started-guides/windows/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/getting-started-guides/windows/README.md b/docs/getting-started-guides/windows/README.md index ec3dfffd4d4ef..80e7fc6e14c48 100644 --- a/docs/getting-started-guides/windows/README.md +++ b/docs/getting-started-guides/windows/README.md @@ -32,12 +32,14 @@ The following diagram illustrates the Windows Server networking setup for Kubern To run Windows Server Containers on Kubernetes, you'll need to set up both your host machines and the Kubernetes node components for Windows and setup Routes for Pod communication on different nodes ### Host Setup **Windows Host Setup** + 1. Windows Server container host running Windows Server 2016 and Docker v1.12. Follow the setup instructions outlined by this blog post: https://msdn.microsoft.com/en-us/virtualization/windowscontainers/quick_start/quick_start_windows_server 2. DNS support for Windows recently got merged to docker master and is currently not supported in a stable docker release. To use DNS build docker from master or download the binary from [Docker master](https://master.dockerproject.org/) 3. Pull the `apprenda/pause` image from `https://hub.docker.com/r/apprenda/pause` 4. RRAS (Routing) Windows feature enabled **Linux Host Setup** + 1. Linux hosts should be setup according to their respective distro documentation and the requirements of the Kubernetes version you will be using. 2. CNI network plugin installed. @@ -48,6 +50,7 @@ Requirements * Important notes and other dependencies are listed [here](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md#building-kubernetes-on-a-local-osshell-environment) **kubelet** + To build the *kubelet*, run: 1. `cd $GOPATH/src/k8s.io/kubernetes` @@ -56,6 +59,7 @@ To build the *kubelet*, run: 2. Windows: `go build cmd/kubelet/kubelet.go` **kube-proxy** + To build *kube-proxy*, run: 1. `cd $GOPATH/src/k8s.io/kubernetes`