Kubernetes windows node not creating network

[kevinbayes]

I followed the scripts provided by msdn and was unable to create a cbr0 network interface required to install a Windows node.

Looking at the start-kubelet.ps1 script I assume that the script should create the cbr0, but it seems to not work correctly and therefore you cannot follow the AddRoutes commands to the windows nodes.

Are there any known issues or manual instructions on how to add this interface?

Getting Started:
https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows
Network Topology:
https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/configuring-host-gateway-mode

Kubernetes:
Kubernetes 1.9.2
System details are below:

OS Name:                   Microsoft Windows Server Standard
OS Version:                10.0.16299 N/A Build 16299
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
Registered Organization:
Product ID:                00396-02142-95046-AA798
Original Install Date:     25/01/2018, 10:09:06 AM
System Boot Time:          25/01/2018, 10:52:10 AM
System Manufacturer:       Dell Inc.
System Model:              OptiPlex 9020
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 60 Stepping 3 GenuineIntel ~1700 Mhz
BIOS Version:              Dell Inc. A03, 17/09/2013
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume1
Total Physical Memory:     16,326 MB
Available Physical Memory: 14,258 MB
Virtual Memory: Max Size:  19,270 MB
Virtual Memory: Available: 17,277 MB
Virtual Memory: In Use:    1,993 MB
Hotfix(s):                 1 Hotfix(s) Installed.
                           [01]: KB4054517
Network Card(s):           2 NIC(s) Installed.
                           [01]: Intel(R) Ethernet Connection I217-LM
                                 Connection Name: Ethernet
                                 DHCP Enabled:    Yes
                                 DHCP Server:     172.20.1.6
                                 IP address(es)
                                 [01]: 172.20.40.126
                                 [02]: fe80::d8d5:fce8:baef:3eca
                           [02]: Hyper-V Virtual Ethernet Adapter
                                 Connection Name: vEthernet (nat)
                                 DHCP Enabled:    Yes
                                 DHCP Server:     255.255.255.255
                                 IP address(es)
                                 [01]: 192.168.96.1
                                 [02]: fe80::b50d:fd19:b328:1fe6
Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

[elee3]

I am having this issue as well.

[funky81]

Have you do manual installation of kubernetes like instructions in the URL above ? Mine having works perfectly after execute startKubulet.ps1, startKuberproxy and script to network (network section).

[elee3]

Line in the AddRoutes.ps1 script such as

$na = Get-NetAdapter | ? Name -Like "vEthernet (Ethernet*"

just do not work since there is no such network adapter in a fresh installation of Windows Server 2016. How did you get it to work?

[funky81]

Have you tried in Powershell?

[elee3]

yes i have tried running in powershell. it didnt work because of the reason i mentioned above

[JMesser81]

@elee3 @funky81 - we just updated the scripts. Can you please try again?

[funky81]

Hi @JMesser81 , i've tried your latest script from Linux and windows nodes. Somehow it didn't worked. I tried to ping to outside cluster from win-webserver (example) pod.

My masterIP (192.168.34.0/24) is outside my worker node IP (192.168.91.0/24). Pod CIDR is 10.168.0.0/16.

Another info, I'm using Centos 7.4 with firewalld, SElinux disable also with Swapoff.

Right now I try to make node (windows) and management node (Centos) as in one network. Also tried to replace firewalld with iptables since generate-routes.sh involve iptables instead of firewalld.

Any solutions for this? I tried hard to make things work but it still doesn't worked.

[Jaspreetc]

@JMesser81
Hi Jason
I am having the same problem with the latest script.
i can list the pods , the Kubernetes is working but for some reason its not listing cbr0 network

$kubectl get pods --all-namespaces
kube-system kube-addon-manager-server1 1/1 Running 0 15m
kube-system kube-apiserver-server1 1/1 Running 0 15m
kube-system kube-controller-manager-server1 1/1 Running 0 14m
kube-system kube-etcd-server1 1/1 Running 0 14m
kube-system kube-scheduler-server1 1/1 Running 0 15m

In the script its trying to use network plugin as "kubenet" i am not sure how kubernetes will come to know about the network plugins, are they specified in the "container image"
Since we have not specified anything specific location for CNI plugins.

$ps aux
root 45250 3.5 2.8 1636748 114956 pts/1 Sl 12:22 0:41 ./bin/hyperkube kubelet --kubeconfig=/root/.kube/config --pod-infra-container-image=gcrio.azureedge.net/google_containers/pause-amd64:3.0 --address=0.0.0.0 --allow-privileged=true --enable-server --enable-debugging-handlers --pod-manifest-path=/root/kube/manifest --cluster-dns=11.0.0.10 --cluster-domain=cluster.local --node-labels=role=master --hairpin-mode=promiscuous-bridge --container-runtime=docker --v=6 --fail-swap-on=false --network-plugin=kubenet

[Jaspreetc]

@JMesser81
do we need to add the below switch somewhere in the shell script
https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#kubenet

kubenet
Kubenet is a very basic, simple network plugin, on Linux only. It does not, of itself, implement more advanced features like cross-node networking or network policy. It is typically used together with a cloud provider that sets up routing rules for communication between nodes, or in single-node environments.

Kubenet creates a Linux bridge named cbr0 and creates a veth pair for each pod with the host end of each pair connected to cbr0. The pod end of the pair is assigned an IP address allocated from a range assigned to the node either through configuration or by the controller-manager. cbr0 is assigned an MTU matching the smallest MTU of an enabled normal interface on the host.

The plugin requires a few things:

The standard CNI bridge, lo and host-local plugins are required, at minimum version 0.2.0. Kubenet will first search for them in /opt/cni/bin. Specify cni-bin-dir to supply additional search path. The first found match will take effect.
Kubelet must be run with the --network-plugin=kubenet argument to enable the plugin
Kubelet should also be run with the --non-masquerade-cidr= argument to ensure traffic to IPs outside this range will use IP masquerade.

The node must be assigned an IP subnet through either the --pod-cidr kubelet command-line option or the --allocate-node-cidrs=true --cluster-cidr= controller-manager command-line options.

[calebpalmer]

I'm in the same boat. I can't add the windows routes because there is no vEthernet (cbr0).

[JMesser81]

@daschott - can you take a look at this? We recently updated our HOW TO guide and David can provide an update (or even better, post on our repo) :-)

[daschott]

Here is the most recent How-to guide for deploying a mixed-OS Kubernetes cluster using Flannel. These instructions will work on either Windows Server, version 1803 or Windows Server 2019. These are also compliant with the way a Linux master is initialized using kubeadm, making deployment of not just Windows, but also Linux easier than before.

The benefit of using Flannel as a network management solution is simpler configuration and scalability, as it programs routes automatically for you. This is in contrast to our (current) public documentation that is trickier to deploy, but offers more control due to manual route management.

[calebpalmer]

I followed the instructions again. Thank you very much for updating them. I'm running into this issue running start.ps1 to join the windows node. It just keeps spinning with this error:

"E0618 09:52:23.273306 1188 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.26.64.195:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dfrd-vtapp-kub03&limit=500&resourceVersion=0: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")"

A couple of differences to note:

• Master node is running kubernetes 1.10.4 on CentOS 7.5.1804
• The windows node is 10.0.16299 N/A Build 16299 (1709)

[calebpalmer]

Please ignore my last comment. I accidentally copied the wrong config file. The windows node was joined! Thanks!

[funky81]

Could you ping to external cluster from inside of pod? I'm having trouble to this therefore my pod can't communicate to external resource. Works fine communicate to other pod instead cluster.

…

On Mon, Jun 18, 2018, 8:03 PM Caleb Palmer ***@***.***> wrote: Please ignore my last comment. I accidentally copied the wrong config file. The windows node was joined! Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#170 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGk7KoGiI8d-0vkL2D-OhFdZySFZ3UJks5t96UKgaJpZM4RsPVU> .

[daschott]

@funky81 For outbound connectivity, ICMP traffic may not work, as this is a limitation today!

Can you try TCP or UDP to double-check pods have outbound connectivity? For example, curl -useb http://bing.com?

[funky81]

Nope, it still doesn't work. Sol

…

On Mon, Jun 18, 2018, 11:48 PM David Schott ***@***.***> wrote: @funky81 <https://github.com/funky81> For outbound connectivity, ICMP traffic may not work, as this is a limitation today! Can you try TCP or UDP to double-check pods have outbound connectivity? For example, curl -useb http://bing.com? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#170 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGk7DA2BrJKVteW2qcm1-h1jIevBECpks5t99npgaJpZM4RsPVU> .

[Moofasax]

Any updates on this issue? We are facing the same, pods do not have outbound connectivity running on a windows worker

[daschott]

This definitely works today. @Moofasax do you have Mac Spoofing enabled if you're using VMs?

[daschott]

The other thing to check is the FAQ on outbound NAT:

“My Windows pods cannot ping external websites”
Windows pods do not have outbound rules programmed for the ICMP protocol today. However, TCP/UDP is supported. When trying to demonstrate connectivity to resources outside of the cluster, please substitute ping with corresponding curl commands. If you are still facing problems, most likely your network configuration in cni.conf deserves some extra attention. You can always edit this static file, the configuration will be applied to any newly created Kubernetes resources.
One of the Kubernetes networking requirements is for cluster communication to occur without NAT internally. To honor this requirement, we have an ExceptionList for all the communication where we do not want outbound NAT to occur. However, this also means that you need to exclude the external IP you are trying to query from the ExceptionList. Only then will the traffic originating from your Windows pods be SNAT’ed correctly to receive a response from the outside world.

[Moofasax]

Sorry i dont think i fully understand yet, does that mean the exception list would be 0.0.0.0/0 if i wanted my pods to reach out to any website?

                                      `{
                                            "Name":  "EndpointPolicy",
                                            "Value":  {
                                                          "Type":  "OutBoundNAT",
                                                          "ExceptionList":  [
                                                                                "0.0.0.0/0"
                                                                            ]
                                                      }
                                        }`

thanks! I can post on stackoverflow if that is a more applicable place for me to ask questions, thanks!

[daschott]

@Moofasax @funky81 Does the following help?
https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems#my-windows-pods-cannot-ping-external-resources

Can you please share CollectLogs.ps1 + IP of the impacted pod so we can see what went wrong?

[JasonChinsen]

I am running into the same issue where the windows node is not able to create a route. I had to update the AddRoutes.ps1 file to change the $endpointName to point to cbr0_ep to match how it was created.

Note that I am trying this on AWS (KVM based), I have not been able to run Set-VMProcessor (Hyper-v based)

PS C:\k\temp\SDN\Kubernetes\windows> Set-VMProcessor
Set-VMProcessor : The term 'Set-VMProcessor' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ Set-VMProcessor
+ ~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Set-VMProcessor:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      17763  0


PS C:\k\temp\SDN\Kubernetes\windows> systeminfo | findstr OS
OS Name:                   Microsoft Windows Server Datacenter
OS Version:                10.0.17763 N/A Build 17763
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
BIOS Version:              Xen 4.2.amazon, 8/24/2006
PS C:\k\temp\SDN\Kubernetes\windows>

==========================================================

Update ... I was able to get passed this issue. I ended up upgrading to kub 1.10.11.

Thanks

[daschott]

sorry for the long delay here... The new overlay instructions are out now. We have made a number of improvements since last year such that Windows Server version 1809 or above are supported on Kubernetes v1.14 or above.

More information on how to troubleshoot Kubernetes networking can be found here:
https://techcommunity.microsoft.com/t5/Networking-Blog/Troubleshooting-Kubernetes-Networking-on-Windows-Part-1/ba-p/508648

Here is the K8s docs: https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-nodes/

Please submit new issue and cc me if you continue seeing problems on latest supported versions, thank you!