[K3s][Windows Port] Build script, multi-call binary, and Flannel

[seanyen]

Proposed Changes

This is the initial set of K3s agent for Windows port. This pull request is focused on:

1. Modifying the build script for Windows build
2. Modifying the multi-call binary logic to accommodate Windows convention.
3. Modifying the Flannel and kube-proxy setup code for Windows node to run.

Types of Changes

New feature

Verification

Run GOOS=windows CXX=x86_64-w64-mingw32-g++ CC=x86_64-w64-mingw32-gcc SKIP_VALIDATE=true ./scripts/ci on Ubuntu machine to proof build Windows port.

NOTE: You may need to install the following packages to build:

sudo apt-get install mingw-w64
sudo apt-get install zstd

Testing

I am testing it on a Windows dev box with Hyper-V enabled, and I created two VMs:

1. Linux VM (in my case, I am using CBL-Mariner 2.0, but Ubuntu should work too).
2. Windows VM (Windows 11 Dev VM from the Hyper-V quickstart).

Two VMs are on the same subnet (in my case, I am using Default Switch).

On Linux VM, I deployed a k3s control plane node with all the default configuration and necessary firewall rules. Additionally, I pass --disable-network-policy to k3s server command, since Windows node doesn't support network policy yet.

On Windows VM, I prepared a k3s-config.yaml file and the content is as follows:

token: '<join token>'
server: 'https://<control plane VIP>:6443'
kubelet-arg=cgroups-per-qos: 'false'
kubelet-arg=enforce-node-allocatable: ''
kubelet-arg=resolv-conf: ''
pause-image: '<the windows pause image matching to your Windows VM>'

Example for pause-image: mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2022-amd64
List available pause images: curl -s -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" https://mcr.microsoft.com/v2/oss/kubernetes/pause/manifests/3.8

And additionally, I added the following firewall rules on Windows VM:

New-NetFirewallRule -Name k3s -DisplayName 'k3s' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250
New-NetFirewallRule -Name k3sOverlayTraffic4789UDP -DisplayName "K3s Overlay Traffic 4789 UDP" -Enabled True -Direction Inbound -Protocol UDP -Action Allow -LocalPort 4789

And finally, I ran K3s on Windows VM by the following command:

.\k3s.exe agent --config k3s-config.yaml

Linked Issues

#7258

User-Facing Change

None.

Further Comments

None.

[codecov-commenter]

Codecov Report

Attention: 23 lines in your changes are missing coverage. Please review.

Comparison is base (ba750e2) 47.27% compared to head (6e5f9ea) 51.08%.
Report is 3 commits behind head on master.

❗ Current head 6e5f9ea differs from pull request most recent head 2279e13. Consider uploading reports for the commit 2279e13 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7259      +/-   ##
==========================================
+ Coverage   47.27%   51.08%   +3.80%     
==========================================
  Files         143      145       +2     
  Lines       14806    14994     +188     
==========================================
+ Hits         6999     7659     +660     
+ Misses       6692     6118     -574     
- Partials     1115     1217     +102     

Flag	Coverage Δ
e2etests	48.34% <30.76%> (?)
inttests	44.28% <30.30%> (-0.33%)	⬇️
unittests	19.51% <0.00%> (-0.21%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
pkg/daemons/executor/embed.go	68.32% <100.00%> (-1.25%)	⬇️
pkg/daemons/executor/embed_linux.go	100.00% <100.00%> (ø)
pkg/agent/run.go	45.12% <25.00%> (+1.18%)	⬆️
pkg/cli/crictl/crictl.go	0.00% <0.00%> (ø)
pkg/kubectl/main.go	47.22% <0.00%> (-4.30%)	⬇️
pkg/agent/flannel/setup.go	36.36% <22.22%> (-2.15%)	⬇️

... and 44 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[brandond]

Thank you for the PR! I have a couple nits on some of the inline os-specific logic, I'm excited to get this merged once those are addressed!

[brandond]

@seanyen are you still working on this?

[seanyen]

@seanyen are you still working on this?

yes, sorry I was little bit slow on this. I am distracted by other works and will be back to this one in 2 week-ish.

[cwayne18]

@seanyen very much appreciate the work on this! Please note we can't merge unsigned commits due to our DCO check, can you please make sure to commit with -s

[seanyen]

@seanyen very much appreciate the work on this! Please note we can't merge unsigned commits due to our DCO check, can you please make sure to commit with -s

sure thing! I am still working on the feedback and definitely will make sure all checks are green before getting your attention again.

[cwayne18]

Thanks so much!

[brandond]

You can update your PR to use the new tag: https://github.com/rancher/plugins/releases/tag/v1.3.0-k3s1

k3s/scripts/version.sh

Line 56 in fe9604c

VERSION_CNIPLUGINS="v1.2.0-k3s1"

[brandond]

lgtm, but looks like it needs a rebase

[seanyen]

lgtm, but looks like it needs a rebase

Thanks. I did a rebase. Let me know anything else to take care.

[brandond]

Did you go mod tidy after rebasing? It looks like there are some missing imports.

[brandond]

Multiple builds are failing with:

+ go build -tags urfave_cli_no_docs -ldflags '
    -X github.com/k3s-io/k3s/pkg/version.Version=v1.27.3+k3s-bbee5d7d
    -X github.com/k3s-io/k3s/pkg/version.GitCommit=bbee5d7d
    -w -s
 -extldflags '\''-static'\''' -o dist/artifacts/k3s ./cmd/k3s
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.

@dereknola do you think this might be related to your change to exclude some of the git repo structure from being copied into the build image?

[seanyen]

Cool, now I saw all checks are all green now. Thank @dereknola.

[manuelbuil]

Looks good! I just have a couple of comments

[seanyen]

@seanyen A quick rebase will be necessary.

Thanks! The rebase was done.

[cwayne18]

YOLO 🎉🎉

[fcabrera23]

Congrats! Thank you all for the hard work

[cwayne18]

Thanks so much Francisco and Sean for all your work on this, we're very excited to continue working with y'all on this!

[amenome]

Silly question - ran through the build process (make download && make generate && make), but didn't get a k3s.exe out of it - what's the proper step-by-step procedure to build the k3s.exe?

Thanks!

[amenome]

Okay, I think maybe I just didn't read closely enough. Looks like we need to run this on a WSL prompt:

wget https://go.dev/dl/go1.21.9.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.21.9.linux-amd64.tar.gz

sudo apt-get -y install mingw-w64
sudo apt-get -y install zstd
git clone https://github.com/k3s-io/k3s.git
cd k3s
GOOS=windows CXX=x86_64-w64-mingw32-g++ CC=x86_64-w64-mingw32-gcc SKIP_VALIDATE=true ./scripts/ci 

Tested and worked! Had to downgrade to go 1.21.9 though!