feat: (IAC-1323) Add support for cilium byoCNI with eBPF dataplane

[hms-tkl]

Hello,

expanding on this issue about CNI overlay [0], I would like to request support for bring your own Container Network Interface (CNI) plugin with Azure Kubernetes Service [1]

I have prepared a PR that demonstrates the feature in a nutshell [2]. It might currently break during initial run, as there seems to be a race-condition when running a "local-exec" provisioner to configure AKS after bootstrapping - current work-around is a simple re-run of terraform apply -var-file input-minimal.tfvars to finish provisioning the whole setup.

Let me know if I can help any further with development of this feature. Would love to see this option officially supported.

[0] #343
[1] https://learn.microsoft.com/en-us/azure/aks/use-byo-cni
[2] #350

[riragh]

@hms-tkl, Thank you for providing the PR with changes required for byo-cni. I reviewed the PR and it needs a lot more adjustments to be in parity with the other supported options.
Also, rerunning apply to finish provisioning is not the ideal option for us to release features/enhancements. The work for Azure CNI Overlay is planned for early next year and this option of byo-cni will be reinvestigated then.

[hms-tkl]

Thank you for the feedback @riragh

I'm totally aware that this is a very dirty first draft - and I'm happy to announce that there is no race-condition and the apply was successfull on first try each time I ran it locally on my machine.

Besides that, I'm happy to hear that you will re-investigate the byo-cni option. To give you a little background of the "why" I was proposing this change:

Utilizing cilium and it's capabilities would allow us to leverage ingress-functionality (and deprecate nginx ingress component on the next layer), observerability (hubble) and network-policies for the cluster. Coming from a platform operator role, I'd greatly appreciate using modern technologies (eBPF) for robust and stable operations, while reducing the toil of upgrading additional components on other layers (aka. the ingress component)

I'd be happy to dedicate time and resources to make this feature happen - please let me know how I can help the project (be it testing or otherwise)

[riragh]

Internal ticket was created to review and investigate this enhancement request further. The associated PR is closed as the changes currently in the PR cannot be accepted to be added to the codebase.